Development of Machine Learning Algorithms for Identifying Patients With Limited Health Literacy

Journal of Evaluation in Clinical Practice, vol. 31, pp. e14248

Abstract

RATIONALE: Limited health literacy (HL) leads to poor health outcomes, psychological stress, and misutilization of medical resources. Although interventions aimed at improving HL may be effective, identifying patients at risk of limited HL in the clinical workflow is challenging. With machine learning (ML) algorithms based on readily available data, healthcare professionals would be enabled to incorporate HL screening without the need for administering in-person HL screening tools. AIMS AND OBJECTIVES: Develop ML algorithms to identify patients at risk for limited HL in spine patients. METHODS: Between December 2021 and February 2023, consecutive English-speaking patients over the age of 18 and new to an urban academic outpatient spine clinic were approached for participation in a cross-sectional survey study. HL was assessed using the Newest Vital Sign and the scores were divided into limited (0-3) and adequate (4-6) HL. Additional patient characteristics were extracted through a sociodemographic survey and electronic health records. Subsequently, feature selection was performed by random forest algorithms with recursive feature selection and five ML models (stochastic gradient boosting, random forest, Bayes point machine, elastic-net penalized logistic regression, support vector machine) were developed to predict limited HL. RESULTS: Seven hundred and fifty-three patients were included for model development, of whom 259 (34.4%) had limited HL. Variables identified for predicting limited HL were age, Area Deprivation Index-national, Social Vulnerability Index, insurance category, Body Mass Index, race, college education, and employment status. The Elastic-Net Penalized Logistic Regression algorithm achieved the best performance with a c-statistic of 0.766, calibration slope/intercept of 1.044/-0.037, and Brier score of 0.179. CONCLUSION: Elastic-Net Penalized Logistic Regression had the best performance when compared with other ML algorithms with a c-statistic of 0.766, calibration slope/intercept of 1.044/-0.037, and a Brier score of 0.179. Over one-third of patients presenting to an outpatient spine center were found to have limited HL. While this algorithm is far from being used in clinical practice, ML algorithms offer a potential opportunity for identifying patients at risk for limited HL without administering in-person HL assessments. This could possibly enable screening and early intervention to mitigate the potential negative consequences of limited HL without taxing the existing clinical workflow.

Authors 6

1. Dylan Koole corresponding Aachen Department of Orthopaedic Surgery Department of Orthopaedic Surgery

   Leiden University · Harvard University · Leiden University Medical Center · Massachusetts General Hospital

   Affiliation as printed

   Department of Orthopaedic Surgery, Leiden University Medical Center Leiden University Leiden The Netherlands

   Department of Orthopaedic Surgery, Orthopaedic Oncology Service Massachusetts General Hospital, Harvard Medical School Boston Massachusetts USA

   Department of Orthopaedic Surgery, Leiden University Medical Center, Leiden University, Leiden, The Netherlands

   Department of Orthopaedic Surgery, Orthopaedic Oncology Service, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts, USA

2. Oscar Yuan-Jie Shen

   Harvard University · Massachusetts General Hospital

   Affiliation as printed

   Department of Orthopaedic Surgery, Orthopaedic Oncology Service Massachusetts General Hospital, Harvard Medical School Boston Massachusetts USA

   Department of Orthopaedic Surgery, Orthopaedic Oncology Service, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts, USA

3. Amanda Lans

   Harvard University · Utrecht University · University Medical Center Utrecht · Massachusetts General Hospital

   Affiliation as printed

   Department of Orthopaedic Surgery, Orthopaedic Oncology Service Massachusetts General Hospital, Harvard Medical School Boston Massachusetts USA

   Department of Orthopaedic Surgery, University Medical Center Utrecht Utrecht University Utrecht The Netherlands

   Department of Orthopaedic Surgery, Orthopaedic Oncology Service, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts, USA

   Department of Orthopaedic Surgery, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

4. Tom Maarten Groot

   University Medical Center Groningen · Harvard University · University of Groningen · Massachusetts General Hospital

   Affiliation as printed

   Department of Orthopaedic Surgery, Orthopaedic Oncology Service Massachusetts General Hospital, Harvard Medical School Boston Massachusetts USA

   Department of Orthopaedic Surgery, University Medical Center Groningen University of Groningen Groningen The Netherlands

   Department of Orthopaedic Surgery, Orthopaedic Oncology Service, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts, USA

   Department of Orthopaedic Surgery, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands

5. Jorrit‐Jan Verlaan

   Utrecht University · University Medical Center Utrecht

   Affiliation as printed

   Department of Orthopaedic Surgery, University Medical Center Utrecht Utrecht University Utrecht The Netherlands

   Department of Orthopaedic Surgery, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

6. Joseph H. Schwab

   Harvard University · Massachusetts General Hospital

   Affiliation as printed

   Department of Orthopaedic Surgery, Orthopaedic Oncology Service Massachusetts General Hospital, Harvard Medical School Boston Massachusetts USA

   Department of Orthopaedic Surgery, Orthopaedic Oncology Service, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts, USA