Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM)
BMC Medical Informatics and Decision Making, vol. 22, pp. 183
Abstract
BACKGROUND: Evaluating patients' experiences is essential when incorporating the patients' perspective in improving healthcare. Experiences are mainly collected using closed-ended questions, although the value of open-ended questions is widely recognized. Natural language processing (NLP) can automate the analysis of open-ended questions for an efficient approach to patient-centeredness. METHODS: We developed the Artificial Intelligence Patient-Reported Experience Measures (AI-PREM) tool, consisting of a new, open-ended questionnaire, an NLP pipeline to analyze the answers using sentiment analysis and topic modeling, and a visualization to guide physicians through the results. The questionnaire and NLP pipeline were iteratively developed and validated in a clinical context. RESULTS: The final AI-PREM consisted of five open-ended questions about the provided information, personal approach, collaboration between healthcare professionals, organization of care, and other experiences. The AI-PREM was sent to 867 vestibular schwannoma patients, 534 of which responded. The sentiment analysis model attained an F1 score of 0.97 for positive texts and 0.63 for negative texts. There was a 90% overlap between automatically and manually extracted topics. The visualization was hierarchically structured into three stages: the sentiment per question, the topics per sentiment and question, and the original patient responses per topic. CONCLUSIONS: The AI-PREM tool is a comprehensive method that combines a validated, open-ended questionnaire with a well-performing NLP pipeline and visualization. Thematically organizing and quantifying patient feedback reduces the time invested by healthcare professionals to evaluate and prioritize patient experiences without being confined to the limited answer options of closed-ended questions.
Authors 6
-
Marieke Meija van Buchem corresponding Aachen Clinical Artificial Intelligence Implementation and Research Lab (CAIRELab) Department of Biomedical Data Sciences Information Technology & Digital Innovation Department
Leiden University Medical Center
Affiliation as printed
Clinical Artificial Intelligence Implementation and Research Lab (CAIRELab), Leiden University Medical Center, Leiden, the Netherlands. m.m.van_buchem@lumc.nl
Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands. m.m.van_buchem@lumc.nl
Information Technology & Digital Innovation Department, Leiden University Medical Center, Leiden, the Netherlands. m.m.van_buchem@lumc.nl
Information Technology & Digital Innovation Department, Leiden University Medical Center, Leiden, the Netherlands
-
Leiden University Medical Center
Affiliation as printed
Department of Otorhinolaryngology and Head and Neck Surgery, Leiden University Medical Center, Leiden, the Netherlands
-
Ilse M.J. Kant Aachen Clinical Artificial Intelligence Implementation and Research Lab (CAIRELab) Department of Biomedical Data Sciences Information Technology & Digital Innovation Department
Leiden University Medical Center
Affiliation as printed
Clinical Artificial Intelligence Implementation and Research Lab (CAIRELab), Leiden University Medical Center, Leiden, the Netherlands
Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands
Information Technology & Digital Innovation Department, Leiden University Medical Center, Leiden, the Netherlands
-
Ewout Willem Steyerberg Aachen Clinical Artificial Intelligence Implementation and Research Lab (CAIRELab) Department of Biomedical Data Sciences
Leiden University Medical Center
Affiliation as printed
Clinical Artificial Intelligence Implementation and Research Lab (CAIRELab), Leiden University Medical Center, Leiden, the Netherlands
Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands
-
Affiliation as printed
Morgens, Leiden, the Netherlands
-
Leiden University Medical Center
Affiliation as printed
Department of Otorhinolaryngology and Head and Neck Surgery, Leiden University Medical Center, Leiden, the Netherlands
Cited by 51 stored of 51
No patents citing this paper on Lens.org (checked 2026-10-11).