Utilizing Large Language Models to Enhance Patient-Reported Outcome Measures: Application to the EQ-5D-5L and Bolt-Ons

Value in Health

Abstract

OBJECTIVES: Large language models (LLMs) may be useful tools for the development/adaptation of patient-reported outcome measures. As a methodological proof-of-concept, we evaluated the use of LLMs to support the identification of potential EuroQol 5-Dimension 5-Level (EQ-5D-5L) bolt-on dimensions, using patient-reported free-text data. METHODS: We used GPT-4o to analyze text data from 1977 members of the Dutch Celiac Association, who completed the EQ-5D-5L and narratively described the impact of celiac disease on their lives. Prompts were designed to identify potential EQ-5D-5L bolt-on dimensions and produce preliminary item wordings for selected dimensions. Evaluations comprised comparisons of dimensions identified using 2 alternative approaches (qualitative analysis and topic modeling) conducted on a subset of 85 text entries, text-entry level agreement (Kappa) between LLM and qualitatively identified dimensions, and suitability of LLM-generated item wordings assessed against existing criteria. RESULTS: The LLM identified 12 potential bolt-on dimensions to the EQ-5D-5L, of which 9 were also identified using qualitative analysis, and 5 using topic modeling. Text-entry-level agreement between the LLM and qualitative approaches was "moderate," "substantial," or "almost perfect," with 2 exceptions of slight/fair agreement (median Kappa = 0.68, IQR = 0.56-0.76). Sensitivity analyses using 4 other LLMs produced similar agreement results. The LLM-generated item wordings for the 4 most common dimensions scored 4.0 to 4.4 out of 5 when assessed against existing criteria. CONCLUSIONS: This study demonstrates the potential of LLMs to support the development/modification of patient-reported outcome measures based on patient-reported text data. Further research should assess the approach's transferability across disease areas and data sources, while better incorporating patient/stakeholder input throughout.

Authors 7

1. Jan Heijdra Suasnabar corresponding Aachen Department of Biomedical Data Science

   Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Science, Leiden University Medical Center, Leiden, The Netherlands. Electronic address: j.m.heijdra_suasnabar@lumc.nl

2. Marieke Meija van Buchem Aachen Department of Information Technology and Digital Innovation

   Leiden University Medical Center

   Affiliation as printed

   Department of Information Technology and Digital Innovation, Leiden University Medical Center, Leiden, The Netherlands

3. Mathieu F. Jansen

   Erasmus MC

   Affiliation as printed

   Department of Psychiatry, Erasmus MC, Rotterdam, The Netherlands

4. Aureliano Finch

   EuroQol Research Foundation

   Affiliation as printed

   EuroQol Office, EuroQol Research Foundation, Rotterdam, The Netherlands; Health Values Research and Consultancy, The Netherlands

5. Brendan James Mulhern

   University of Technology Sydney

   Affiliation as printed

   Centre for Health Economics Research and Evaluation, University of Technology Sydney, Sydney, Australia

6. Marco René Spruit Aachen

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Public Health and Primary Care, Leiden University Medical Center, Leiden, The Netherlands; Leiden Institute of Advanced Computer Science, Leiden University, Leiden, The Netherlands

7. M. Elske van den Akker – van Marle Aachen Department of Biomedical Data Science

   Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Science, Leiden University Medical Center, Leiden, The Netherlands