Evaluating AI Performance in Systematic Literature Reviews for HEOR: A Case Study

Journal of health economics and outcomes research, vol. 13, pp. 46–54

Abstract

Background: Systematic literature reviews (SLRs) are central to evidence generation in health economics and outcomes research, but traditional SLR workflows are labor-intensive. Artificial intelligence (AI) is increasingly being explored to support evidence synthesis tasks. Objectives: To describe methods used to assess AI performance in screening and data extraction in SLRs, and to provide recommendations on how to ensure appropriate use of AI in literature reviews. Methods: We conducted an AI-assisted SLR that mirrored a traditional SLR performed by humans only and assessed the performance of the AI tools employed. AI was used to screen titles/abstracts, screen full texts, and conduct data extraction. Using the traditional SLR as the benchmark, AI performance for title/abstract screening was evaluated in terms of accuracy, recall, and precision at predefined screening milestones, followed by the accuracy assessment of full-text screening by AI. Data items extracted by AI were verified by humans and categorized as correct, incomplete, missing, incorrect, or requiring human checks. The average accuracy rate calculated on the basis of correct data items extracted was used as an indicator of AI performance in data extraction. Results: During title/abstract screening, accuracy and recall remained high but precision was low, increasing false positives. Full-text screening identified a minority of studies included in the traditional SLR. Mean extraction accuracy was 72.93% (range, 57.69%-88.46%). No data were hallucinated by the AI model used.

Authors 6

1. Jing Wang-Silvanto
   Affiliation as printed

   Astellas Pharma A/S, filial I Finland, Espoo; Observational Health Data Sciences and Informatics (OHDSI), Finland

2. Mansee Jajoo

   Astellas Pharma (United States)

   Affiliation as printed

   Astellas Pharma, Astellas Pharma Global Development Inc. Northbrook, Illinois, USA

3. He Guo

   Astellas Pharma (United States)

   Affiliation as printed

   Astellas Pharma, Astellas Pharma Global Development Inc. Northbrook, Illinois, USA

4. Rishi Ohri

   Astellas Pharma (United States)

   Affiliation as printed

   Astellas Pharma, Astellas Pharma Global Development Inc. Northbrook, Illinois, USA

5. Judith Nelissen Aachen Astellas Pharma Europe BV

   Astellas Pharma (Netherlands)

   Affiliation as printed

   Astellas Pharma Europe BV, Leiden, The Netherlands

6. Carolina Casañas i Comabella

   Thermo Fisher Scientific (United Kingdom)

   Affiliation as printed

   PPD Evidera Health Economics & Market Access, Thermo Fisher Scientific, London, UK