Automated literature screening using large language models in headache research: a comparative performance evaluation of seven models using zero-shot and iterative prompt refinement

BMC Medical Informatics and Decision Making

Abstract

Abstract Background Large language models (LLMs) show promise for automating literature screening, although their performance varies across clinical domains and further evaluation is required for broader implementation. This study aimed to assess the accuracy, reliability and feasibility of literature screening in headache research across seven LLMs, spanning compact to frontier models (GPT-4o, DeepSeek-V3, and Meta’s LLaMA-3 at multiple scales). Methods This study is a comparative methodological evaluation of LLM-assisted title and abstract screening. Using a dataset from a real-world systematic review including randomized controlled trials on placebo and nocebo responses in migraine, we evaluated classification performance, processing runtimes, and computational costs across seven models using zero-shot prompting and IMAPR (Iterative Multi-Agent Prompt Refinement). The IMAPR system is our in-house developed, LLM-driven multi-agent framework in which a single, fixed LLM iteratively refines prompts. Each model was run five times per method, and sensitivity was assessed against a predefined threshold of ≥ 95%. Results Zero-shot sensitivity means ranged from 95.1 to 100%, with all models except GPT-4o meeting the ≥ 95% threshold in every run, while specificity ranged from 9.0 to 89.9%. IMAPR increased GPT-4o sensitivity above 95% in all runs, with a minor specificity decrease (-0.9%), while sensitivity became more variable in other models. Specificity improved significantly for LLaMA-3.1 8B (59.2% to 82.4%) and DeepSeek-V3 (77.8% to 85.8%), reducing screening workload at the mean cost of two to three fewer identified relevant citations. Positive predictive value improved but remained low (3.4–23.3%). Zero-shot screening completed within 50 min per model at a cost of up to $4.54, and the complete IMAPR workflow within three hours at a cost below $11. Conclusion This study suggests that LLM-based automated literature screening could be reliable, fast, and affordable in headache research. We encourage researchers and clinicians to critically and prospectively evaluate LLM-assisted screening across diverse medical specialties, independent datasets, and future generations of LLMs to facilitate responsible broader adoption. Trial registration Not applicable.

Authors 10

1. Noor Bruijn

   Erasmus MC

   Affiliation as printed

   Department of Internal Medicine, Division of Vascular Medicine and Pharmacology, Erasmus MC University Medical Center, PO Box 2040, Rotterdam, 3000 CA, The Netherlands

2. A. Kuiper

   Delft University of Technology

   Affiliation as printed

   Department of Software Technology, Web Information Systems, Delft University of Technology, Delft, Netherlands

3. L. Al-Hassany

   Erasmus MC

   Affiliation as printed

   Department of Internal Medicine, Division of Vascular Medicine and Pharmacology, Erasmus MC University Medical Center, PO Box 2040, Rotterdam, 3000 CA, The Netherlands

4. Z. Sun

   Singapore University of Technology and Design

   Affiliation as printed

   Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore, Singapore

5. R. Fronczek Aachen Department of Neurology, Leiden University Medical Center (LUMC)

   Leiden University Medical Center · Stichting Epilepsie Instellingen Nederland

   Affiliation as printed

   Department of Neurology, Leiden University Medical Center, Leiden, Netherlands

   Stichting Epilepsie Instellingen Nederland (SEIN), Sleep Wake Centre, Heemstede, the Netherlands

6. E. M. Huessler

   Essen University Hospital · University of Duisburg-Essen

   Affiliation as printed

   Institute for Medical Informatics, Biometry and Epidemiology, University Hospital Essen, University Duisburg-Essen, Essen, Germany

7. H. C. Diener

   Essen University Hospital · University of Duisburg-Essen

   Affiliation as printed

   Institute for Medical Informatics, Biometry and Epidemiology, University Hospital Essen, University Duisburg-Essen, Essen, Germany

8. M. D. Ferrari Aachen Department of Neurology, Leiden University Medical Center (LUMC)

   Leiden University Medical Center

   Affiliation as printed

   Department of Neurology, Leiden University Medical Center, Leiden, Netherlands

9. J. Yang

   Delft University of Technology

   Affiliation as printed

   Department of Software Technology, Web Information Systems, Delft University of Technology, Delft, Netherlands

10. A. MaassenVanDenBrink corresponding

    Erasmus MC

    Affiliation as printed

    Department of Internal Medicine, Division of Vascular Medicine and Pharmacology, Erasmus MC University Medical Center, PO Box 2040, Rotterdam, 3000 CA, The Netherlands