Evaluating machine learning tools to assist title and abstract screening in systematic literature reviews: a report based on the EULAR RA Management Recommendations Task Force
Annals of the Rheumatic Diseases
Abstract
OBJECTIVES: Systematic literature reviews (SLRs) provide the scientific basis for European Alliance of Associations for Rheumatology (EULAR) task force projects, but they are highly time- and labour-intensive in an ever-growing research landscape, repetitive, and susceptible to human error. In this study, we evaluate the performance of machine learning (ML) models developed for semiautomated title and abstract screening, aiming at supporting and accelerating future review processes. METHODS: Title and abstract screening of 4 SLRs, conducted to inform the 2025 update of the EULAR management recommendations for rheumatoid arthritis, was replicated using different ML models, with manual screening serving as reference standard. Eligible software was identified via the Systematic Review Toolbox, online and ML-based searches, and reference checking. Tools were included if they were actively maintained, accessible, and offered screening functionality beyond commonly used reference managers. RESULTS: Nine tools employing ML-based record prioritisation and relevance prediction were identified, of which 3 met all inclusion criteria and were evaluated in detail. These were applied to search results from 3 SLRs on randomised controlled trials (RCTs) and 1 that also included observational studies. Across reviews, substantial workload reductions were achieved (mean 77.8%, SD 12.8%) while consistently capturing over 95% of all relevant records. Additionally, ML-supported classification and filtering of RCTs prior to screening resulted in a mean reduction of abstracts to screen by 57.3% (SD 7.4%). CONCLUSIONS: Replication of manual title and abstract screening demonstrates the considerable potential of ML tools to support systematic reviews, while highlighting important limitations and pitfalls. Prospective evaluations will identify optimal strategies for integrating these tools into future review processes while maintaining high methodological standards.
Authors 18
-
Victoria Konzett corresponding
Affiliation as printed
Department of Medicine III, Division of Rheumatology, Medical University of Vienna, Vienna, Austria. Electronic address: victoria.konzett@meduniwien.ac.at
-
MRC Lifecourse Epidemiology Unit · University of Southampton
Affiliation as printed
MRC Lifecourse Epidemiology Centre Southampton, University of Southampton, Southampton, UK
-
Affiliation as printed
Department of Medicine III, Division of Rheumatology, Medical University of Vienna, Vienna, Austria
-
University Hospital Southampton NHS Foundation Trust · NIHR Southampton Biomedical Research Centre
Affiliation as printed
NIHR Southampton Clinical Research Facility, University Hospital Southampton, Southampton, UK
-
Affiliation as printed
Department of Medicine III, Division of Rheumatology, Medical University of Vienna, Vienna, Austria
-
Leiden University Medical Center
Affiliation as printed
Department of Rheumatology, Leiden University Medical Center, Leiden, The Netherlands
-
Oregon Health & Science University
Affiliation as printed
School of Public Health, Oregon Health & Science University, Portland, OR, USA
-
Affiliation as printed
Department of Rheumatology and Applied Immunology/Clinical Immunology, Saitama Medical University, Moroyama-machi, Iruma, Saitama, Japan
-
Affiliation as printed
Departments of Clinical Science and Community Health, University of Milan, Milan, Italy; Department of Rheumatology and Medical Sciences, ASST Pini-CTO, Milan, Italy
-
Affiliation as printed
Department of Rheumatology, University Hospitals Leuven, Skeletal Biology and Engineering Research Centre, KU Leuven, Leuven, Belgium
-
Affiliation as printed
Schulich School of Medicine, University of Western Ontario, London, ON, Canada
-
University of Manchester · NIHR Manchester Biomedical Research Centre
Affiliation as printed
Centre for Musculoskeletal Research, National Institute for Health and Care Research Manchester Biomedical Research Centre, Manchester University NHS Trust, The University of Manchester, Manchester, UK
-
Affiliation as printed
Clinical Research Facility, King's College Hospital, London, UK
-
Ludwig Boltzmann Institute for Arthritis and Rehabilitation · Medical University of Vienna
Affiliation as printed
Institute of Outcomes Research, Centre for Medical Data Science, Medical University of Vienna, Vienna, Austria; Ludwig Boltzmann Institute for Arthritis and Rehabilitation, Vienna, Austria
-
University of Southern Denmark
Affiliation as printed
Danish Hospital for Rheumatic Diseases, University Hospital of Southern Denmark, Sønderborg, Denmark; Department of Regional Health Research, University of Southern Denmark, Odense, Denmark
-
Leiden University Medical Center
Affiliation as printed
Directorate of Research & Valorisation, Leiden University Medical Center, Leiden, The Netherlands
-
Amsterdam UMC Location University of Amsterdam · Zuyderland Medisch Centrum
Affiliation as printed
Amsterdam Rheumatology Center, AMC, Amsterdam, The Netherlands; Rheumatology, Zuyderland Medical Center, Heerlen, The Netherlands
-
Stanford Medicine · Medical University of Vienna · Stanford University
Affiliation as printed
Department of Medicine III, Division of Rheumatology, Medical University of Vienna, Vienna, Austria; Division of Immunology and Rheumatology, Department of Medicine, Stanford University, School of Medicine, Stanford, CA, USA
Cited by 2 stored of 2
2 results
No patents citing this paper on Lens.org (checked 2026-10-11).