Retrieval-augmented reasoning reshapes collective reliability in text-based radiology question answering

Patterns, vol. 7, pp. 101639

Abstract

Retrieval-augmented reasoning pipelines are increasingly used to structure how large language models (LLMs) incorporate external evidence in clinical question answering, but their reliability under model variability remains unclear. We evaluated 34 LLMs on 169 expert-curated, text-only, multiple-choice radiology questions, comparing zero-shot inference with a fixed shared-evidence condition in which all models received identical structured evidence reports before answer selection. Primary endpoints were inter-model decision dispersion and panel-based robustness of correctness. Shared evidence reduced decision dispersion (median entropy 0.48→0.13; p = 5.6 × 10 −9 ) and increased robustness of correctness (mean 0.74→0.81; p = 5.6 × 10 −9 ). Majority consensus increased, and consensus strength remained strongly correlated with robustness. However, high agreement did not guarantee correctness, and rare high-consensus failures persisted. These findings show that shared evidence can improve reproducibility under model variability, but accuracy and agreement alone are insufficient to characterize reliability.

Authors 12

1. Mina Farajiamiri Aachen Lab for AI in Medicine

   RWTH Aachen University

   Affiliation as printed

   Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany

2. Jeta Sopa

   Friedrich-Alexander-Universität Erlangen-Nürnberg

   Affiliation as printed

   Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

3. Saba Afza

   Friedrich-Alexander-Universität Erlangen-Nürnberg

   Affiliation as printed

   Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

4. Lisa C. Adams

   TUM Klinikum · Technical University of Munich

   Affiliation as printed

   Department of Diagnostic and Interventional Radiology, TUM University Klinikum, School of Medicine and Health, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany

5. Felix Barajas Ordonez Aachen Department of Diagnostic and Interventional Radiology Lab for AI in Medicine

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany

   Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany

6. Tri-Thien Nguyen

   Friedrich-Alexander-Universität Erlangen-Nürnberg · Universitätsklinikum Erlangen

   Affiliation as printed

   Institute of Radiology, University Hospital Erlangen, Erlangen, Germany

   Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

7. Mahshad Lotfinia Aachen Department of Diagnostic and Interventional Radiology Lab for AI in Medicine

   Friedrich-Alexander-Universität Erlangen-Nürnberg · Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany

   Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany

   Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

8. Sebastian Wind

   Friedrich-Alexander-Universität Erlangen-Nürnberg

   Affiliation as printed

   Erlangen National High Performance Computing Center, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

   Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

9. Keno K. Bressem

   TUM Klinikum · Deutsches Herzzentrum München · Technical University of Munich

   Affiliation as printed

   Department of Cardiovascular Radiology and Nuclear Medicine, TUM University Clinic, School of Medicine and Health, German Heart Center, Technical University of Munich, Munich, Germany

   Department of Diagnostic and Interventional Radiology, TUM University Klinikum, School of Medicine and Health, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany

10. Sven Nebelung Aachen Department of Diagnostic and Interventional Radiology Lab for AI in Medicine

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany

    Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany

11. Daniel Truhn Aachen Department of Diagnostic and Interventional Radiology Lab for AI in Medicine

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany

    Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany

12. Soroosh Tayebi Arasteh corresponding Aachen Department of Diagnostic and Interventional Radiology Lab for AI in Medicine

    Universitätsklinikum Aachen · Stanford Medicine · RWTH Aachen University · Stanford University

    Affiliation as printed

    Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany

    Department of Radiology, Stanford University, Stanford, CA, USA

    Department of Urology, Stanford University, Stanford, CA, USA

    Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany