Retrieval-augmented reasoning reshapes collective reliability in text-based radiology question answering - Data and code availability

Zenodo (CERN European Organization for Nuclear Research)

Abstract

This Zenodo archive accompanies the manuscript: Retrieval-augmented reasoning reshapes collective reliability in text-based radiology question answering The study evaluates how a fixed shared-evidence retrieval-augmented reasoning pipeline affects collective reliability under model variability in text-based radiology question answering. We compare zero-shot inference with a standardized shared-evidence retrieval-augmented reasoning condition across a heterogeneous panel of 34 large language models (LLMs). Models were evaluated on 169 expert-curated, publicly available radiology multiple-choice questions from two datasets: Benchmark-RadQA and Board-RadQA. Methodology The study focuses on reliability under model variability rather than only single-model accuracy. Each question was answered by all evaluated models under two main inference conditions: Zero-shot inferenceModels received only the question stem and answer options and were instructed to select the single best answer. Shared-evidence retrieval-augmented reasoningModels received the same question and answer options, preceded by an identical structured evidence report generated by a fixed retrieval-and-synthesis pipeline. Retrieval was restricted to Radiopaedia.org. The structured report was generated once per question and then provided unchanged to all evaluated models. The final answer was always selected by the evaluated model itself. This design isolates how different models use the same structured external evidence, rather than comparing their independent retrieval or tool-use abilities. The primary endpoints were: Inter-model decision stability, quantified by Shannon entropy of the answer distribution across models. Panel-based robustness of correctness, defined as the fraction of models in the 34-model panel that answered each question correctly. Secondary and exploratory analyses included: Majority-vote agreement strength. Consensus-correctness coupling. High-consensus, low-robustness failure cases. Output verbosity as a proxy for correctness. Exploratory clinical severity annotation of incorrect answer options. An additional three-level ablation analysis was conducted on a matched 15-model subset to disentangle retrieval access from structured evidence synthesis: Zero-shot inference. Standard retrieval-augmented inference using Radiopaedia-restricted retrieved passages. Shared-evidence retrieval-augmented reasoning using synthesized structured evidence reports. Quickstart Prepare the 169 text-only multiple-choice radiology questions from: Benchmark-RadQA, derived from RSNA-RadioQA and ExtendedQA items from the RadioRAG study. Board-RadQA, derived from the RaR study. Run zero-shot inference by providing each model with only: Question stem. Answer options. Run shared-evidence retrieval-augmented inference by launching the RaR orchestration pipeline: Retrieval is restricted to Radiopaedia.org. SearXNG is used as the metasearch engine. LangGraph and LangChain are used for orchestration. One structured evidence report is generated per question. The same report is provided to all evaluated models. Optionally run the standard RAG ablation on the matched 15-model subset: Provide top-ranked retrieved Radiopaedia passages. Do not use option-wise decomposition or synthesized structured evidence reports. Execute the statistical analysis scripts in the stability repository to compute: Entropy. Majority fraction. Panel-based robustness of correctness. Robustness-bin transitions. Consensus-correctness coupling. Verbosity-correctness associations. High-consensus, low-robustness failure cases. Clinical severity summaries. Environment setup Evaluations and statistical analyses were conducted between July 1, 2025, and February 25, 2026. The implementation was based on Python 3.11.7 and Microsoft Excel. Python was used for data processing, statistical analysis, visualization, tokenization, and reproducibility checks. Core Python packages included: NumPy v1.24.0 Pandas v2.0.0 SciPy v1.10.0 Matplotlib v3.7.0 Seaborn v0.12.0 Plotly v5.0.0 Kaleido v0.2.1 OpenPyXL v3.0.0 TikToken v0.5.0 The shared-evidence retrieval-augmented pipeline used: LangChain v0.3.25 LangGraph v0.4.1 OpenAI Python SDK v1.77.0 LangChain Open Deep Research SearXNG metasearch engine Docker v25.0.2 Locally hosted open-weight models were served using vLLM v0.9.0. Prerequisites A compatible Python environment can be created with Conda and pip: conda create -n shared-evidence-radqa python=3.11 -y conda activate shared-evidence-radqa python -m pip install --upgrade pip python -m pip install \ numpy==1.24.0 \ pandas==2.0.0 \ scipy==1.10.0 \ matplotlib==3.7.0 \ seaborn==0.12.0 \ plotly==5.0.0 \ kaleido==0.2.1 \ openpyxl==3.0.0 \ tiktoken==0.5.0 \ vllm==0.9.0 \ langchain==0.3.25 \ langgraph==0.4.1 \ openai==1.77.0 For the retrieval pipeline, a local SearXNG instance and Docker installation are required. Models evaluated We evaluated a fixed heterogeneous panel of 34 LLMs spanning proprietary, open-weight, general-purpose, reasoning-optimized, instruction-tuned, mixture-of-experts, and medically adapted model families. Full 34-model panel The evaluated models were: Claude-Sonnet-4.6 Gemini-3.1-Pro MiniMax-M2.5 GLM-5 LFM2.5-1.2B-Thinking Kimi-K2.5 Palmyra-X5 MiMo-V2-Flash Llama4-Scout-16E Llama3.3-8B Llama3.3-70B Llama3-Med42-8B Llama3-Med42-70B DeepSeek R1-70B DeepSeek-R1 DeepSeek-V3 GPT-5 GPT-5.2 o3 GPT-3.5-turbo GPT-4-turbo Mistral Large Ministral-8B MedGemma-4B-it MedGemma-27B-text-it Gemma-3-4B-it Gemma-3-27B-it Qwen3-8B Qwen3-235B Qwen2.5-0.5B Qwen2.5-3B Qwen2.5-7B Qwen2.5-14B Qwen2.5-70B Open-weight and locally hosted models Locally hosted models were run between July 1 and August 22, 2025. These models were served using vLLM v0.9.0. Tensor parallelism matched the number of GPUs per node; models under 3B parameters were served without tensor parallelism. Open-weight model sources: Qwen 2.5-0.5B: https://huggingface.co/Qwen/Qwen2.5-0.5B Qwen 2.5-3B: https://huggingface.co/Qwen/Qwen2.5-3B Qwen 2.5-7B: https://huggingface.co/Qwen/Qwen2.5-7B Qwen 2.5-14B: https://huggingface.co/Qwen/Qwen2.5-14B Qwen 2.5-70B: https://huggingface.co/Qwen/Qwen2.5-72B Qwen 3-8B: https://huggingface.co/Qwen/Qwen3-8B Qwen 3-235B: https://huggingface.co/Qwen/Qwen3-235B-A22B Llama 3.3-8B: https://huggingface.co/meta-llama/Meta-Llama-3-8B Llama 3.3-70B: https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct Llama 3-Med42-8B: https://huggingface.co/m42-health/Llama3-Med42-8B Llama 3-Med42-70B: https://huggingface.co/m42-health/Llama3-Med42-70B Llama4 Scout 16E: https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E Mistral Large: https://huggingface.co/mistralai/Mistral-Large-Instruct-2407 Ministral 8B: https://huggingface.co/mistralai/Ministral-8B-Instruct-2410 Gemma-3-4B-it: https://huggingface.co/google/gemma-3-4b-it Gemma-3-27B-it: https://huggingface.co/google/gemma-3-27b-it MedGemma-4B-it: https://huggingface.co/google/medgemma-4b-it MedGemma-27B-text-it: https://huggingface.co/google/medgemma-27b-text-it DeepSeek-V3: https://huggingface.co/deepseek-ai/DeepSeek-V3 DeepSeek-R1: https://huggingface.co/deepseek-ai/DeepSeek-R1 DeepSeek R1-70B: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B OpenAI proprietary models OpenAI proprietary models were accessed through the official OpenAI API. The versions used were: GPT-5.2: 2025-08-07 GPT-5: 2025-08-07 o3: 2025-04-16 GPT-4-turbo: 2024-04-09 GPT-3.5-turbo: 2024-01-25 Models accessed through OpenRouter The OpenRouter unified API was used to access the following models: google/gemini-3.1-pro-preview anthropic/claude-sonnet-4.6 z-ai/glm-5 LiquidAI/LFM2.5-1.2B-Thinking minimax/minimax-m2.5 moonshotai/kimi-k2.5 writer/palmyra-x5 xiaomi/mimo-v2-flash Ablation subset The three-level ablation compared zero-shot inference, standard RAG, and shared-evidence retrieval-augmented reasoning on a matched 15-model subset. The ablation subset included: Llama3.3-8B Llama3.3-70B Llama4-Scout-16E Mistral Large DeepSeek-R1 DeepSeek-V3 Qwen2.5-0.5B Qwen2.5-3B Qwen2.5-7B Qwen2.5-14B Qwen3-8B Gemma-3-4B-it Gemma-3-27B-it MedGemma-4B-it MedGemma-27B-text-it Models not included in the ablation were excluded because they were not evaluated under all three conditions or because the available checkpoint differed from the main-panel version. Code structure The codebase is split across two primary repositories: stability/https://github.com/minafarajiamiri/stabilityContains the evaluation and analysis pipeline used to compute inter-model decision stability, consensus, robustness, verbosity statistics, clinical severity summaries, and related metrics. RaR/https://github.com/sopajeta/RaRContains the retrieval-augmented orchestration pipeline used to generate structured evidence reports for the shared-evidence condition. Data availability All data analyzed in this study originate from publicly available, expert-curated radiology question-answering datasets. Benchmark-RadQA contains 104 questions and is derived from the RadioRAG study. It combines RSNA-RadioQA and ExtendedQA items. Questions are text-only and formatted as four-option multiple-choice questions. Board-RadQA contains 65 board-style radiology questions from the RaR study. Questions are text-only and formatted as five-option multiple-choice questions. No images, patient rec

Authors 12

1. Mina Farajiamiri
2. Jeta Sopa
3. Saba Afza
4. Lisa Adams
5. Felix Barajas Ordonez Aachen

   University Hospital Magdeburg · RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

   University Hospital Magdeburg

6. Tri-Thien Nguyen

   Friedrich-Alexander-Universität Erlangen-Nürnberg

   Affiliation as printed

   Friedrich-Alexander-Universität Erlangen-Nürnberg

7. Mahshad Lotfinia
   Affiliation as printed

   Generali Operations Service Platform

8. Sebastian Wind
9. Keno K. Bressem

   Deutsches Herzzentrum München

   Affiliation as printed

   German Heart Center Munich

10. Sven Nebelung Aachen

    Universitätsklinikum Aachen

    Affiliation as printed

    Universitätsklinikum Aachen

11. Daniel Truhn
12. Soroosh Tayebi Arasteh Aachen

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    University Hospital RWTH Aachen