Automatic structuring of radiology reports with on-premise open-source large language models
European Radiology, vol. 35, pp. 2018–2029
Abstract
OBJECTIVES: Structured reporting enhances comparability, readability, and content detail. Large language models (LLMs) could convert free text into structured data without disrupting radiologists' reporting workflow. This study evaluated an on-premise, privacy-preserving LLM for automatically structuring free-text radiology reports. MATERIALS AND METHODS: We developed an approach to controlling the LLM output, ensuring the validity and completeness of structured reports produced by a locally hosted Llama-2-70B-chat model. A dataset with de-identified narrative chest radiograph (CXR) reports was compiled retrospectively. It included 202 English reports from a publicly available MIMIC-CXR dataset and 197 German reports from our university hospital. Senior radiologist prepared a detailed, fully structured reporting template with 48 question-answer pairs. All reports were independently structured by the LLM and two human readers. Bayesian inference (Markov chain Monte Carlo sampling) was used to estimate the distributions of Matthews correlation coefficient (MCC), with [-0.05, 0.05] as the region of practical equivalence (ROPE). RESULTS: The LLM generated valid structured reports in all cases, achieving an average MCC of 0.75 (94% HDI: 0.70-0.80) and F1 score of 0.70 (0.70-0.80) for English, and 0.66 (0.62-0.70) and 0.68 (0.64-0.72) for German reports, respectively. The MCC differences between LLM and humans were within ROPE for both languages: 0.01 (-0.05 to 0.07), 0.01 (-0.05 to 0.07) for English, and -0.01 (-0.07 to 0.05), 0.00 (-0.06 to 0.06) for German, indicating approximately comparable performance. CONCLUSION: Locally hosted, open-source LLMs can automatically structure free-text radiology reports with approximately human accuracy. However, the understanding of semantics varied across languages and imaging findings. KEY POINTS: Question Why has structured reporting not been widely adopted in radiology despite clear benefits and how can we improve this? Findings A locally hosted large language model successfully structured narrative reports, showing variation between languages and findings. Critical relevance Structured reporting provides many benefits, but its integration into the clinical routine is limited. Automating the extraction of structured information from radiology reports enables the capture of structured data while allowing the radiologist to maintain their reporting workflow.
Authors 12
-
Piotr Woźnicki corresponding
Affiliation as printed
Department of Diagnostic and Interventional Radiology, University Hospital Würzburg, Würzburg, Germany. woznicki_p@ukw.de
-
University of Würzburg · Universitätsklinikum Würzburg
Affiliation as printed
Department of Diagnostic and Interventional Radiology, University Hospital Würzburg, Würzburg, Germany
-
University of Würzburg · Universitätsklinikum Würzburg
Affiliation as printed
Department of Diagnostic and Interventional Radiology, University Hospital Würzburg, Würzburg, Germany
-
University of Würzburg · Universitätsklinikum Würzburg
Affiliation as printed
Department of Diagnostic and Interventional Radiology, University Hospital Würzburg, Würzburg, Germany
-
Affiliation as printed
Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
-
Heidelberg University · University Hospital Heidelberg · German Centre for Cardiovascular Research
Affiliation as printed
DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim, Heidelberg, Germany
Department of Internal Medicine III, Heidelberg University Hospital, Heidelberg, Germany
-
University Hospital Heidelberg · National Center for Tumor Diseases · University Hospital Carl Gustav Carus · Else Kröner Fresenius Center for Digital Health · Technische Universität Dresden
Affiliation as printed
Department of Internal Medicine I, University Hospital Carl Gustav Carus, Technical University Dresden, Dresden, Germany
Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
-
University Medical Center of the Johannes Gutenberg University Mainz
Affiliation as printed
Institute of Pathology, University Medical Center Mainz, Mainz, Germany
-
Affiliation as printed
Institute of Radiology and Nuclear Medicine, Cantonal Hospital Baselland, Liestal, Switzerland
-
University of Cologne · University Hospital Frankfurt
Affiliation as printed
Department of Diagnostic and Interventional Radiology, University of Cologne, Cologne, Germany
Department of Radiology, University Hospital of Frankfurt, Frankfurt, Germany
-
University of Würzburg · Universitätsklinikum Würzburg
Affiliation as printed
Department of Diagnostic and Interventional Radiology, University Hospital Würzburg, Würzburg, Germany
-
University of Würzburg · Universitätsklinikum Würzburg
Affiliation as printed
Department of Diagnostic and Interventional Radiology, University Hospital Würzburg, Würzburg, Germany
Cited by 41 stored of 42
No patents citing this paper on Lens.org (checked 2026-10-06).