Guidelines for Reporting Studies on Large Language Models in Radiology: An International Delphi Expert Survey
Radiology, vol. 318, pp. e250913
Abstract
Large language models (LLMs) have transformative potential in radiology, including textual summaries, diagnostic decision support, proofreading, and image analysis. However, the rapid increase in studies investigating these models, along with the lack of standardized LLM-specific reporting practices, affects reproducibility, reliability, and clinical applicability. To address this, reporting guidelines for LLM studies in radiology were developed using a two-step process. First, a systematic review of LLM studies in radiology was conducted across PubMed, IEEE Xplore, and the ACM Digital Library, covering publications between May 2023 and March 2024. Of 511 screened studies, 57 were included to identify relevant aspects for the guidelines. Then, in a Delphi process, 20 international experts developed the final list of items for inclusion. Items consented as relevant were summarized into a structured checklist containing 32 items across six key categories: general information and data input; prompting and fine-tuning; performance metrics; ethics and data transparency; implementation, risks, and limitations; and further/optional aspects. The final FLAIR (Framework for LLM Assessment in Radiology) checklist aims to standardize reporting of LLM studies in radiology, fostering transparency, reproducibility, comparability, and clinical applicability to enhance clinical translation and patient care. © The Author(s) 2026. Published by the Radiological Society of North America under a CC BY 4.0 license.
Authors 28
-
Jonathan Kottlors corresponding
University of Cologne · University Hospital Cologne
Affiliation as printed
Institute of Diagnostic and Interventional Radiology, Faculty of Medicine and University Hospital Cologne, Kerpener Str 62, 50937, Cologne, Germany
-
Andra-Iza Iuga corresponding
University of Cologne · University Hospital Cologne
Affiliation as printed
Institute of Diagnostic and Interventional Radiology, Faculty of Medicine and University Hospital Cologne, Kerpener Str 62, 50937, Cologne, Germany
-
University of Zurich · University Hospital Zurich
Affiliation as printed
Diagnostic and Interventional Radiology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
-
TUM Klinikum · Deutsches Herzzentrum München · Technical University of Munich
Affiliation as printed
School of Medicine and Health, Institute for Cardiovascular Radiology and Nuclear Medicine, German Heart Center Munich, TUM University Hospital, Technical University of Munich, Munich, Germany
School of Medicine and Health, Institute for Diagnostic and Interventional Radiology, Klinikum Rechts der Isar, TUM University Hospital, Technical University of Munich, Munich, Germany
-
Heidelberg University · University Hospital Heidelberg · National Center for Tumor Diseases · University Hospital Carl Gustav Carus · Else Kröner Fresenius Center for Digital Health · Technische Universität Dresden
Affiliation as printed
Department of Medicine I, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
Else Kroener Fresenius Center for Digital Health, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
-
Affiliation as printed
Department of Radiology, New York University Grossman School of Medicine, New York, NY
-
Lahey Hospital and Medical Center · Lahey Medical Center
Affiliation as printed
Department of Radiology, Lahey Hospital & Medical Center, Burlington, Mass
-
Sun Yat-sen University · The First Affiliated Hospital, Sun Yat-sen University
Affiliation as printed
Department of Medical Ultrasonics, Ultrasomics Artificial Intelligence X-Laboratory, Institute of Diagnostic and Interventional Ultrasound, First Affiliated Hospital of Sun Yat-sen University, Guangzhou, China
-
Affiliation as printed
School of Computing, The University of Georgia, Athens, Ga
-
Affiliation as printed
Department of Radiology, St. Nikolaus Hospital, Eupen, Belgium
Faculty of Medicine and Health Sciences, Ghent University, Ghent, Belgium
-
University of Cologne · University Hospital Cologne
Affiliation as printed
Institute of Diagnostic and Interventional Radiology, Faculty of Medicine and University Hospital Cologne, Kerpener Str 62, 50937, Cologne, Germany
-
Affiliation as printed
Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany
-
University of Cologne · University Hospital Cologne
Affiliation as printed
Institute of Diagnostic and Interventional Radiology, Faculty of Medicine and University Hospital Cologne, Kerpener Str 62, 50937, Cologne, Germany
-
Affiliation as printed
Department of Biomedical Informatics, Harvard Medical School, Boston, Mass
-
University of California, Los Angeles
Affiliation as printed
Department of Radiological Sciences, University of California, Los Angeles, Calif
-
Heidelberg University · University Hospital Heidelberg
Affiliation as printed
Clinic for Diagnostic and Interventional Radiology, University Hospital Heidelberg, Heidelberg, Germany
-
Affiliation as printed
Department of Psychosomatic Medicine and Psychotherapy, Center for Mental Health, Faculty of Medicine, University of Freiburg, Freiburg, Germany
-
Stanford Medicine · Stanford University
Affiliation as printed
Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, Calif
Department of Biomedical Data Science, Stanford University, Stanford, Calif
Department of Radiology, Stanford University, Stanford, Calif
-
Massachusetts General Hospital
Affiliation as printed
Department of Radiology, Massachusetts General Hospital, Boston, Mass
-
University College London · University of Glasgow
Affiliation as printed
Institute of Health Informatics, University College London, London, United Kingdom
School of Health and Wellbeing, University of Glasgow, Glasgow, United Kingdom
-
Affiliation as printed
Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany
Institute of Interventional and Diagnostic Radiology and Neuroradiology, University Hospital Essen, Essen, Germany
-
Chinese University of Hong Kong · Chinese University of Hong Kong, Shenzhen
Affiliation as printed
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China
-
University of Cologne · University Hospital Cologne
Affiliation as printed
Institute of Diagnostic and Interventional Radiology, Faculty of Medicine and University Hospital Cologne, Kerpener Str 62, 50937, Cologne, Germany
-
University of Cologne · University Hospital Cologne
Affiliation as printed
Institute of Diagnostic and Interventional Radiology, Faculty of Medicine and University Hospital Cologne, Kerpener Str 62, 50937, Cologne, Germany
-
University of Cologne · University Hospital Cologne
Affiliation as printed
Institute of Diagnostic and Interventional Radiology, Faculty of Medicine and University Hospital Cologne, Kerpener Str 62, 50937, Cologne, Germany
-
University of Cologne · University Hospital Cologne
Affiliation as printed
Institute of Diagnostic and Interventional Radiology, Faculty of Medicine and University Hospital Cologne, Kerpener Str 62, 50937, Cologne, Germany
-
Affiliation as printed
Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
-
Simon Lennartz corresponding
University of Cologne · University Hospital Cologne
Affiliation as printed
Institute of Diagnostic and Interventional Radiology, Faculty of Medicine and University Hospital Cologne, Kerpener Str 62, 50937, Cologne, Germany
Cited by 7 stored of 8
7 results
No patents citing this paper on Lens.org (checked 2026-10-06).