Large Language Model-Based Uncertainty-Adjusted Label Extraction for Artificial Intelligence Model Development in Upper Extremity Radiography

arXiv (Cornell University)

Abstract

Objectives: To evaluate GPT-4o's ability to extract diagnostic labels (with uncertainty) from free-text radiology reports and to test how these labels affect multi-label image classification of musculoskeletal radiographs. Methods: This retrospective study included radiography series of the clavicle (n=1,170), elbow (n=3,755), and thumb (n=1,978). After anonymization, GPT-4o filled out structured templates by indicating imaging findings as present ("true"), absent ("false"), or "uncertain." To assess the impact of label uncertainty, "uncertain" labels of the training and validation sets were automatically reassigned to "true" (inclusive) or "false" (exclusive). Label-image-pairs were used for multi-label classification using ResNet50. Label extraction accuracy was manually verified on internal (clavicle: n=233, elbow: n=745, thumb: n=393) and external test sets (n=300 for each). Performance was assessed using macro-averaged receiver operating characteristic (ROC) area under the curve (AUC), precision recall curves, sensitivity, specificity, and accuracy. AUCs were compared with the DeLong test. Results: Automatic extraction was correct in 98.6% (60,618 of 61,488) of labels in the test sets. Across anatomic regions, label-based model training yielded competitive performance measured by macro-averaged AUC values for inclusive (e.g., elbow: AUC=0.80 [range, 0.62-0.87]) and exclusive models (elbow: AUC=0.80 [range, 0.61-0.88]). Models generalized well on external datasets (elbow [inclusive]: AUC=0.79 [range, 0.61-0.87]; elbow [exclusive]: AUC=0.79 [range, 0.63-0.89]). No significant differences were observed across labeling strategies or datasets (p>=0.15). Conclusion: GPT-4o extracted labels from radiologic reports to train competitive multi-label classification models with high accuracy. Detected uncertainty in the radiologic reports did not influence the performance of these models.

Authors 6

1. Anne-Sophie Caselitz Aachen Department of Diagnostic and Interventional Radiology Lab for Artificial Intelligence in Medicine

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Department of Diagnostic and Interventional Radiology , University Hospital Aachen , Aachen , Germany

   Lab for Artificial Intelligence in Medicine , Department of Diagnostic and Interventional Radiology , University Hospital Aachen , Aachen , Germany

2. Thomas Dratsch

   University of Cologne · University Hospital Cologne

   Affiliation as printed

   Institute for Diagnostic and Interventional Radiology , Faculty of Medicine and University Hospital Cologne , University of Cologne , Cologne , Germany

3. Daniel Pinto dos Santos
4. Christiane Katharina Kuhl Aachen Department of Diagnostic and Interventional Radiology

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Department of Diagnostic and Interventional Radiology , University Hospital Aachen , Aachen , Germany

5. Daniel Truhn Aachen Department of Diagnostic and Interventional Radiology Lab for Artificial Intelligence in Medicine

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Department of Diagnostic and Interventional Radiology , University Hospital Aachen , Aachen , Germany

   Lab for Artificial Intelligence in Medicine , Department of Diagnostic and Interventional Radiology , University Hospital Aachen , Aachen , Germany

6. Sven Nebelung Aachen Department of Diagnostic and Interventional Radiology Lab for Artificial Intelligence in Medicine

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Department of Diagnostic and Interventional Radiology , University Hospital Aachen , Aachen , Germany

   Lab for Artificial Intelligence in Medicine , Department of Diagnostic and Interventional Radiology , University Hospital Aachen , Aachen , Germany