Using ensemble radiomic models to identify uncertainty in lung nodule classifications

European Journal of Radiology Artificial Intelligence, vol. 4, pp. 100047

Abstract

Aims: The performance of machine learning models for tasks such as tumor malignancy classification depends heavily on representativeness of the training data. Under-represented cases can lead to less reliable and error-prone predictions. To increase trust, it is crucial to determine when a prediction should be considered reliable. Methods Radiomic features were extracted from 918 lung nodules in the LIDC dataset. Images were discretized with a fixed bin size of 64 before texture analysis. Features highly correlated with nodule volume or each other were excluded, retaining 80 features. Features were normalized using min-max scaling. The dataset was split into 80% training and 20% testing data, where the model was applied and the uncertainty analysis was performed. An ensemble model was trained using 1000 bootstraps of 90% of training data, Random Forest, Support Vector Machine, Gradient Boosting, AdaBoost, and ExtraTree classifiers were trained, each of them on 200 bootstraps. Aleatoric and epistemic uncertainties were calculated, and percentile-based thresholds were used to identify uncertain predictions. Evaluation metrics assessed overall performance and performance on certain versus uncertain cases. Results: The ensemble model achieved 86% accuracy. Depending on the uncertainty threshold, 11%-32% of test cases were flagged as uncertain. The proportion of certain incorrect predictions decreased from 8.1% at the highest threshold to 2.1% at the lowest. Thresholds near the 80th or 85th percentile balanced reducing manual review workload and identifying errors effectively. Conclusion: The ensemble model successfully identifies most incorrect predictions as uncertain. Future work will explore the current missing validation on external datasets.

Authors 6

1. Elisabeth A. G. Pfaehler corresponding

   Forschungszentrum Jülich

   Affiliation as printed

   Forschungszentrum Jülich GmbH, Institute of Advanced Simulations, IAS-8: Data Analytics and Machine Learning, Jülich, 52428, Germany

2. Lena Krieger

   Forschungszentrum Jülich

   Affiliation as printed

   Forschungszentrum Jülich GmbH, Institute of Advanced Simulations, IAS-8: Data Analytics and Machine Learning, Jülich, 52428, Germany

3. Ira Assent

   Forschungszentrum Jülich · Aarhus University

   Affiliation as printed

   Aarhus University, Department of Computer Science, Aarhus N, 8200, Denmark

   Forschungszentrum Jülich GmbH, Institute of Advanced Simulations, IAS-8: Data Analytics and Machine Learning, Jülich, 52428, Germany

4. Sven Nebelung Aachen Department of Diagnostic and Interventional Radiology

   Universitätsklinikum Aachen

   Affiliation as printed

   University Hospital Aachen, Department of Diagnostic and Interventional Radiology, Aachen, 52074, Germany

5. Alex Zwanenburg

   Helmholtz-Zentrum Dresden-Rossendorf · OncoRay · University Hospital Carl Gustav Carus · Nationales Centrum für Tumorerkrankungen Dresden · Technische Universität Dresden

   Affiliation as printed

   National Center for Tumor Diseases (NCT), NCT/UCC Dresden, a partnership between DKFZ, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, and Helmholtz-Zentrum Dresden-Rossendorf (HZDR), Germany

   OncoRay – National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Helmholtz-Zentrum Dresden-Rossendorf, Dresden, Germany

6. Daniel Truhn Aachen Department of Diagnostic and Interventional Radiology

   Universitätsklinikum Aachen

   Affiliation as printed

   University Hospital Aachen, Department of Diagnostic and Interventional Radiology, Aachen, 52074, Germany