Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance

The Lancet Digital Health, vol. 7, pp. 100916

Abstract

Numerous measures have been proposed to illustrate the performance of predictive artificial intelligence (AI) models. Selecting appropriate performance measures is essential for predictive AI models intended for use in medical practice. Poorly performing models are misleading and may lead to wrong clinical decisions that can be detrimental to patients and increase financial costs. In this Viewpoint, we assess the merits of classic and contemporary performance measures when validating predictive AI models for medical practice, focusing on models that estimate probabilities for a binary outcome. We discuss 32 performance measures covering five performance domains (discrimination, calibration, overall performance, classification, and clinical utility) along with corresponding graphical assessments. The first four domains address statistical performance, whereas the fifth domain covers decision-analytical performance. We discuss two key characteristics when selecting a performance measure and explain why these characteristics are important: (1) whether the measure's expected value is optimised when calculated using the correct probabilities (ie, whether it is a proper measure) and (2) whether the measure solely reflects statistical performance or decision-analytical performance by properly accounting for misclassification costs. 17 measures showed both characteristics, 14 showed one, and one (F1 score) showed neither. All classification measures were improper for clinically relevant decision thresholds other than when the threshold was 0·5 or equal to the true prevalence. We illustrate these measures and characteristics using the ADNEX model which predicts the probability of malignancy in women with an ovarian tumour. We recommend the following measures and plots as essential to report: area under the receiver operating characteristic curve, calibration plot, a clinical utility measure such as net benefit with decision curve analysis, and a plot showing probability distributions by outcome category.

Authors 14

1. Ben Van Calster Aachen

   Utrecht University · Leiden University Medical Center · University Medical Center Utrecht · KU Leuven

   Affiliation as printed

   Department of Development and Regeneration, KU Leuven, Leuven, Belgium; Leuven Unit for Health Technology Assessment Research (LUHTAR), KU Leuven, Leuven, Belgium; Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, Netherlands; Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands

2. Professor Gary S. Collins

   University of Birmingham

   Affiliation as printed

   Department of Applied Health Sciences, School of Health Sciences, College and Medicine and Health, University of Birmingham, Birmingham, UK

3. Andrew Julian Vickers

   Memorial Sloan Kettering Cancer Center

   Affiliation as printed

   Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, NY, USA

4. Laure Wynants

   Maastricht University · Care and Public Health Research Institute · KU Leuven

   Affiliation as printed

   Department of Development and Regeneration, KU Leuven, Leuven, Belgium; Leuven Unit for Health Technology Assessment Research (LUHTAR), KU Leuven, Leuven, Belgium; Department of Epidemiology, Care and Public Health Research Institute (CAPHRI), Maastricht University, Maastricht, Netherlands

5. Kathleen F. Kerr

   University of Washington

   Affiliation as printed

   Department of Biostatistics, University of Washington School of Public Health, Seattle, WA, USA

6. Lasai Barreñada

   KU Leuven

   Affiliation as printed

   Department of Development and Regeneration, KU Leuven, Leuven, Belgium; Leuven Unit for Health Technology Assessment Research (LUHTAR), KU Leuven, Leuven, Belgium

7. Gaël Varoquaux

   Institut national de recherche en sciences et technologies du numérique · Centre Inria de Saclay

   Affiliation as printed

   Parietal project team, INRIA Saclay-Île de France, Palaiseau, France

8. Karandeep Singh

   University of California San Diego

   Affiliation as printed

   Division of Biomedical Informatics, Department of Medicine, University of California, San Diego, CA, USA

9. Karel G. M. Moons

   Utrecht University · University Medical Center Utrecht

   Affiliation as printed

   Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands

10. Tina Hernandez‐Boussard

    Stanford Medicine · Stanford University

    Affiliation as printed

    Department of Medicine (Biomedical Informatics), Stanford University, Stanford, CA, USA; Department of Biomedical Data Science, Stanford University, Stanford, CA, USA

11. Dirk Timmerman

    KU Leuven

    Affiliation as printed

    Department of Development and Regeneration, KU Leuven, Leuven, Belgium; Department of Obstetrics and Gynecology, University Hospitals Leuven, Leuven, Belgium

12. David J. McLernon

    University of Aberdeen

    Affiliation as printed

    Institute of Applied Health Sciences, University of Aberdeen, Aberdeen, UK

13. Maarten van Smeden

    Utrecht University · University Medical Center Utrecht

    Affiliation as printed

    Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands

14. Ewout Willem Steyerberg corresponding Aachen

    Utrecht University · Leiden University Medical Center · University Medical Center Utrecht

    Affiliation as printed

    Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, Netherlands; Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands. Electronic address: e.w.steyerberg@umcutrecht.nl