When accurate prediction models yield harmful self-fulfilling prophecies

Patterns, vol. 6, pp. 101229

Abstract

Prediction models are popular in medical research and practice. Many expect that by predicting patient-specific outcomes, these models have the potential to inform treatment decisions, and they are frequently lauded as instruments for personalized, data-driven healthcare. We show, however, that using prediction models for decision-making can lead to harm, even when the predictions exhibit good discrimination after deployment. These models are harmful self-fulfilling prophecies: their deployment harms a group of patients, but the worse outcome of these patients does not diminish the discrimination of the model. Our main result is a formal characterization of a set of such prediction models. Next, we show that models that are well calibrated before and after deployment are useless for decision-making, as they make no change in the data distribution. These results call for a reconsideration of standard practices for validation and deployment of prediction models that are used in medical decisions.

Authors 5

1. Wouter A. C. van Amsterdam

   Utrecht University · University Medical Center Utrecht

   Affiliation as printed

   Department of Data Science and Biostatistics, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, University of Utrecht, Heidelberglaan 100, 3584 CX Utrecht, the Netherlands

2. Nan van Geloven Aachen Department of Biomedical Data Sciences

   Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Center, Albinusdreef 2, 2333 ZA Leiden, the Netherlands

3. Jesse H. Krijthe

   Delft University of Technology

   Affiliation as printed

   Pattern Recognition & Bioinformatics, Delft University of Technology, Mekelweg 5, 2628 CD Delft, the Netherlands

4. Rajesh Ranganath

   Courant Institute of Mathematical Sciences · New York University

   Affiliation as printed

   Courant Institute of Mathematical Science, Department of Computer Science, Center for Data Science, New York University, 251 Mercer St., New York, NY 10012, USA

5. Giovanni Cinà

   Amsterdam UMC Location University of Amsterdam · Amsterdam University Medical Centers · University of Amsterdam

   Affiliation as printed

   Department of Medical Informatics, Amsterdam University Medical Center, Meibergdreef 9, 1105 AZ Amsterdam, the Netherlands

   Institute for Logic, Language and Computation, University of Amsterdam, Amsterdam, the Netherlands

   Pacmed, Amsterdam, the Netherlands