Quantitative prediction error analysis to investigate predictive performance under predictor measurement heterogeneity at model implementation

Diagnostic and Prognostic Research, vol. 6, pp. 7

Abstract

BACKGROUND: When a predictor variable is measured in similar ways at the derivation and validation setting of a prognostic prediction model, yet both differ from the intended use of the model in practice (i.e., "predictor measurement heterogeneity"), performance of the model at implementation needs to be inferred. This study proposed an analysis to quantify the impact of anticipated predictor measurement heterogeneity. METHODS: A simulation study was conducted to assess the impact of predictor measurement heterogeneity across validation and implementation setting in time-to-event outcome data. The use of the quantitative prediction error analysis was illustrated using an example of predicting the 6-year risk of developing type 2 diabetes with heterogeneity in measurement of the predictor body mass index. RESULTS: In the simulation study, calibration-in-the-large of prediction models was poor and overall accuracy was reduced in all scenarios of predictor measurement heterogeneity. Model discrimination decreased with increasing random predictor measurement heterogeneity. CONCLUSIONS: Heterogeneity of predictor measurements across settings of validation and implementation reduced predictive performance at implementation of prognostic models with a time-to-event outcome. When validating a prognostic model, the targeted clinical setting needs to be considered and analyses can be conducted to quantify the impact of anticipated predictor measurement heterogeneity on model performance at implementation.

Authors 3

1. Kim Luijken corresponding Aachen Department of Clinical Epidemiology

   Leiden University Medical Center

   Affiliation as printed

   Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, the Netherlands. k.luijken@lumc.nl

   Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, the Netherlands

2. Jia Tao Song Aachen Department of Clinical Epidemiology

   Leiden University Medical Center

   Affiliation as printed

   Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, the Netherlands

3. Rolf H. H. Groenwold Aachen Department of Biomedical Data Sciences Department of Clinical Epidemiology

   Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands

   Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, the Netherlands