Developing Clinical Prediction Models Using Primary Care Electronic Health Record Data: The Impact of Data Preparation Choices on Model Performance

Frontiers in Epidemiology, vol. 2, pp. 871630

Abstract

Objective: To quantify prediction model performance in relation to data preparation choices when using electronic health records (EHR). Study Design and Setting: Cox proportional hazards models were developed for predicting the first-ever main adverse cardiovascular events using Dutch primary care EHR data. The reference model was based on a 1-year run-in period, cardiovascular events were defined based on both EHR diagnosis and medication codes, and missing values were multiply imputed. We compared data preparation choices based on (i) length of the run-in period (2- or 3-year run-in); (ii) outcome definition (EHR diagnosis codes or medication codes only); and (iii) methods addressing missing values (mean imputation or complete case analysis) by making variations on the derivation set and testing their impact in a validation set. Results: We included 89,491 patients in whom 6,736 first-ever main adverse cardiovascular events occurred during a median follow-up of 8 years. Outcome definition based only on diagnosis codes led to a systematic underestimation of risk (calibration curve intercept: 0.84; 95% CI: 0.83-0.84), while complete case analysis led to overestimation (calibration curve intercept: -0.52; 95% CI: -0.53 to -0.51). Differences in the length of the run-in period showed no relevant impact on calibration and discrimination. Conclusion: Data preparation choices regarding outcome definition or methods to address missing values can have a substantial impact on the calibration of predictions, hampering reliable clinical decision support. This study further illustrates the urgency of transparent reporting of modeling choices in an EHR data setting.

Authors 10

1. Hendrikus J. A. van Os corresponding Aachen Department of Neurology, Leiden University Medical Hospital Department of Public Health and Primary Care (PHEG) National eHealth Living Lab

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Neurology, Leiden University Medical Hospital, Leiden, Netherlands

   Department of Public Health & Primary Care, Leiden University Medical Hospital, Leiden, Netherlands

   National eHealth Living Lab, Leiden University Medical Hospital, Leiden, Netherlands

2. Jos P. Kanning

   University Medical Center Utrecht

   Affiliation as printed

   Department of Neurology, University Medical Center Utrecht, Utrecht, Netherlands

3. Marieke J.H. Wermer Aachen Department of Neurology, Leiden University Medical Hospital

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Neurology, Leiden University Medical Hospital, Leiden, Netherlands

4. Niels Henrik Chavannes Aachen Department of Public Health and Primary Care (PHEG) National eHealth Living Lab

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Public Health & Primary Care, Leiden University Medical Hospital, Leiden, Netherlands

   National eHealth Living Lab, Leiden University Medical Hospital, Leiden, Netherlands

5. Mattijs Everard Numans Aachen Department of Public Health and Primary Care (PHEG)

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Public Health & Primary Care, Leiden University Medical Hospital, Leiden, Netherlands

6. Ynte M. Ruigrok

   University Medical Center Utrecht

   Affiliation as printed

   Department of Neurology, University Medical Center Utrecht, Utrecht, Netherlands

7. Erik W. van Zwet Aachen Department of Biomedical Data Sciences

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Hospital, Leiden, Netherlands

8. Hein Putter Aachen Department of Biomedical Data Sciences

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Hospital, Leiden, Netherlands

9. Ewout Willem Steyerberg Aachen Department of Biomedical Data Sciences

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Hospital, Leiden, Netherlands

10. Rolf H. H. Groenwold Aachen Department of Biomedical Data Sciences Department of Clinical Epidemiology

    Leiden University · Leiden University Medical Center

    Affiliation as printed

    Department of Biomedical Data Sciences, Leiden University Medical Hospital, Leiden, Netherlands

    Department of Clinical Epidemiology, Leiden University Medical Hospital, Leiden, Netherlands