Regression shrinkage methods for clinical prediction models do not guarantee improved performance: Simulation study

Statistical Methods in Medical Research, vol. 29, pp. 3166–3178

Abstract

When developing risk prediction models on datasets with limited sample size, shrinkage methods are recommended. Earlier studies showed that shrinkage results in better predictive performance on average. This simulation study aimed to investigate the variability of regression shrinkage on predictive performance for a binary outcome. We compared standard maximum likelihood with the following shrinkage methods: uniform shrinkage (likelihood-based and bootstrap-based), penalized maximum likelihood (ridge) methods, LASSO logistic regression, adaptive LASSO, and Firth's correction. In the simulation study, we varied the number of predictors and their strength, the correlation between predictors, the event rate of the outcome, and the events per variable. In terms of results, we focused on the calibration slope. The slope indicates whether risk predictions are too extreme (slope < 1) or not extreme enough (slope > 1). The results can be summarized into three main findings. First, shrinkage improved calibration slopes on average. Second, the between-sample variability of calibration slopes was often increased relative to maximum likelihood. In contrast to other shrinkage approaches, Firth's correction had a small shrinkage effect but showed low variability. Third, the correlation between the estimated shrinkage and the optimal shrinkage to remove overfitting was typically negative, with Firth's correction as the exception. We conclude that, despite improved performance on average, shrinkage often worked poorly in individual datasets, in particular when it was most needed. The results imply that shrinkage methods do not solve problems associated with small sample size or low number of events per variable.

Authors 4

1. Ben Van Calster corresponding Aachen Department of Biomedical Data Sciences

   Leiden University Medical Center · KU Leuven

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, Netherlands

   Department of Development and Regeneration, KU Leuven, Leuven, Belgium

2. Maarten van Smeden Aachen Department of Biomedical Data Sciences Department of Clinical Epidemiology

   Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, Netherlands

   Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, Netherlands

3. B. De Cock

   KU Leuven

   Affiliation as printed

   Department of Accountancy, KU Leuven, Finance and Insurance, Leuven, Belgium

   Department of Development and Regeneration, KU Leuven, Leuven, Belgium

4. Ewout Willem Steyerberg Aachen Department of Biomedical Data Sciences

   Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, Netherlands