Developing risk models for multicenter data using standard logistic regression produced suboptimal predictions: A simulation study

Biometrical Journal, vol. 62, pp. 932–944

Abstract

Although multicenter data are common, many prediction model studies ignore this during model development. The objective of this study is to evaluate the predictive performance of regression methods for developing clinical risk prediction models using multicenter data, and provide guidelines for practice. We compared the predictive performance of standard logistic regression, generalized estimating equations, random intercept logistic regression, and fixed effects logistic regression. First, we presented a case study on the diagnosis of ovarian cancer. Subsequently, a simulation study investigated the performance of the different models as a function of the amount of clustering, development sample size, distribution of center-specific intercepts, the presence of a center-predictor interaction, and the presence of a dependency between center effects and predictors. The results showed that when sample sizes were sufficiently large, conditional models yielded calibrated predictions, whereas marginal models yielded miscalibrated predictions. Small sample sizes led to overfitting and unreliable predictions. This miscalibration was worse with more heavily clustered data. Calibration of random intercept logistic regression was better than that of standard logistic regression even when center-specific intercepts were not normally distributed, a center-predictor interaction was present, center effects and predictors were dependent, or when the model was applied in a new center. Therefore, to make reliable predictions in a specific center, we recommend random intercept logistic regression.

Authors 4

1. Nora Falconieri

   KU Leuven

   Affiliation as printed

   Department of Development and Regeneration KU Leuven Leuven Belgium

   Department of Development and Regeneration, KU Leuven, Leuven, Belgium

2. Ben Van Calster Aachen Department of Biomedical Data Sciences Leiden University Medical Center Leiden The Netherlands Department of Biomedical Data Sciences

   Leiden University Medical Center · KU Leuven

   Affiliation as printed

   Department of Biomedical Data Sciences Leiden University Medical Center (LUMC) Leiden The Netherlands

   Department of Development and Regeneration KU Leuven Leuven Belgium

   Department of Biomedical Data Sciences, Leiden University Medical Center (LUMC), Leiden, The Netherlands

   Department of Development and Regeneration, KU Leuven, Leuven, Belgium

3. Dirk Timmerman

   Universitair Ziekenhuis Leuven · KU Leuven

   Affiliation as printed

   Department of Development and Regeneration KU Leuven Leuven Belgium

   Department of Obstetrics and Gynecology University Hospitals Leuven Leuven Belgium

   Department of Development and Regeneration, KU Leuven, Leuven, Belgium

   Department of Obstetrics and Gynecology, University Hospitals Leuven, Leuven, Belgium

4. Laure Wynants corresponding

   Maastricht University · Care and Public Health Research Institute · KU Leuven

   Affiliation as printed

   Department of Development and Regeneration KU Leuven Leuven Belgium

   Department of Epidemiology CAPHRI Care and Public Health Research Institute Maastricht University Maastricht The Netherlands

   Department of Development and Regeneration, KU Leuven, Leuven, Belgium

   Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Maastricht, The Netherlands