Recent criteria and simple rules agreed often on required sample size for developed clinical prediction models

Journal of Clinical Epidemiology, vol. 199, pp. 112461

Abstract

OBJECTIVES: Adequate sample size is essential to the development of new prediction models with binary outcomes. We aim to relate recent approaches to traditional rules of thumb ('simple rules') and assess agreement and differences in judging adequacy of sample size for developed prediction models. STUDY DESIGN AND SETTING: ) ≤ 0.05'); and 3) precise estimation of the overall event rate (margin of error <0.05). We use statistical theory to compare simple rules to these three recent criteria. Furthermore, we compare sample size assessments for 299 published COVID-19 prediction models. RESULTS: At a 10% event rate, criterion 1 (limited overfitting) corresponded to the classic EPP>10 rule for an area under the receiver operating characteristic curve (c) of 0.767 and EPP>20 for c = 0.694. Criterion 2 implied EPP>4 (largely irrespective of c), and criterion 3 was equivalent to E > 14 at a 10% event rate. We classified largely the same COVID-19 prediction models as adequate or inadequate for sample size according to recent criteria (maximum of three) and a simple combination rule ('E = 100 plus 10∗p', or max['E = 100, 10∗p']). CONCLUSION: Simple rules have direct relations with recent criteria for sample size calculations to limit overfitting and guarantee reliability of predictions. Recent criteria are essential to inform prediction modeling efforts, while simple rules may often be sufficient to help appraise the quality of already developed regression-based prediction models. PLAIN LANGUAGE SUMMARY: ≤ 0.05), and (3) that the overall event rate is precisely estimated. Using theory and examples from 299 published COVID-19 models, we found that the simple rules often map closely to the newer criteria. Overall, we conclude that while the new criteria are better for planning prospective studies and developing a prediction model, simple rules remain useful and are usually adequate for judging whether already developed prediction models had enough data.

Authors 5

1. Ewout Willem Steyerberg Aachen

   Leiden University Medical Center · University Medical Center Utrecht

   Affiliation as printed

   Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands; Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands. Electronic address: e.w.steyerberg@umcutrecht.nl

2. Toby Hackmann Aachen Department of Biomedical Data Sciences

   Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands

3. Ben Van Calster

   University Medical Center Utrecht · KU Leuven

   Affiliation as printed

   Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands; Department of Development and Regeneration, KU Leuven, Leuven, Belgium

4. Maarten van Smeden

   Utrecht University · University Medical Center Utrecht

   Affiliation as printed

   Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands

5. Laure Wynants

   Maastricht University · Care and Public Health Research Institute · KU Leuven

   Affiliation as printed

   Department of Development and Regeneration, KU Leuven, Leuven, Belgium; Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Maastricht, The Netherlands