Minimum sample size for developing a multivariable prediction model using multinomial logistic regression

Statistical Methods in Medical Research, vol. 32, pp. 555–571

Abstract

Aims Multinomial logistic regression models allow one to predict the risk of a categorical outcome with > 2 categories. When developing such a model, researchers should ensure the number of participants ([Formula: see text]) is appropriate relative to the number of events ([Formula: see text]) and the number of predictor parameters ([Formula: see text]) for each category k. We propose three criteria to determine the minimum n required in light of existing criteria developed for binary outcomes. Proposed criteria The first criterion aims to minimise the model overfitting. The second aims to minimise the difference between the observed and adjusted [Formula: see text] Nagelkerke. The third criterion aims to ensure the overall risk is estimated precisely. For criterion (i), we show the sample size must be based on the anticipated Cox-snell [Formula: see text] of distinct ‘one-to-one’ logistic regression models corresponding to the sub-models of the multinomial logistic regression, rather than on the overall Cox-snell [Formula: see text] of the multinomial logistic regression. Evaluation of criteria We tested the performance of the proposed criteria (i) through a simulation study and found that it resulted in the desired level of overfitting. Criterion (ii) and (iii) were natural extensions from previously proposed criteria for binary outcomes and did not require evaluation through simulation. Summary We illustrated how to implement the sample size criteria through a worked example considering the development of a multinomial risk prediction model for tumour type when presented with an ovarian mass. Code is provided for the simulation and worked example. We will embed our proposed criteria within the pmsampsize R library and Stata modules.

Authors 7

1. Alexander Pate corresponding

   Manchester Academic Health Science Centre · University of Manchester

   Affiliation as printed

   Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK

2. Richard D Riley

   Keele University

   Affiliation as printed

   Centre for Prognosis Research, School of Medicine, Keele University, Staffordshire, UK

3. Professor Gary S. Collins

   John Radcliffe Hospital · University of Oxford · NIHR Oxford Biomedical Research Centre

   Affiliation as printed

   Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK

   NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford, UK

4. Maarten van Smeden Aachen Department of Clinical Epidemiology

   Utrecht University · Leiden University Medical Center · University Medical Center Utrecht

   Affiliation as printed

   Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, Netherlands

   Julius Center for Health Sciences, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands

5. Ben Van Calster Aachen Department of Biomedical Data Sciences

   Leiden University · Leiden University Medical Center · KU Leuven

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, Netherlands

   Department of Development and Regeneration, KU Leuven, Leuven, Belgium

   EPI-center, KU Leuven, Leuven, Belgium

6. Joie Ensor

   Keele University

   Affiliation as printed

   Centre for Prognosis Research, School of Medicine, Keele University, Staffordshire, UK

7. Glen Philip Martin

   Manchester Academic Health Science Centre · University of Manchester

   Affiliation as printed

   Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK