Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review

BMC Medical Research Methodology, vol. 22, pp. 101

Abstract

BACKGROUND: Describe and evaluate the methodological conduct of prognostic prediction models developed using machine learning methods in oncology. METHODS: We conducted a systematic review in MEDLINE and Embase between 01/01/2019 and 05/09/2019, for studies developing a prognostic prediction model using machine learning methods in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement, Prediction model Risk Of Bias ASsessment Tool (PROBAST) and CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) to assess the methodological conduct of included publications. Results were summarised by modelling type: regression-, non-regression-based and ensemble machine learning models. RESULTS: Sixty-two publications met inclusion criteria developing 152 models across all publications. Forty-two models were regression-based, 71 were non-regression-based and 39 were ensemble models. A median of 647 individuals (IQR: 203 to 4059) and 195 events (IQR: 38 to 1269) were used for model development, and 553 individuals (IQR: 69 to 3069) and 50 events (IQR: 17.5 to 326.5) for model validation. A higher number of events per predictor was used for developing regression-based models (median: 8, IQR: 7.1 to 23.5), compared to alternative machine learning (median: 3.4, IQR: 1.1 to 19.1) and ensemble models (median: 1.7, IQR: 1.1 to 6). Sample size was rarely justified (n = 5/62; 8%). Some or all continuous predictors were categorised before modelling in 24 studies (39%). 46% (n = 24/62) of models reporting predictor selection before modelling used univariable analyses, and common method across all modelling types. Ten out of 24 models for time-to-event outcomes accounted for censoring (42%). A split sample approach was the most popular method for internal validation (n = 25/62, 40%). Calibration was reported in 11 studies. Less than half of models were reported or made available. CONCLUSIONS: The methodological conduct of machine learning based clinical prediction models is poor. Guidance is urgently needed, with increased awareness and education of minimum prediction modelling standards. Particular focus is needed on sample size estimation, development and validation analysis methods, and ensuring the model is available for independent validation, to improve quality of machine learning based clinical prediction models.

Authors 12

1. Paula Dhiman corresponding

   University of Oxford · NIHR Oxford Biomedical Research Centre · Oxford University Hospitals NHS Trust

   Affiliation as printed

   Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK. paula.dhiman@csm.ox.ac.uk

   NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK. paula.dhiman@csm.ox.ac.uk

   Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK

2. Jie Ma

   University of Oxford

   Affiliation as printed

   Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK

3. Constanza L. Andaur Navarro

   Utrecht University · University Medical Center Utrecht

   Affiliation as printed

   Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

   Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

4. Benjamin Speich

   University of Basel · University Hospital of Basel · University of Oxford

   Affiliation as printed

   Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland

   Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK

5. Garrett Scott Bullock

   University of Oxford

   Affiliation as printed

   Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK

6. Johanna AAG Damen

   Utrecht University · University Medical Center Utrecht

   Affiliation as printed

   Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

   Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

7. Lotty Hooft

   Utrecht University · University Medical Center Utrecht

   Affiliation as printed

   Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

   Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

8. Shona Kirtley

   University of Oxford

   Affiliation as printed

   Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK

9. Richard D Riley

   Keele University

   Affiliation as printed

   Centre for Prognosis Research, School of Medicine, Keele University, Staffordshire, ST5 5BG, UK

10. Ben Van Calster Aachen Department of Biomedical Data Sciences

    Leiden University Medical Center · KU Leuven

    Affiliation as printed

    Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands

    Department of Development and Regeneration, KU Leuven, Leuven, Belgium

    EPI-centre, KU Leuven, Leuven, Belgium

11. Karel G.M. Moons

    Utrecht University · University Medical Center Utrecht

    Affiliation as printed

    Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

    Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

12. Professor Gary S. Collins

    University of Oxford · NIHR Oxford Biomedical Research Centre

    Affiliation as printed

    Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK

    NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK