Prediction models with survival data: a comparison between machine learning and the Cox proportional hazards model

medRxiv

Abstract

Recent years have seen increased interest in using machine learning (ML) methods for survival prediction, chiefly using big datasets with mixed datatypes and/or many predictors Model comparisons have frequently been limited to performance measure evaluation, with the chosen measure often suboptimal for assessing survival predictive performance. We investigated ML model performance in an application to osteosarcoma data from the EURAMOS-1 clinical trial ( NCT00134030 ). We compared the performance of survival neural networks (SNN), random survival forests (RSF) and the Cox proportional hazards model. Three performance measures suitable for assessing survival model predictive performance were considered: the C-index, and the time-dependent Brier and Kullback-Leibler scores. Comparisons were also made on predictor importance and patient-specific survival predictions. Additionally, the effect of ML model hyper-parameters on performance was investigated. All three models had comparable performance as assessed by the C-index and Brier and Kullback-Leibler scores, with the Cox model and SNN also comparable in terms of relative predictor importance and patient-specific survival predictions. RSFs showed a tendency for according less importance to predictors with uneven class distributions and predicting clustered survival curves, the latter a result of tuning hyperparameters that influence forest shape through restrictions on terminal node size and tree depth. SNNs were comparatively more sensitive to hyperparameter misspecification, with decreased regularization resulting in inconsistent predicted survival probabilities. We caution against using RSF for predicting patient-specific survival, as standard model tuning practices may result in aggregated predictions, which is not reflected in performance measure values, and recommend performing multiple reruns of SNNs to verify prediction consistency.

Authors 3

1. Audinga‐Dea Hazewinkel corresponding Aachen Department of Medical Oncology

   University of Bristol · MRC Integrative Epidemiology Unit · Leiden University Medical Center

   Affiliation as printed

   Leiden University Medical Center, Department of Medical Oncology, Albinusdreef 2, 2333 ZA, Leiden, The Netherlands

   MRC Integrative Epidemiology Unit, Bristol Medical School, University of Bristol, Oakfield House, Oakfield Grove, BS8 2BN, Bristol, United Kingdom

   Population Health Sciences, Bristol Medical School, University of Bristol, First Floor, 5 Tyndall Avenue, BS8 1DU, Bristol, United Kingdom

   Leiden University Medical Center, Department of Medical Oncology, Albinusdreef 2, 2 ZA, Leiden, The Netherlands

   MRC Integrative Epidemiology Unit, Bristol Medical School, University of Bristol, Oakfield House, Oakfield Grove, BS8 BN, Bristol, United Kingdom

   Population Health Sciences, Bristol Medical School, University of Bristol, First Floor, 5 Tyndall Avenue, BS8 DU, Bristol, United Kingdom

2. Hans A. J. Gelderblom Aachen Department of Medical Oncology

   Leiden University Medical Center

   Affiliation as printed

   Leiden University Medical Center, Department of Medical Oncology, Albinusdreef 2, 2333 ZA, Leiden, The Netherlands

   Leiden University Medical Center, Department of Medical Oncology, Albinusdreef 2, 2 ZA, Leiden, The Netherlands

3. Marta F. Fiocco Aachen Department of Biomedical Data Science Mathematical Institute

   Leiden University · Leiden University Medical Center · Princess Máxima Center

   Affiliation as printed

   Leiden University Medical Center, Medical Statistics, Department of Biomedical Data Science, Albinusdreef 2, 2333 ZA, Leiden, The Netherlands

   Mathematical Institute, Leiden University, Niels Bohrweg 1, 2333 CA, Leiden, The Netherlands

   Princess Maxima Center for Pediatric Oncology, Dept Solid Tumors. Heidelberglaan 25, 3584 CS, Utrecht, The Netherlands