A Simulation Study to Compare the Predictive Performance of Survival Neural Networks with Cox Models for Clinical Trial Data

Computational and Mathematical Methods in Medicine, vol. 2021, pp. 1–15

Abstract

BACKGROUND: Studies focusing on prediction models are widespread in medicine. There is a trend in applying machine learning (ML) by medical researchers and clinicians. Over the years, multiple ML algorithms have been adapted to censored data. However, the choice of methodology should be motivated by the real-life data and their complexity. Here, the predictive performance of ML techniques is compared with statistical models in a simple clinical setting (small/moderate sample size and small number of predictors) with Monte-Carlo simulations. METHODS: Synthetic data (250 or 1000 patients) were generated that closely resembled 5 prognostic factors preselected based on a European Osteosarcoma Intergroup study (MRC BO06/EORTC 80931). Comparison was performed between 2 partial logistic artificial neural networks (PLANNs) and Cox models for 20, 40, 61, and 80% censoring. Survival times were generated from a log-normal distribution. Models were contrasted in terms of the C-index, Brier score at 0-5 years, integrated Brier score (IBS) at 5 years, and miscalibration at 2 and 5 years (usually neglected). The endpoint of interest was overall survival. RESULTS: PLANNs original/extended were tuned based on the IBS at 5 years and the C-index, achieving a slightly better performance with the IBS. Comparison with Cox models showed that PLANNs can reach similar predictive performance on simulated data for most scenarios with respect to the C-index, Brier score, or IBS. However, Cox models were frequently less miscalibrated. Performance was robust in scenario data where censored patients were removed before 2 years or curtailing at 5 years was performed (on training data). CONCLUSION: Survival neural networks reached a comparable predictive performance with Cox models but were generally less well calibrated. All in all, researchers should be aware of burdensome aspects of ML techniques such as data preprocessing, tuning of hyperparameters, and computational intensity that render them disadvantageous against conventional regression models in a simple clinical setting.

Authors 4

1. Georgios Kantidakis corresponding Aachen Department of Biomedical Data Sciences Section Medical Statistics

   Leiden University · Leiden University Medical Center · European Organisation for Research and Treatment of Cancer

   Affiliation as printed

   Department of Biomedical Data Sciences, Section Medical Statistics, Leiden University Medical Center (LUMC), Albinusdreef 2, 2333 ZA Leiden, Netherlands

   Department of Statistics, European Organisation for Research and Treatment of Cancer (EORTC) Headquarters, Ave E. Mounier 83/11, 1200 Brussels, Belgium

   Mathematical Institute Leiden University, Niels Bohrweg 1, 2333 CA Leiden, Netherlands

2. Elia Mario Biganzoli

   University of Milan · Fondazione IRCCS Istituto Nazionale dei Tumori

   Affiliation as printed

   Department of Clinical Sciences and Community Health & DSRC, University of Milan, Fondazione IRCCS Istituto Nazionale Tumori, Via Giacomo Venezian 1, 20133 Milan, Italy

3. Hein Putter Aachen Department of Biomedical Data Sciences Section Medical Statistics

   Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Sciences, Section Medical Statistics, Leiden University Medical Center (LUMC), Albinusdreef 2, 2333 ZA Leiden, Netherlands

4. Marta F. Fiocco Aachen Department of Biomedical Data Sciences Section Medical Statistics

   Leiden University · Leiden University Medical Center · Princess Máxima Center

   Affiliation as printed

   Department of Biomedical Data Sciences, Section Medical Statistics, Leiden University Medical Center (LUMC), Albinusdreef 2, 2333 ZA Leiden, Netherlands

   Mathematical Institute Leiden University, Niels Bohrweg 1, 2333 CA Leiden, Netherlands

   Trial and Data Center, Princess Máxima Center for Pediatric Oncology (PMC), Heidelberglaan 25, 3584 CS Utrecht, Netherlands