Assessing performance and clinical usefulness in prediction models with survival outcomes: practical guidance for Cox proportional hazards models

medRxiv

Abstract

Abstract Risk prediction models need thorough validation to assess their performance. Validation of models for survival outcomes poses challenges due to the censoring of observations and the varying time horizon at which predictions can be made. We aim to give a description of measures to evaluate predictions and the potential improvement in decision making from survival models based on Cox proportional hazards regression. As a motivating case study, we consider the prediction of the composite outcome of recurrence and death (the ‘event’) in breast cancer patients following surgery. We develop a Cox regression model with three predictors as in the Nottingham Prognostic Index in 2982 women (1275 events within 5 years of follow-up) and externally validate this model in 686 women (285 events within 5 years). The improvement in performance was assessed following the addition of circulating progesterone as a prognostic biomarker. The model predictions can be evaluated across the full range of observed follow up times or for the event occurring by a fixed time horizon of interest. We first discuss recommended statistical measures that evaluate model performance in terms of discrimination, calibration, or overall performance. Further, we evaluate the potential clinical utility of the model to support clinical decision making. SAS and R code is provided to illustrate apparent, internal, and external validation, both for the three predictor model and when adding progesterone. We recommend the proposed set of performance measures for transparent reporting of the validity of predictions from survival models.

Authors 8

1. David J. McLernon corresponding

   University of Aberdeen

   Affiliation as printed

   Institute of Applied Health Sciences, University of Aberdeen, Aberdeen, UK

2. Daniele Giardiello Aachen Department of Biomedical Data Sciences

   Eurac Research · Leiden University Medical Center · The Netherlands Cancer Institute · Institute for Biomedicine · University of Lübeck

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands

   Institute of Biomedicine, Eurac Research, Affiliated Institute of the University of Lübeck, Bolzano, Italy

   Netherlands Cancer Institute, Amsterdam, the Netherlands

3. Ben Van Calster Aachen Department of Biomedical Data Sciences

   Leiden University Medical Center · KU Leuven

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands

   Department of Development and Regeneration, KU Leuven

4. Laure Wynants

   Maastricht University

   Affiliation as printed

   School for Public Health and Primary Care, Maastricht University

5. Nan van Geloven Aachen Department of Biomedical Data Sciences

   Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands

6. Maarten van Smeden

   Utrecht University · University Medical Center Utrecht

   Affiliation as printed

   Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands

7. Terry M. Therneau

   Mayo Clinic

   Affiliation as printed

   Department of Quantitative Health Sciences, Mayo Clinic, Rochester MN, USA

8. Ewout Willem Steyerberg Aachen Department of Biomedical Data Sciences

   Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands