Machine Learning Did Not Outperform Conventional Competing Risk Modeling to Predict Revision Arthroplasty
Clinical Orthopaedics and Related Research, vol. 482, pp. 1472–1482
Abstract
BACKGROUND: Estimating the risk of revision after arthroplasty could inform patient and surgeon decision-making. However, there is a lack of well-performing prediction models assisting in this task, which may be due to current conventional modeling approaches such as traditional survivorship estimators (such as Kaplan-Meier) or competing risk estimators. Recent advances in machine learning survival analysis might improve decision support tools in this setting. Therefore, this study aimed to assess the performance of machine learning compared with that of conventional modeling to predict revision after arthroplasty. QUESTION/PURPOSE: Does machine learning perform better than traditional regression models for estimating the risk of revision for patients undergoing hip or knee arthroplasty? METHODS: Eleven datasets from published studies from the Dutch Arthroplasty Register reporting on factors associated with revision or survival after partial or total knee and hip arthroplasty between 2018 and 2022 were included in our study. The 11 datasets were observational registry studies, with a sample size ranging from 3038 to 218,214 procedures. We developed a set of time-to-event models for each dataset, leading to 11 comparisons. A set of predictors (factors associated with revision surgery) was identified based on the variables that were selected in the included studies. We assessed the predictive performance of two state-of-the-art statistical time-to-event models for 1-, 2-, and 3-year follow-up: a Fine and Gray model (which models the cumulative incidence of revision) and a cause-specific Cox model (which models the hazard of revision). These were compared with a machine-learning approach (a random survival forest model, which is a decision tree-based machine-learning algorithm for time-to-event analysis). Performance was assessed according to discriminative ability (time-dependent area under the receiver operating curve), calibration (slope and intercept), and overall prediction error (scaled Brier score). Discrimination, known as the area under the receiver operating characteristic curve, measures the model's ability to distinguish patients who achieved the outcomes from those who did not and ranges from 0.5 to 1.0, with 1.0 indicating the highest discrimination score and 0.50 the lowest. Calibration plots the predicted versus the observed probabilities; a perfect plot has an intercept of 0 and a slope of 1. The Brier score calculates a composite of discrimination and calibration, with 0 indicating perfect prediction and 1 the poorest. A scaled version of the Brier score, 1 - (model Brier score/null model Brier score), can be interpreted as the amount of overall prediction error. RESULTS: Using machine learning survivorship analysis, we found no differences between the competing risks estimator and traditional regression models for patients undergoing arthroplasty in terms of discriminative ability (patients who received a revision compared with those who did not). We found no consistent differences between the validated performance (time-dependent area under the receiver operating characteristic curve) of different modeling approaches because these values ranged between -0.04 and 0.03 across the 11 datasets (the time-dependent area under the receiver operating characteristic curve of the models across 11 datasets ranged between 0.52 to 0.68). In addition, the calibration metrics and scaled Brier scores produced comparable estimates, showing no advantage of machine learning over traditional regression models. CONCLUSION: Machine learning did not outperform traditional regression models. CLINICAL RELEVANCE: Neither machine learning modeling nor traditional regression methods were sufficiently accurate in order to offer prognostic information when predicting revision arthroplasty. The benefit of these modeling approaches may be limited in this context.
Authors 13
-
Amsterdam University Medical Centers · University of Amsterdam · Delft University of Technology
Affiliation as printed
Amsterdam UMC, University of Amsterdam, Department of Orthopedic Surgery and Sports Medicine, Amsterdam, the Netherlands
Department of Engineering Systems and Services, Faculty of Technology Policy and Management, Delft University of Technology, Delft, the Netherlands
-
Anne de Hond Aachen Clinical AI Implementation and Research Lab Department of Biomedical Data Sciences
Utrecht University · Leiden University Medical Center · University Medical Center Utrecht
Affiliation as printed
Clinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, the Netherlands
Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
-
Affiliation as printed
Department of Orthopaedic Surgery, Medical Center Leeuwarden, Leeuwarden, the Netherlands
-
Affiliation as printed
Dutch Arthroplasty Register (LROI), ‘s-Hertogenbosch, the Netherlands
Dutch Arthroplasty Register (LROI), 's-Hertogenbosch, the Netherlands
-
Leiden University · Leiden University Medical Center
Affiliation as printed
Department of Orthopaedic Surgery, Leiden University Medical Centre, Leiden, the Netherlands
-
Affiliation as printed
Department of Orthopaedic Surgery, Medical Center Leeuwarden, Leeuwarden, the Netherlands
-
Leiden University · Leiden University Medical Center
Affiliation as printed
Department of Orthopaedic Surgery, Leiden University Medical Centre, Leiden, the Netherlands
-
The University of Texas at Austin
Affiliation as printed
Department of Surgery and Perioperative Care, Dell Medical School, University of Texas, Austin, TX, USA
-
University Medical Center Groningen · University of Groningen
Affiliation as printed
Department of Orthopaedic and Trauma Surgery, University Medical Center Groningen, University of Groningen, the Netherlands
-
Amsterdam University Medical Centers · University of Amsterdam
Affiliation as printed
Amsterdam UMC, University of Amsterdam, Department of Orthopedic Surgery and Sports Medicine, Amsterdam, the Netherlands
-
Leiden University Medical Center
Affiliation as printed
Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands
-
Ewout Willem Steyerberg Aachen Clinical AI Implementation and Research Lab Department of Biomedical Data Sciences
Leiden University Medical Center
Affiliation as printed
Clinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, the Netherlands
Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands
-
University Medical Center Groningen · University of Groningen
Affiliation as printed
Department of Orthopaedic and Trauma Surgery, University Medical Center Groningen, University of Groningen, the Netherlands
Cited by 12 stored of 12
12 results
No patents citing this paper on Lens.org (checked 2026-10-11).