Development and validation of artificial intelligence models for early detection of postoperative infections (PERISCOPE): a multicentre study using electronic health record data

The Lancet Regional Health - Europe, vol. 49, pp. 101163

Abstract

Background: Postoperative infections significantly impact patient outcomes and costs, exacerbated by late diagnoses, yet early reliable predictors are scarce. Existing artificial intelligence (AI) models for postoperative infection prediction often lack external validation or perform poorly in local settings when validated. We aimed to develop locally valid models as part of the PERISCOPE AI system to enable early detection, safer discharge, and more timely treatment of patients. Methods: We developed and validated XGBoost models to predict postoperative infections within 7 and 30 days of surgery. Using retrospective pre-operative and intra-operative electronic health record data from 2014 to 2023 across various surgical specialities, the models were developed at Hospital A and validated and updated at Hospitals B and C in the Netherlands and Belgium. Model performance was evaluated before and after updating using the two most recent years of data as temporal validation datasets. Main outcome measures were model discrimination (area under the receiver operating characteristic curve (AUROC)), calibration (slope, intercept, and plots), and clinical utility (decision curve analysis with net benefit). Findings: The study included 253,010 surgical procedures with 23,903 infections within 30-days. Discriminative performance, calibration properties, and clinical utility significantly improved after updating. Final AUROCs after updating for Hospitals A, B, and C were 0.82 (95% confidence interval (CI) 0.81-0.83), 0.82 (95% CI 0.81-0.83), and 0.91 (95% CI 0.90-0.91) respectively for 30-day predictions on the temporal validation datasets (2022-2023). Calibration plots demonstrated adequate correspondence between observed outcomes and predicted risk. All local models were deemed clinically useful as the net benefit was higher than default strategies (treat all and treat none) over a wide range of clinically relevant decision thresholds. Interpretation: PERISCOPE can accurately predict overall postoperative infections within 7- and 30-days post-surgery. The robust performance implies potential for improving clinical care in diverse clinical target populations. This study supports the need for approaches to local updating of AI models to account for domain shifts in patient populations and data distributions across different clinical settings. Funding: This study was funded by a REACT EU grant from European Regional Development Fund (ERDF) and Kansen voor West.

Authors 10

1. Siri Lise van der Meijden corresponding Aachen Intensive Care Unit

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Healthplus.ai B.V., Amsterdam, the Netherlands

   Intensive Care Unit, Leiden University Medical Centre, Leiden, the Netherlands

2. Anna M van Boekel Aachen Intensive Care Unit

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Intensive Care Unit, Leiden University Medical Centre, Leiden, the Netherlands

3. Laurens Schinkelshoek
   Affiliation as printed

   Healthplus.ai B.V., Amsterdam, the Netherlands

4. Harry van Goor

   Radboud University Nijmegen · Radboud University Medical Center

   Affiliation as printed

   General Surgery Department, Radboud University Medical Centre, Nijmegen, the Netherlands

5. Ewout Willem Steyerberg Aachen Department of Biomedical Data Sciences

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, the Netherlands

6. Rob G. H. H. Nelissen Aachen Department of Orthopaedics, Leiden University Medical Center

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Orthopaedics, Leiden University Medical Centre, Leiden, the Netherlands

7. Dieter Mesotten

   Ziekenhuis Oost-Limburg · Hasselt University

   Affiliation as printed

   Department of Anaesthesiology, Intensive Care Medicine, Ziekenhuis Oost-Limburg, Genk, Belgium

   Faculty of Medicine and Life Sciences, Limburg Clinical Research Centre, UHasselt, Diepenbeek, Belgium

8. Bart F. Geerts
   Affiliation as printed

   Healthplus.ai B.V., Amsterdam, the Netherlands

9. Mark G. J. de Boer Aachen Department of Infectious Diseases

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Infectious Diseases, Leiden University Medical Centre, Leiden, the Netherlands

10. M. Sesmu Arbous Aachen Intensive Care Unit

    Leiden University · Leiden University Medical Center

    Affiliation as printed

    Intensive Care Unit, Leiden University Medical Centre, Leiden, the Netherlands