Development and validation of a reinforcement learning algorithm to dynamically optimize mechanical ventilation in critical care

npj Digital Medicine, vol. 4, pp. 32

Abstract

Abstract The aim of this work was to develop and evaluate the reinforcement learning algorithm VentAI, which is able to suggest a dynamically optimized mechanical ventilation regime for critically-ill patients. We built, validated and tested its performance on 11,943 events of volume-controlled mechanical ventilation derived from 61,532 distinct ICU admissions and tested it on an independent, secondary dataset (200,859 ICU stays; 25,086 mechanical ventilation events). A patient “data fingerprint” of 44 features was extracted as multidimensional time series in 4-hour time steps. We used a Markov decision process, including a reward system and a Q-learning approach, to find the optimized settings for positive end-expiratory pressure (PEEP), fraction of inspired oxygen (FiO 2 ) and ideal body weight-adjusted tidal volume (Vt). The observed outcome was in-hospital or 90-day mortality. VentAI reached a significantly increased estimated performance return of 83.3 (primary dataset) and 84.1 (secondary dataset) compared to physicians’ standard clinical care (51.1). The number of recommended action changes per mechanically ventilated patient constantly exceeded those of the clinicians. VentAI chose 202.9% more frequently ventilation regimes with lower Vt (5–7.5 mL/kg), but 50.8% less for regimes with higher Vt (7.5–10 mL/kg). VentAI recommended 29.3% more frequently PEEP levels of 5–7 cm H 2 O and 53.6% more frequently PEEP levels of 7–9 cmH 2 O. VentAI avoided high (>55%) FiO 2 values (59.8% decrease), while preferring the range of 50–55% (140.3% increase). In conclusion, VentAI provides reproducible high performance by dynamically choosing an optimized, individualized ventilation strategy and thus might be of benefit for critically ill patients.

Authors 13

1. Arne Peine Aachen Department of Intensive Care and Intermediate Care

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Department of Intensive Care and Intermediate Care, University Hospital RWTH Aachen, Pauwelsstreet 30, Aachen, Germany

2. Ahmed Hallawa Aachen Chair for Integrated Signal Processing Systems Department of Intensive Care and Intermediate Care

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Chair for Integrated Signal Processing Systems, RWTH Aachen University, Kopernikusstreet 16, Aachen, Germany

   Department of Intensive Care and Intermediate Care, University Hospital RWTH Aachen, Pauwelsstreet 30, Aachen, Germany

3. Johannes Bickenbach Aachen Department of Intensive Care and Intermediate Care

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Department of Intensive Care and Intermediate Care, University Hospital RWTH Aachen, Pauwelsstreet 30, Aachen, Germany

4. Guido Dartmann

   Trier University of Applied Sciences

   Affiliation as printed

   Environmental Campus Birkenfeld, Trier University of Applied Sciences, Schneidershof, Trier, Germany

5. Lejla Begic Fazlic

   Trier University of Applied Sciences

   Affiliation as printed

   Environmental Campus Birkenfeld, Trier University of Applied Sciences, Schneidershof, Trier, Germany

6. Anke Schmeink Aachen

   RWTH Aachen University

   Affiliation as printed

   Research Area Information Theory and Systematic Design of Communication Systems, RWTH Aachen University, Kopernikusstreet 16, Aachen, Germany

7. Gerd Ascheid Aachen Chair for Integrated Signal Processing Systems

   RWTH Aachen University

   Affiliation as printed

   Chair for Integrated Signal Processing Systems, RWTH Aachen University, Kopernikusstreet 16, Aachen, Germany

8. Christoph Thiemermann

   Queen Mary University of London · William Harvey Research Institute

   Affiliation as printed

   William Harvey Research Institute, Queen Mary University London, Charterhouse Square, London, United Kingdom

9. Andreas Schuppert Aachen Joint Research Centre for Computational Biomedicine (JRC-COMBINE)

   RWTH Aachen University

   Affiliation as printed

   Joint Research Center for Computational Biomedicine, RWTH Aachen University, Pauwelsstreet 30, Aachen, Germany

10. Ryan D. Kindle

    Beth Israel Deaconess Medical Center · Harvard–MIT Division of Health Sciences and Technology · Massachusetts Institute of Technology

    Affiliation as printed

    Division of Pulmonary, Critical Care and Sleep Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA

    Laboratory for Computational Physiology, Harvard-MIT Division of Health Sciences & Technology, Cambridge, MA, USA

    Laboratory for Computational Physiology, Harvard–MIT Division of Health Sciences & Technology, Cambridge, MA, USA

11. Leo Anthony Celi

    Beth Israel Deaconess Medical Center · Harvard University · Harvard–MIT Division of Health Sciences and Technology · Massachusetts Institute of Technology

    Affiliation as printed

    Department of Biostatistics Harvard T.H, Chan School of Public Health, Boston, MA, USA

    Division of Pulmonary, Critical Care and Sleep Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA

    Laboratory for Computational Physiology, Harvard-MIT Division of Health Sciences & Technology, Cambridge, MA, USA

    Laboratory for Computational Physiology, Harvard–MIT Division of Health Sciences & Technology, Cambridge, MA, USA

12. Gernot Marx Aachen Department of Intensive Care and Intermediate Care

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Department of Intensive Care and Intermediate Care, University Hospital RWTH Aachen, Pauwelsstreet 30, Aachen, Germany

13. Lukas Märtin corresponding Aachen Department of Intensive Care and Intermediate Care

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Department of Intensive Care and Intermediate Care, University Hospital RWTH Aachen, Pauwelsstreet 30, Aachen, Germany. lmartin@ukaachen.de

    Department of Intensive Care and Intermediate Care, University Hospital RWTH Aachen, Pauwelsstreet 30, Aachen, Germany