From prediction to practice: early implementation of a machine learning–based hospitalization prediction tool in the emergency department

International Journal of Medical Informatics, vol. 215, pp. 106446

Abstract

BACKGROUND: Machine learning (ML) models can accurately predict hospital admissions in emergency departments (EDs), but real-world adoption remains rare. We evaluated the feasibility and early implementation of an ML-based hospitalization prediction tool in an ED of a tertiary care centre using the Medical Research Council (MRC) framework. METHODS: A prospective mixed-methods study was conducted at the ED of the Leiden University Medical Centre (∼23,000 annual visits). A validated ML-based hospitalization prediction tool was deployed via a web-based dashboard for four months. The dashboard was designed with stakeholders' input obtained during focus groups. Staff use, perceptions, and implementation barriers were assessed through pre-/post-implementation surveys with Likert-scales and open questions, and were analyzed using Mann-Whitney U-tests. Operational outcomes (ED length of stay [LOS], hospitalization rates) were extracted from the Netherlands Emergency department Evaluation Database and compared across pre- and post-implementation periods via logistic regression adjusted for confounders. RESULTS: Tool utilization was low despite an initial positive attitude towards the tool: 67% of staff reported rarely or never using it. Post-implementation surveys indicated a decline in perceived utility and reduced concern about AI replacing clinical roles. Reported barriers included lack of electronic health record integration, absence of linked actions, and misalignment with existing workflows. Hospitalization rose from 31.8% to 33.3% (p = 0.027), while ED-LOS increased during the implementation period (19.8% to 26.1%, p < 0.001); these changes could not be attributed to the tool given limited adoption. CONCLUSION: This early implementation study demonstrated low adoption of a ML prediction tool for hospitalization at the ED. Context-specific implementation barriers were identified. Guided by the MRC framework, the findings offer concrete strategies to enhance future implementation, including EHR integration, embedded action protocols, and role-specific responsibilities.

Authors 9

1. Wouter Raven corresponding Aachen Department of Emergency Medicine

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Emergency Medicine, Leiden University Medical Centre, Albinusdreef 2, 2300 RC Leiden, the Netherlands. Electronic address: w.raven@lumc.nl

2. Anne de Hond

   University Medical Center Utrecht

   Affiliation as printed

   Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, the Netherlands

3. J. Vermeire Aachen Department of Emergency Medicine

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Emergency Medicine, Leiden University Medical Centre, Albinusdreef 2, 2300 RC Leiden, the Netherlands

4. Laurens Schinkelshoek
   Affiliation as printed

   Healthplus.ai, Van Diemenstraat 50, 1013 NH Amsterdam, the Netherlands

5. L. Mulder Aachen CAIRElab AI expertise center

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Cairelab| AI expertise center, Leiden University Medical Centre, Albinusdreef 2, 2300 RC Leiden, the Netherlands

6. A. van Someren Aachen CAIRElab AI expertise center

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Cairelab| AI expertise center, Leiden University Medical Centre, Albinusdreef 2, 2300 RC Leiden, the Netherlands

7. M.I. Gaakeer

   Admiraal De Ruyter Ziekenhuis

   Affiliation as printed

   Department of Emergency Medicine, Adrz Hospital, 's-Gravenpolderseweg 114, 4462 RA Goes, the Netherlands

8. E. de Jonge Aachen Department of Intensive Care

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Intensive Care, Leiden University Medical Centre, Albinusdreef 2, 2300 RC Leiden, the Netherlands

9. Bas de Groot

   Radboud University Nijmegen · Aarhus University Hospital · Radboud University Medical Center

   Affiliation as printed

   Department of Emergency Medicine, Radboud University Medical Centre, Postbus 9101, 6500 HB, Geert Grooteplein-Zuid, Nijmegen, the Netherlands; Research Centre for Emergency Medicine, Department of Clinical Medicine, Aarhus University Hospital, Aarhus, Denmark