Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review
npj Digital Medicine, vol. 5, pp. 2
Abstract
While the opportunities of ML and AI in healthcare are promising, the growth of complex data-driven prediction models requires careful quality and applicability assessment before they are applied and disseminated in daily practice. This scoping review aimed to identify actionable guidance for those closely involved in AI-based prediction model (AIPM) development, evaluation and implementation including software engineers, data scientists, and healthcare professionals and to identify potential gaps in this guidance. We performed a scoping review of the relevant literature providing guidance or quality criteria regarding the development, evaluation, and implementation of AIPMs using a comprehensive multi-stage screening strategy. PubMed, Web of Science, and the ACM Digital Library were searched, and AI experts were consulted. Topics were extracted from the identified literature and summarized across the six phases at the core of this review: (1) data preparation, (2) AIPM development, (3) AIPM validation, (4) software development, (5) AIPM impact assessment, and (6) AIPM implementation into daily healthcare practice. From 2683 unique hits, 72 relevant guidance documents were identified. Substantial guidance was found for data preparation, AIPM development and AIPM validation (phases 1-3), while later phases clearly have received less attention (software development, impact assessment and implementation) in the scientific literature. The six phases of the AIPM development, evaluation and implementation cycle provide a framework for responsible introduction of AI-based prediction models in healthcare. Additional domain and technology specific research may be necessary and more practical experience with implementing AIPMs is needed to support further guidance.
Authors 14
-
Anne de Hond corresponding Aachen Clinical AI Implementation and Research Lab Department of Biomedical Data Sciences Department of Information Technology and Digital Innovation
Leiden University Medical Center
Affiliation as printed
Clinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, The Netherlands. a.a.h.de_hond@lumc.nl
Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands. a.a.h.de_hond@lumc.nl
Department of Information Technology and Digital Innovation, Leiden University Medical Center, Leiden, The Netherlands. a.a.h.de_hond@lumc.nl
Department of Information Technology and Digital Innovation, Leiden University Medical Center, Leiden, The Netherlands
-
Artuur M. Leeuwenberg corresponding
Utrecht University · University Medical Center Utrecht
Affiliation as printed
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands. a.m.leeuwenberg-15@umcutrecht.nl
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
-
Utrecht University · University Medical Center Utrecht
Affiliation as printed
Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
-
Ilse M.J. Kant Aachen Clinical AI Implementation and Research Lab Department of Biomedical Data Sciences Department of Information Technology and Digital Innovation
Leiden University Medical Center
Affiliation as printed
Clinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, The Netherlands
Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands
Department of Information Technology and Digital Innovation, Leiden University Medical Center, Leiden, The Netherlands
-
Utrecht University · University Medical Center Utrecht
Affiliation as printed
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
-
Leiden University Medical Center
Affiliation as printed
Clinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, The Netherlands
National eHealth Living Lab, Leiden, The Netherlands
-
Leiden University Medical Center
Affiliation as printed
Department of Public Health and Primary Care, Leiden University Medical Center, Leiden, The Netherlands
National eHealth Living Lab, Leiden, The Netherlands
-
Utrecht University · University Medical Center Utrecht
Affiliation as printed
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
-
Utrecht University · University Medical Center Utrecht
Affiliation as printed
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
-
Utrecht University · University Medical Center Utrecht
Affiliation as printed
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
-
Utrecht University · University Medical Center Utrecht
Affiliation as printed
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
-
Ewout Willem Steyerberg Aachen Clinical AI Implementation and Research Lab Department of Biomedical Data Sciences
Leiden University Medical Center
Affiliation as printed
Clinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, The Netherlands
Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands
-
Leiden University Medical Center
Affiliation as printed
Department of Public Health and Primary Care, Leiden University Medical Center, Leiden, The Netherlands
National eHealth Living Lab, Leiden, The Netherlands
-
Utrecht University · University Medical Center Utrecht
Affiliation as printed
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
Cited by 496 stored of 501
No patents citing this paper on Lens.org (checked 2026-10-11).