Beyond benchmarks: Towards robust artificial intelligence bone segmentation in socio-technical systems
Expert Systems with Applications, vol. 299, pp. 130031
Abstract
Despite the advances in automated medical image segmentation, AI models still underperform in various clinical settings, posing challenges for integration into real-world workflows. In this pre-registered prospective multicenter evaluation, we analyzed 20 state-of-the-art mandibular segmentation models across 19,218 segmentations of 1,000 clinically resampled CT/CBCT scans. Our results suggest that for a given model, segmentation accuracy can vary by up to 25% in Dice score as socio-technical factors such as voxel size, bone orientation, and patient conditions (e.g., osteosynthesis or pathology) shift from favorable to adverse. Higher sharpness, isotropic smaller voxels, and neutral orientation significantly improved results, while metallic osteosynthesis and anatomical complexity led to significant degradation. Our findings challenge the common view of AI models as “plug-and-play” tools and suggest evidence-based optimization recommendations for both clinicians and developers. This will in turn boost the integration of AI segmentation tools in routine healthcare.
Authors 50
-
Universitätsklinikum Aachen · RWTH Aachen University
Affiliation as printed
Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen 52074 Aachen, Germany
Institute of Medical Informatics, University Hospital RWTH Aachen 52074 Aachen, Germany
-
Universitätsmedizin Göttingen · University of Göttingen
Affiliation as printed
Department of Oral and Maxillofacial Surgery, University Medical Center Goettingen 37075 Goettingen, Germany
-
Universitätsklinikum Aachen · RWTH Aachen University
Affiliation as printed
Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen 52074 Aachen, Germany
Institute of Medical Informatics, University Hospital RWTH Aachen 52074 Aachen, Germany
-
Universitätsklinikum Aachen · RWTH Aachen University
Affiliation as printed
Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen 52074 Aachen, Germany
Institute of Medical Informatics, University Hospital RWTH Aachen 52074 Aachen, Germany
-
André Filipe Ferreira Aachen Department of Oral and Maxillofacial Surgery Institute of Medical Informatics
Essen University Hospital · Universitätsklinikum Aachen · RWTH Aachen University · University of Minho
Affiliation as printed
Center Algoritmi/LASI, University of Minho, 4710-057 Braga, Portugal
Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen 52074 Aachen, Germany
Institute of Artificial Intelligence in Medicine (IKIM), University Hospital Essen 45131 Essen, Germany
Institute of Medical Informatics, University Hospital RWTH Aachen 52074 Aachen, Germany
-
UMIT TIROL - Private University for Health Sciences and Health Technology
Affiliation as printed
UMIT TIROL - Private University For Health Sciences and Health Technology, 6060 Hall in Tyrol, Austria
-
University of Michigan · École d'Ingénieurs en Chimie et Sciences du Numérique
Affiliation as printed
CPE Lyon, 69100 Villeurbanne, Lyon, France
University of Michigan, Ann Arbor, 48109-1079 Michigan, United States
-
Affiliation as printed
Materialise NV, 3001 Leuven, Belgium
-
Affiliation as printed
School of Mechanical Engineering, Shanghai Jiao Tong University, 200240 Shanghai, China
-
Universitätsklinikum Aachen · RWTH Aachen University
Affiliation as printed
Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen 52074 Aachen, Germany
Inzipio GmbH, 52070 Aachen, Germany
-
University of Basel · University Hospital of Basel
Affiliation as printed
Department of Oral and Cranio-Maxillofacial Surgery, University Hospital Basel 4031 Basel, Switzerland
Medical Additive Manufacturing Research Group (Swiss MAM), Department of Biomedical Engineering, University of Basel, 4123 Allschwil, Switzerland
-
Affiliation as printed
Institute of Artificial Intelligence in Medicine (IKIM), University Hospital Essen 45131 Essen, Germany
-
University Medical Center Groningen · University of Groningen
Affiliation as printed
3D Lab University Medical Center Groningen, University of Groningen 9713 GZ Groningen, the Netherlands
Department of Oral & Maxillofacial Surgery, University Medical Center Groningen 9713 GZ Groningen, the Netherlands
-
Affiliation as printed
Universite Paris Cité, UFR Odontologie, F-75006 Paris, France
-
Affiliation as printed
1000shapes GmbH, 12247 Berlin, Germany
-
Radboud University Nijmegen · Radboud University Medical Center · Humboldt-Universität zu Berlin · Einstein Center Digital Future · Freie Universität Berlin · Charité - Universitätsmedizin Berlin
Affiliation as printed
Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Department of Oral and Maxillofacial Surgery, 12203 Berlin, Germany
Department of Oral and Maxillofacial Surgery, Radboud University Medical Center, 6525 GA Nijmegen, the Netherlands
Einstein Center Digital Future (ECDF), 10117 Berlin, Germany
-
Universitair Ziekenhuis Leuven · KU Leuven
Affiliation as printed
Department of Oral and Maxillofacial Surgery, University Hospitals Leuven 3000 Leuven, Belgium
OMFS-IMPATH Research Group, Department of Imaging and Pathology, Faculty of Medicine, KU Leuven 3000 Leuven, Belgium
-
Shanghai Jiao Tong University · Universitätsklinikum Aachen · RWTH Aachen University
Affiliation as printed
Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen 52074 Aachen, Germany
School of Mechanical Engineering, Shanghai Jiao Tong University, 200240 Shanghai, China
-
Universitätsklinikum Aachen · RWTH Aachen University
Affiliation as printed
Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen 52074 Aachen, Germany
-
Universitätsklinikum Aachen · RWTH Aachen University
Affiliation as printed
Institute of Medical Informatics, University Hospital RWTH Aachen 52074 Aachen, Germany
Cited by 3 stored of 3
3 results
No patents citing this paper on Lens.org (checked 2026-10-06).