Beyond Benchmarks: Towards Robust Artificial Intelligence Bone Segmentation in Socio-Technical Systems

medRxiv

Abstract

Abstract Despite the advances in automated medical image segmentation, AI models still underperform in various clinical settings, challenging real-world integration. In this multicenter evaluation, we analyzed 20 state-of-the-art mandibular segmentation models across 19,218 segmentations of 1,000 clinically resampled CT/CBCT scans. We show that segmentation accuracy varies by up to 25% depending on socio-technical factors such as voxel size, bone orientation, and patient conditions such as osteosynthesis or pathology. Higher sharpness, isotropic smaller voxels, and neutral orientation significantly improved results, while metallic osteosynthesis and anatomical complexity led to significant degradation. Our findings challenge the common view of AI models as “plug-and-play” tools and suggest evidence-based optimization recommendations for both clinicians and developers. This will in turn boost the integration of AI segmentation tools in routine healthcare.

Authors 50

1. Kunpeng Xie Aachen Department of Oral and Maxillofacial Surgery

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen;

   Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, 52074 Aachen, Germany

2. Lennart Johannes Gruber

   Universitätsmedizin Göttingen · University of Göttingen

   Affiliation as printed

   Department of Oral and Maxillofacial Surgery, University Medical Center Goettingen;

   Department of Oral and Maxillofacial Surgery, University Medical Center Goettingen, 7075 Goettingen, Germany

3. Martin Crampen Aachen Institute of Medical Informatics Department of Oral and Maxillofacial Surgery

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Institute of Medical Informatics, University Hospital RWTH Aachen;

   Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, 52074 Aachen, Germany

4. Yao Li Aachen Department of Oral and Maxillofacial Surgery

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen;

   Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, 52074 Aachen, Germany

5. André Filipe Ferreira Aachen Department of Oral and Maxillofacial Surgery

   Essen University Hospital · Universitätsklinikum Aachen · RWTH Aachen University · University of Minho

   Affiliation as printed

   Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen;

   Center Algoritmi/LASI, University of Minho, 710-057 Braga, Portugal

   Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, 52074 Aachen, Germany

   Institute of Artificial Intelligence in Medicine (IKIM), University Hospital Essen, 45131 Essen, Germany

6. Elias Tappeiner

   UMIT TIROL - Private University for Health Sciences and Health Technology

   Affiliation as printed

   UMIT TIROL - Private University For Health Sciences and Health Technology;

   UMIT TIROL – Private University For Health Sciences and Health Technology, 6060 Hall in Tyrol, Austria

7. Maxime Gillot

   University of Michigan · École d'Ingénieurs en Chimie et Sciences du Numérique

   Affiliation as printed

   University of Michigan, Ann Arbor;

   CPE Lyon, 69100 Villeurbanne, Lyon, France

   University of Michigan, Ann Arbor, 48109-1079 Michigan, United States

8. Jan Schepers

   Materialise (Belgium)

   Affiliation as printed

   Materialise NV;

   Materialise NV, 3001 Leuven, Belgium

9. Jiangchang Xu

   Shanghai Jiao Tong University

   Affiliation as printed

   School of Mechanical Engineering, Shanghai Jiao Tong University;

   School of Mechanical Engineering, Shanghai Jiao Tong University, 200240 Shanghai, China

10. Tobias Pankert Aachen Department of Oral and Maxillofacial Surgery

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Inzipio GmbH;

    Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, 52074 Aachen, Germany

    Inzipio GmbH, 52070 Aachen, Germany

11. Michel Beyer

    University of Basel · University Hospital of Basel

    Affiliation as printed

    Department of Oral and Cranio-Maxillofacial Surgery, University Hospital Basel;

    Department of Oral and Cranio-Maxillofacial Surgery, University Hospital Basel, 4031 Basel, Switzerland

    Medical Additive Manufacturing Research Group (Swiss MAM), Department of Biomedical Engineering, University of Basel, 4123 Allschwil, Switzerland

12. Negar Shahamiri

    Essen University Hospital

    Affiliation as printed

    Institute of Artificial Intelligence in Medicine (IKIM), University Hospital Essen;

    Institute of Artificial Intelligence in Medicine (IKIM), University Hospital Essen, 45131 Essen, Germany

13. Reinier ten Brink

    University Medical Center Groningen · University of Groningen

    Affiliation as printed

    Department of Oral & Maxillofacial Surgery, University Medical Center Groningen;

    3D Lab University Medical Center Groningen, University of Groningen, 9713 GZ Groningen, The Netherlands

    Department of Oral & Maxillofacial Surgery, University Medical Center Groningen, 9713 GZ Groningen, The Netherlands

14. Gauthier Dot

    Université Paris Cité

    Affiliation as printed

    Universite Paris Cité, UFR Odontologie;

    Universite Paris Cité, UFR Odontologie, F-75006 Paris, France

15. Charlotte Weschke
    Affiliation as printed

    1000shapes GmbH;

    1000shapes GmbH, 12247 Berlin, Germany

16. Niels van Nistelrooij

    Radboud University Nijmegen · Radboud University Medical Center · Humboldt-Universität zu Berlin · Einstein Center Digital Future · Freie Universität Berlin · Charité - Universitätsmedizin Berlin

    Affiliation as printed

    Department of Oral and Maxillofacial Surgery, Radboud University Medical Center;

    Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Department of Oral and Maxillofacial Surgery, 12203 Berlin, Germany

    Department of Oral and Maxillofacial Surgery, Radboud University Medical Center, 6525 GA Nijmegen, The Netherlands

    Einstein Center Digital Future (ECDF), 10117 Berlin, Germany

17. Pieter-Jan Verhelst

    Universitair Ziekenhuis Leuven

    Affiliation as printed

    Department of Oral and Maxillofacial Surgery, University Hospitals Leuven;

    Department of Oral and Maxillofacial Surgery, University Hospitals Leuven, 3000 Leuven, Belgium

18. Yan Long Guo Aachen Department of Oral and Maxillofacial Surgery

    Shanghai Jiao Tong University · Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    School of Mechanical Engineering, Shanghai Jiao Tong University;

    Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, 52074 Aachen, Germany

    School of Mechanical Engineering, Shanghai Jiao Tong University, 200240 Shanghai, China

19. Zhibin Xu Aachen Department of Oral and Maxillofacial Surgery

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen;

    Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, 52074 Aachen, Germany

20. Jonas Bienzeisler Aachen Institute of Medical Informatics

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Institute of Medical Informatics, University Hospital RWTH Aachen;

    Institute of Medical Informatics, University Hospital RWTH Aachen, 5074 Aachen, Germany

21. Ashkan Rashad Aachen Department of Oral and Maxillofacial Surgery

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen;

    Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, 52074 Aachen, Germany

22. Tabea Viktoria Flügge

    Humboldt-Universität zu Berlin · Einstein Center Digital Future · Freie Universität Berlin · Charité - Universitätsmedizin Berlin

    Affiliation as printed

    Charité Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Department of Oral and Maxillofacial Surgery;

    Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Department of Oral and Maxillofacial Surgery, 12203 Berlin, Germany

    Einstein Center Digital Future (ECDF), 10117 Berlin, Germany

23. Ross Cotton
    Affiliation as printed

    Synopsys Northern Europe Ltd.;

    Synopsys Northern Europe Ltd., Exeter EX4 3PL, United Kingdom

24. Shankeeth Vinayahalingam

    Radboud University Nijmegen · Radboud University Medical Center

    Affiliation as printed

    Department of Oral and Maxillofacial Surgery, Radboud University Medical Center;

    Department of Oral and Maxillofacial Surgery, Radboud University Medical Center, 6525 GA Nijmegen, The Netherlands

25. Robert Radu Ileșan

    Luzerner Kantonsspital

    Affiliation as printed

    Department of Oral and Maxillofacial Surgery, Lucerne Cantonal Hospital;

    Department of Oral and Maxillofacial Surgery, Lucerne Cantonal Hospital, 6000 Lucerne, Switzerland

26. Stefan Raith
    Affiliation as printed

    Inzipio GmbH;

    Inzipio GmbH, 52070 Aachen, Germany

27. Constantin Seibold

    Essen University Hospital

    Affiliation as printed

    Institute of Artificial Intelligence in Medicine (IKIM), University Hospital Essen;

    Institute of Artificial Intelligence in Medicine (IKIM), University Hospital Essen, 45131 Essen, Germany

28. Tong Xi

    Radboud University Nijmegen · Radboud University Medical Center

    Affiliation as printed

    Department of Oral and Maxillofacial Surgery, Radboud University Medical Center;

    Department of Oral and Maxillofacial Surgery, Radboud University Medical Center, 6525 GA Nijmegen, The Netherlands

29. Stefaan Jozef Bergé

    Radboud University Nijmegen · Radboud University Medical Center

    Affiliation as printed

    Department of Oral and Maxillofacial Surgery, Radboud University Medical Center;

    Department of Oral and Maxillofacial Surgery, Radboud University Medical Center, 6525 GA Nijmegen, The Netherlands

30. Sven Nebelung Aachen Department of Interventional and Diagnostic Radiology

    RWTH Aachen University

    Affiliation as printed

    Department of Interventional and Diagnostic Radiology, RWTH Aachen University;

    Department of Interventional and Diagnostic Radiology, RWTH Aachen University, 52074 Aachen, Germany

31. Oldřich Kodym

    Tescan (Czechia)

    Affiliation as printed

    TESCAN 3DIM;

    TESCAN 3DIM, 61700 Brno, Czech Republic

32. Osku Sundqvist

    Planmeca (Finland)

    Affiliation as printed

    Planmeca Oy;

    Planmeca Oy, FIN-00880 Helsinki, Finland

33. Florian M. Thieringer

    University of Basel · University Hospital of Basel

    Affiliation as printed

    Department of Oral and Cranio-Maxillofacial Surgery, University Hospital Basel;

    Department of Oral and Cranio-Maxillofacial Surgery, University Hospital Basel, 4031 Basel, Switzerland

    Medical Additive Manufacturing Research Group (Swiss MAM), Department of Biomedical Engineering, University of Basel, 4123 Allschwil, Switzerland

34. Hans Lamecker
    Affiliation as printed

    1000shapes GmbH;

    1000shapes GmbH, 12247 Berlin, Germany

35. Antoine Coppens
    Affiliation as printed

    Relu BV;

    Relu BV, 3001 Leuven, Belgium

36. Thomas Potrusil

    KLS Martin (Germany)

    Affiliation as printed

    CADS GmbH;

    CADS GmbH, 4320 Perg, Austria

    KLS Martin Group, 78532 Tuttlingen, Germany

37. Joep Kraeima

    University Medical Center Groningen · University of Groningen

    Affiliation as printed

    Department of Oral & Maxillofacial Surgery, University Medical Center Groningen;

    3D Lab University Medical Center Groningen, University of Groningen, 9713 GZ Groningen, The Netherlands

    Department of Oral & Maxillofacial Surgery, University Medical Center Groningen, 9713 GZ Groningen, The Netherlands

38. Max J. H. Witjes

    University Medical Center Groningen

    Affiliation as printed

    Department of Oral & Maxillofacial Surgery, University Medical Center Groningen;

    Department of Oral & Maxillofacial Surgery, University Medical Center Groningen, 9713 GZ Groningen, The Netherlands

39. Guomin Wu

    Jilin University

    Affiliation as printed

    Craniomaxillofacial Plastic and Cosmetic Center, Hospital of Stomatology, Jilin University;

    Craniomaxillofacial Plastic and Cosmetic Center, Hospital of Stomatology, Jilin University, 130021 Changchun, China

40. Xiaojun Chen

    Shanghai Jiao Tong University

    Affiliation as printed

    School of Mechanical Engineering, Shanghai Jiao Tong University;

    School of Mechanical Engineering, Shanghai Jiao Tong University, 200240 Shanghai, China

41. Adriaan Lambrechts

    Materialise (Belgium)

    Affiliation as printed

    Materialise NV;

    Materialise NV, 3001 Leuven, Belgium

42. Lucía Cevidanes

    University of Michigan

    Affiliation as printed

    University of Michigan, Ann Arbor;

    University of Michigan, Ann Arbor, 48109-1079 Michigan, United States

43. Stefan Zachow

    Zuse Institute Berlin · Humboldt-Universität zu Berlin · Freie Universität Berlin · Charité - Universitätsmedizin Berlin

    Affiliation as printed

    Department of Visual- and Data-centric Computing, Zuse Institute;

    Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Department of Oral and Maxillofacial Surgery, 12203 Berlin, Germany

    Department of Visual– and Data-centric Computing, Zuse Institute, 14195 Berlin, Germany

44. Alexander Hermans Aachen Visual Computing Institute Department of Interventional and Diagnostic Radiology

    RWTH Aachen University

    Affiliation as printed

    Visual Computing Institute, Computer Science and Natural Sciences, RWTH Aachen University;

    Department of Interventional and Diagnostic Radiology, RWTH Aachen University, 52074 Aachen, Germany

45. Daniel Truhn Aachen Department of Interventional and Diagnostic Radiology

    RWTH Aachen University

    Affiliation as printed

    Department of Interventional and Diagnostic Radiology, RWTH Aachen University;

    Department of Interventional and Diagnostic Radiology, RWTH Aachen University, 52074 Aachen, Germany

46. Victor Alves

    Centro ALGORITMI · University of Minho

    Affiliation as printed

    Center Algoritmi / LASI, University of Minho

    Center Algoritmi/LASI, University of Minho, 710-057 Braga, Portugal

47. Jan Egger

    Essen University Hospital · Stiftung Universitätsmedizin Essen · University of Duisburg-Essen

    Affiliation as printed

    Institute of Artificial Intelligence in Medicine (IKIM), University Hospital Essen;

    Cancer Research Center Cologne Essen (CCCE), University Medicine Essen (AöR), 45147 Essen, Germany

    Institute of Artificial Intelligence in Medicine (IKIM), University Hospital Essen, 45131 Essen, Germany

    University of Duisburg-Essen, Faculty of Computer Science, 45127 Essen, Germany

48. Rainer Röhrig Aachen Institute of Medical Informatics

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Institute of Medical Informatics, University Hospital RWTH Aachen;

    Institute of Medical Informatics, University Hospital RWTH Aachen, 5074 Aachen, Germany

49. Frank Hölzle Aachen Department of Oral and Maxillofacial Surgery

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen;

    Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, 52074 Aachen, Germany

50. Behrus Puladi corresponding Aachen Department of Oral and Maxillofacial Surgery

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen;

    Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, 52074 Aachen, Germany