OSS: Open Suturing Skills Vision-Based Assessment Challenge 2024-2025
Abstract
Achieving high levels of surgical skill through effective training is essential for optimal patient outcomes. Automated, data-driven skill assessment holds significant potential to improve surgical training. While machine learning-based methods are increasingly popular for assessing skills in minimally invasive surgery, their application to open surgery remains limited. We present the results of a dedicated MICCAI challenge designed to benchmark and advance vision-based skill assessment in open surgery. The challenge dataset comprises videos of an open suturing training task recorded with a static GoPro camera in a dry-lab setting, with instrument trajectories available in addition to the primary video modality. The OSS Challenge was hosted over two consecutive years, comprising two and three independent tasks, respectively: (1) classifying skill level into four classes, (2) predicting the full Objective Structured Assessment of Technical Skills across eight categories, and (3) tracking hands and surgical tools. Participants submitted diverse solutions including deep learning-based video models, tracking-driven methods, and hybrid approaches. General-purpose spatiotemporal video models consistently achieved the strongest performance, though conceptually diverse approaches reached competitive levels when well-executed. Predicting fine-grained OSATS scores remains challenging but benefits substantially from increased training data. Keypoint tracking proves difficult given frequent occlusions and out-of-frame instances, limiting current applicability for motion-based skill analysis. This work benchmarks innovative and diverse solutions for surgical skill assessment, highlighting both the promise and current limitations of video-based evaluation in open surgery and identifying critical directions for advancing automated skill assessment toward clinical impact.
Authors 57
-
Technische Universität Dresden · University Hospital Carl Gustav Carus · National Center for Tumor Diseases · Nationales Centrum für Tumorerkrankungen Dresden
Affiliation as printed
Department of Translational Surgical Oncology , National Center for Tumor Diseases (NCT/UCC) Dresden , , Dresden , Germany
Faculty of Medicine and University Hospital Carl Gustav Carus , TUD Dresden University of Technology , , Dresden , Germany
-
Technische Universität Dresden · University Hospital Carl Gustav Carus · National Center for Tumor Diseases · Nationales Centrum für Tumorerkrankungen Dresden
Affiliation as printed
Department of Translational Surgical Oncology , National Center for Tumor Diseases (NCT/UCC) Dresden , , Dresden , Germany
Faculty of Medicine and University Hospital Carl Gustav Carus , TUD Dresden University of Technology , , Dresden , Germany
-
Heidelberg University · German Cancer Research Center · National Center for Tumor Diseases · Nationales Centrum für Tumorerkrankungen Dresden
Affiliation as printed
Department of Translational Surgical Oncology , National Center for Tumor Diseases (NCT/UCC) Dresden , , Dresden , Germany
German Cancer Research Center (DKFZ) , , Heidelberg , Germany
-
Essen University Hospital · Institute for Advanced Study in the Humanities Essen
Affiliation as printed
Institute for AI in Medicine (IKIM) , University Medicine Essen , Essen , Germany
-
Affiliation as printed
Institute of Medical Informatics , University Hospital RWTH Aachen , , Aachen , Germany
-
Affiliation as printed
Department of Oral and Maxillofacial Surgery , University Hospital RWTH Aachen , , Aachen , Germany
-
RWTH Aachen University · Nagoya Institute of Technology · Aichi Institute of Technology
Affiliation as printed
Department of Information Science , Aichi Institute of Technology , Nagoya , Japan
Department of Oral and Maxillofacial Surgery , University Hospital RWTH Aachen , , Aachen , Germany
-
Affiliation as printed
Department of Oral and Maxillofacial Surgery , University Hospital RWTH Aachen , , Aachen , Germany
-
Affiliation as printed
Center Algoritmi/LASI , University of Minho , Braga , Portugal
-
Affiliation as printed
Center Algoritmi/LASI , University of Minho , Braga , Portugal
-
Affiliation as printed
Center Algoritmi/LASI , University of Minho , Braga , Portugal
-
Affiliation as printed
Center Algoritmi/LASI , University of Minho , Braga , Portugal
-
Hong Kong University of Science and Technology
Affiliation as printed
Department of Computer Science and Engineering , The Hong Kong University of Science and Technology , Hong Kong , China
-
Affiliation as printed
School of Computing , Queen's University , Kingston , Canada
-
Affiliation as printed
Center Algoritmi/LASI , University of Minho , Braga , Portugal
-
Affiliation as printed
Graduate School of Informatics , Nagoya University , Nagoya , Japan
-
Konica Minolta (United States) · Konica Minolta (Japan)
Affiliation as printed
Konica Minolta, Inc. , , Japan
-
Affiliation as printed
School of Computing , Queen's University , Kingston , Canada
-
Seoul National University Hospital
Affiliation as printed
Department of Transdisciplinary Medicine , Seoul National University Hospital , Seoul , South Korea
-
Seoul National University Hospital
Affiliation as printed
Department of Transdisciplinary Medicine , Seoul National University Hospital , Seoul , South Korea
-
Niigata University of Health and Welfare
Affiliation as printed
Niigata University of Health and Welfare , , Japan
-
University of California, Los Angeles · UCLA Health
Affiliation as printed
Computational and Systems Biology Program , UCLA , Los Angeles , USA
Jmees, Inc. , , Japan
-
Nagoya Institute of Technology · Aichi Institute of Technology
Affiliation as printed
Department of Information Science , Aichi Institute of Technology , Nagoya , Japan
-
Muroran Institute of Technology
Affiliation as printed
Muroran Institute of Technology , , Japan
-
Seoul National University · Seoul National University Hospital · National Medical Center
Affiliation as printed
Department of Transdisciplinary Medicine , Seoul National University Hospital , Seoul , South Korea
Interdisciplinary Program in Medical Informatics , Seoul National University , Seoul , South Korea
-
Affiliation as printed
Jmees, Inc. , , Japan
-
Technion – Israel Institute of Technology
Affiliation as printed
The Faculty of Data and Decisions Science , Technion -Israel Institute of Technology , Haifa , Israel
-
Affiliation as printed
School of Computing , Queen's University , Kingston , Canada
-
Affiliation as printed
School of Computing , Queen's University , Kingston , Canada
-
Affiliation as printed
Jmees, Inc. , , Japan
-
University College London · The London College
Affiliation as printed
UCL Hawkes Institute , University College London , London , England
-
Technion – Israel Institute of Technology
Affiliation as printed
The Faculty of Data and Decisions Science , Technion -Israel Institute of Technology , Haifa , Israel
-
Affiliation as printed
School of Computing , Queen's University , Kingston , Canada
-
Affiliation as printed
Graduate School of Informatics , Nagoya University , Nagoya , Japan
Information Technology Center , Nagoya University , Nagoya , Japan
-
Technion – Israel Institute of Technology
Affiliation as printed
The Faculty of Data and Decisions Science , Technion -Israel Institute of Technology , Haifa , Israel
-
Seoul National University · Seoul National University Hospital · National Medical Center
Affiliation as printed
Department of Transdisciplinary Medicine , Seoul National University Hospital , Seoul , South Korea
Interdisciplinary Program in Medical Informatics , Seoul National University , Seoul , South Korea
-
Affiliation as printed
Center Algoritmi/LASI , University of Minho , Braga , Portugal
-
Affiliation as printed
Center Algoritmi/LASI , University of Minho , Braga , Portugal
-
Seoul National University · Seoul National University Hospital · National Medical Center
Affiliation as printed
Department of Transdisciplinary Medicine , Seoul National University Hospital , Seoul , South Korea
Interdisciplinary Program in Medical Informatics , Seoul National University , Seoul , South Korea
-
Hong Kong University of Science and Technology
Affiliation as printed
Department of Computer Science and Engineering , The Hong Kong University of Science and Technology , Hong Kong , China
-
Technion – Israel Institute of Technology
Affiliation as printed
The Faculty of Data and Decisions Science , Technion -Israel Institute of Technology , Haifa , Israel
-
University College London · The London College
Affiliation as printed
UCL Hawkes Institute , University College London , London , England
-
Hong Kong University of Science and Technology
Affiliation as printed
Department of Computer Science and Engineering , The Hong Kong University of Science and Technology , Hong Kong , China
-
Affiliation as printed
Graduate School of Informatics , Nagoya University , Nagoya , Japan
-
Technische Universität Dresden · National Center for Tumor Diseases · Nationales Centrum für Tumorerkrankungen Dresden · Centre for Tactile Internet with Human-in-the-Loop
Affiliation as printed
Department of Translational Surgical Oncology , National Center for Tumor Diseases (NCT/UCC) Dresden , , Dresden , Germany
The Centre for Tactile Internet with Human-in-the-Loop (CeTI) , TUD Dresden University of Technology , , Dresden , Germany
Cited by 0 stored of 0
No patents citing this paper on Lens.org (checked 2026-10-06).