From XR-derived performance metrics to an objective performance score in point-of-care ultrasound

BMC Medical Education

Abstract

Point-of-care ultrasound (POCUS) quality depends on operator proficiency. As POCUS training expands, scalable approaches to performance evaluation are needed. Current performance evaluation relies largely on examiner-intensive Objective Structured Clinical Examinations (OSCE). Extended Reality (XR) technologies offer opportunities to capture objective performance metrics and provide the digitization needed to support automated and standardised performance evaluations. The objective of this study was to explore the use of XR-derived performance metrics and examine how an automated performance score using these metrics relates to OSCE performance. Medical students (novices), physicians with limited ultrasound experience (intermediates), and experienced physicians and sonographers (experts) performed a cardiac ultrasound task on a phantom, while using a phone as a probe and wearing a HoloLens that immerses them in an XR environment. They were tasked to obtain four cardiac POCUS views. OSCE was scored by two independent experts. Probe trajectories were derived from the phone and were used to extract kinematic metrics, such as time, path length, straightness index, and dimensionless squared jerk. These were complemented with image quality surrogates (position and angle deviation), and gaze behaviour (gaze shifts to the display). A metric-based performance score was constructed using multivariable regression and compared with OSCE scores. Ninety participants were included (34 novices, 31 intermediates, 25 experts). The OSCE scoring demonstrated excellent inter-rater reliability (ICC 0.97) and clear discrimination across experience levels. Kinematic and gaze metrics aligned with experience level and OSCE performance, whereas image quality surrogates showed weaker associations. Factor analysis of the XR-derived performance metrics identified two underlying domains of technical performance; probe handling and image quality, with gaze behaviour emerging as a distinct construct. The automated performance score showed a moderate to good correlation with OSCE performance (ρ = 0.69). Replacing XR-derived image quality surrogates with expert-rated image quality substantially improved associations (ρ = 0.91). XR-derived, automated performance scoring represents a promising, scalable approach for formative feedback and early-stage screening, particularly in novice learners, while reducing examiner burden. These findings lay the foundation for further development toward fully automated ultrasound performance tracking. Further validation is required before use in high-stakes or fully automated assessment.

Authors 10

1. Tessa A. Mulder corresponding Aachen Department of Internal Medicine

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Internal Medicine, Leiden University Medical Centre, Albinusdreef 2, Leiden, 2333 ZA, The Netherlands

2. Déan van Tuil Aachen Interventional Molecular Imaging Laboratory

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Interventional Molecular Imaging Laboratory, Leiden University Medical Centre, Leiden, The Netherlands

3. Kateryna Pirkovets Aachen Interventional Molecular Imaging Laboratory

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Interventional Molecular Imaging Laboratory, Leiden University Medical Centre, Leiden, The Netherlands

4. Youssra Khanfour Aachen Department of Internal Medicine

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Internal Medicine, Leiden University Medical Centre, Albinusdreef 2, Leiden, 2333 ZA, The Netherlands

5. Madelien Veronique Regeer Aachen Department of Cardiology, Leiden University Medical Centre

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Cardiology, Leiden University Medical Centre, Leiden, The Netherlands

6. Martijn P. Bauer Aachen Department of Internal Medicine

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Internal Medicine, Leiden University Medical Centre, Albinusdreef 2, Leiden, 2333 ZA, The Netherlands

7. Alexandr Šrámek Aachen Centre for Innovation in Medical Education Department of Radiology, Leiden University Medical Center

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Centre for Innovation in Medical Education, Leiden University Medical Centre, Leiden, The Netherlands

   Department of Radiology, Leiden University Medical Centre, Leiden, The Netherlands

8. Fijs W. B. van Leeuwen Aachen Interventional Molecular Imaging Laboratory

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Interventional Molecular Imaging Laboratory, Leiden University Medical Centre, Leiden, The Netherlands

9. Beerend P. Hierck

   Utrecht University

   Affiliation as printed

   Department of Clinical Sciences-Anatomy and Physiology, Veterinary Medicine Faculty, Utrecht University, Utrecht, The Netherlands

   SkillsLab, Educational Centre, Veterinary Medicine Faculty, Utrecht University, Utrecht, The Netherlands

10. Alexandra M. J. Langers Aachen Centre for Innovation in Medical Education Department of Gastroenterology and Hepatology

    Leiden University · Leiden University Medical Center

    Affiliation as printed

    Centre for Innovation in Medical Education, Leiden University Medical Centre, Leiden, The Netherlands

    Department of Gastroenterology and Hepatology, Leiden University Medical Centre, Leiden, The Netherlands