Outcomes, Measurement Instruments, and Their Validity Evidence in Randomized Controlled Trials on Virtual, Augmented, and Mixed Reality in Undergraduate Medical Education: Systematic Mapping Review

JMIR Serious Games, vol. 10, pp. e29594

Abstract

BACKGROUND: Extended reality, which encompasses virtual reality (VR), augmented reality (AR), and mixed reality (MR), is increasingly used in medical education. Studies assessing the effectiveness of these new educational modalities should measure relevant outcomes using outcome measurement tools with validity evidence. OBJECTIVE: Our aim is to determine the choice of outcomes, measurement instruments, and the use of measurement instruments with validity evidence in randomized controlled trials (RCTs) on the effectiveness of VR, AR, and MR in medical student education. METHODS: We conducted a systematic mapping review. We searched 7 major bibliographic databases from January 1990 to April 2020, and 2 reviewers screened the citations and extracted data independently from the included studies. We report our findings in line with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. RESULTS: Of the 126 retrieved RCTs, 115 (91.3%) were on VR and 11 (8.7%) were on AR. No RCT on MR in medical student education was found. Of the 115 studies on VR, 64 (55.6%) were on VR simulators, 30 (26.1%) on screen-based VR, 9 (7.8%) on VR patient simulations, and 12 (10.4%) on VR serious games. Most studies reported only a single outcome and immediate postintervention assessment data. Skills outcome was the most common outcome reported in studies on VR simulators (97%), VR patient simulations (100%), and AR (73%). Knowledge was the most common outcome reported in studies on screen-based VR (80%) and VR serious games (58%). Less common outcomes included participants' attitudes, satisfaction, cognitive or mental load, learning efficacy, engagement or self-efficacy beliefs, emotional state, competency developed, and patient outcomes. At least one form of validity evidence was found in approximately half of the studies on VR simulators (55%), VR patient simulations (56%), VR serious games (58%), and AR (55%) and in a quarter of the studies on screen-based VR (27%). Most studies used assessment methods that were implemented in a nondigital format, such as paper-based written exercises or in-person assessments where examiners observed performance (72%). CONCLUSIONS: RCTs on VR and AR in medical education report a restricted range of outcomes, mostly skills and knowledge. The studies largely report immediate postintervention outcome data and use assessment methods that are in a nondigital format. Future RCTs should include a broader set of outcomes, report on the validity evidence of the measurement instruments used, and explore the use of assessments that are implemented digitally.

Authors 7

1. Lorainne Tudor Car corresponding

   Nanyang Technological University · Imperial College London

   Affiliation as printed

   Department of Primary Care and Public Health, School of Public Health, Imperial College London, London, United Kingdom

   Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore, Singapore

2. Bhone Myint Kyaw

   Nanyang Technological University

   Affiliation as printed

   Centre for Population Health Sciences, Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore, Singapore

3. Andrew Teo

   Nanyang Technological University

   Affiliation as printed

   Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore, Singapore

4. Tatiana Erlikh Fox

   Nanyang Technological University · OLVG

   Affiliation as printed

   Centre for Population Health Sciences, Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore, Singapore

   Department of Internal Medicine, Onze Lieve Vrouwen Gasthuis, Amsterdam, Netherlands

5. Sunitha Vimalesvaran

   Nanyang Technological University

   Affiliation as printed

   Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore, Singapore

6. Christian Apfelbacher

   Nanyang Technological University · Otto-von-Guericke-Universität Magdeburg

   Affiliation as printed

   Family Medicine and Primary Care, Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore, Singapore

   Institute of Social Medicine and Health Systems Research, Otto von Guericke University Magdeburg, Magdegurg, Germany

7. Niels Henrik Chavannes Aachen Department of Public Health and Primary Care (PHEG)

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Public Health and Primary Care, Leiden University Medical Centre, Leiden, Netherlands