Artificial Intelligence in Medical Assessment: Reliability and Performance of Multimodal Large Language Models in a High-Stakes Licensing Examination

Behavioral Sciences, vol. 16, pp. 822

Abstract

Artificial intelligence (AI) is increasingly integrated into assessment contexts, yet evidence on the reliability and measurement properties of large language models (LLMs) in high-stakes evaluation settings remains limited. This study examines the performance and reproducibility of contemporary multimodal LLMs in a structured medical assessment environment. A cross-sectional dual-setup design was applied using a complete national medical licensing examination (240 multiple-choice items, including image-based questions). Setup 1 evaluated ten models in a single run to characterize overall performance. Setup 2 assessed six models across five independent runs each to quantify measurement stability. Accuracy with 95% confidence intervals, inter-run agreement using Cohen's kappa, and paired comparisons using McNemar's test were analyzed. Accuracy ranged from 72.08% to 92.92%. All models demonstrated near-perfect inter-run agreement (mean κ ≥ 0.96) with minimal variability. After correction, only a small number of pairwise comparisons remained significant, indicating convergence among leading systems. In an exploratory submodule, performance on the small set of image-based items was comparable to or slightly higher than performance on text-only items. These findings demonstrate that multimodal LLMs achieve high accuracy and high inter-run reproducibility on a large-scale assessment, supporting their use as objects of AI-based assessment research while leaving questions of cognitive equivalence with human examinees beyond the scope of accuracy-based evaluation.

Authors 7

1. Ibrahim Güler

   Friedrich-Alexander-Universität Erlangen-Nürnberg · Otto-von-Guericke-Universität Magdeburg · Gemeinschaftskrankenhaus Havelhöhe

   Affiliation as printed

   Department of Health Management, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Lange Gasse 20, 90403 Nürnberg, Germany

   Department of Plastic Surgery and Hand Surgery, Gemeinschaftskrankenhaus Havelhöhe, Kladower Damm 221, 14089 Berlin, Germany

   Department of Plastic, Aesthetic and Hand Surgery, Otto-von-Guericke University, 39120 Magdeburg, Germany

2. Gerrit Grieb Aachen Department of Plastic Surgery and Hand Surgery

   RWTH Aachen University · Gemeinschaftskrankenhaus Havelhöhe

   Affiliation as printed

   Department of Plastic Surgery and Hand Surgery, Gemeinschaftskrankenhaus Havelhöhe, Kladower Damm 221, 14089 Berlin, Germany

   Department of Plastic Surgery and Hand Surgery, RWTH Aachen University, Pauwelsstrasse 30, 52074 Aachen, Germany

3. Armin Kraus

   Otto-von-Guericke-Universität Magdeburg

   Affiliation as printed

   Department of Plastic, Aesthetic and Hand Surgery, Otto-von-Guericke University, 39120 Magdeburg, Germany

4. Philipp Moog

   Technical University of Munich · TUM Klinikum

   Affiliation as printed

   Department of Plastic Surgery and Hand Surgery, Klinikum Rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675 Munich, Germany

5. Uzay Cambaz

   University of Tübingen

   Affiliation as printed

   Faculty of Medicine, Eberhard Karls University of Tübingen, Geschwister-Scholl-Platz, 72074 Tübingen, Germany

6. Ezgi Yavasca

   Klinikum Ernst von Bergmann

   Affiliation as printed

   Department of Nuclear Medicine, Klinikum Ernst von Bergmann, 14467 Potsdam, Germany

7. Henrik Stelling

   Friedrich-Alexander-Universität Erlangen-Nürnberg · Klinikum Ernst von Bergmann

   Affiliation as printed

   Department of Health Management, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Lange Gasse 20, 90403 Nürnberg, Germany

   Department of Nuclear Medicine, Klinikum Ernst von Bergmann, 14467 Potsdam, Germany

   Practices for Nuclear Medicine, Rubensstraße 125, 12157 Berlin, Germany