Performance of Large Language Models on Cognitive Aptitude Testing: A Multi-Run Evaluation on the German Medical School Admission Test (TMS)

European Journal of Investigation in Health Psychology and Education, vol. 16, pp. 23

Abstract

BACKGROUND AND OBJECTIVES: Large language models (LLMs) have demonstrated high performance on knowledge-based medical examinations but their capabilities on cognitive aptitude tests emphasizing reasoning and abstraction remain underexplored. The Test for Medical Studies (TMS), a German medical school admission test, provides a standardized framework to examine these capabilities. This study aimed to evaluate the performance and consistency of multiple LLMs on text-based and visual-analytic TMS items. MATERIALS AND METHODS: Eight contemporary LLMs, comprising proprietary and open-source systems, were evaluated using a multi-run design on standardized TMS items spanning text-based and visual-analytic cognitive domains. RESULTS: Mean accuracy remained substantially below levels typically reported for knowledge-based medical examinations, with marked performance differences between text-based and visual-analytic subtests. Open-source models performed competitively compared with proprietary systems. Inter-run reliability was heterogeneous, indicating notable variability across repeated evaluations. CONCLUSIONS: Current LLMs show limited and domain-dependent performance on cognitive aptitude tasks relevant to medical school admission. High accuracy on knowledge-based examinations does not translate into stable performance on aptitude tests emphasizing fluid intelligence. The observed modality-dependent performance patterns and inter-run variability highlight the importance of differentiated, multi-run evaluation strategies when assessing LLMs for applications in medical education.

Authors 4

1. Henrik Stelling
   Affiliation as printed

   Practices for Nuclear Medicine, Rubensstraße 125, 12157 Berlin, Germany

2. Armin Kraus

   Otto-von-Guericke-Universität Magdeburg

   Affiliation as printed

   Department of Plastic, Aesthetic and Hand Surgery, Otto-von-Guericke University, 39120 Magdeburg, Germany

3. Gerrit Grieb Aachen Department of Plastic Surgery and Hand Surgery Burn Center Medical Faculty

   Gemeinschaftskrankenhaus Havelhöhe · RWTH Aachen University

   Affiliation as printed

   Department of Plastic Surgery and Hand Surgery, Burn Center, Medical Faculty, RWTH Aachen University, Pauwelsstrasse 30, 52074 Aachen, Germany

   Department of Plastic Surgery and Hand Surgery, Gemeinschaftskrankenhaus Havelhoehe, Kladower Damm 221, 14089 Berlin, Germany

4. Ibrahim Güler corresponding

   Friedrich-Alexander-Universität Erlangen-Nürnberg · Otto-von-Guericke-Universität Magdeburg

   Affiliation as printed

   Department of Health Management, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Lange Gasse 20, 90403 Nürnberg, Germany

   Department of Plastic, Aesthetic and Hand Surgery, Otto-von-Guericke University, 39120 Magdeburg, Germany