AI-based burn image assessment: Reliability and clinical error patterns of multimodal large language models in a repeated-inference study

Journal of Plastic Reconstructive & Aesthetic Surgery, vol. 119, pp. 366–374

Abstract

Accurate assessment of burn depth and total body surface area (TBSA) is critical for clinical decision-making; however, it remains subjective and prone to interobserver variability. Multimodal large language models (MLLMs) are increasingly encountered in clinical contexts, but whether these systems can reliably assess burn images remains unclear. We evaluated four MLLMs (GPT-5.4 Pro, Grok 4.1, Gemini 3.1 Pro, and Claude Opus 4.6) on 50 clinical burn photographs using a repeated-inference design with five independent runs per model. Burn depth classification was assessed in numeric and text-based formats, alongside ordinal TBSA estimation. Performance varied across the models, with burn depth accuracy ranging from 34.0 ± 6.5% to 76.4 ± 6.8% and TBSA accuracy from 32.8 ± 9.4% to 68.4 ± 3.3%. Inter-run reliability (Fleiss' κ) ranged from slight (κ = 0.171) to almost perfect (κ = 0.916), demonstrating response variability not captured by single-query evaluations. Notably, no model combined high accuracy and high reliability, indicating a dissociation between performance and consistency. All models showed a tendency toward overestimation of burn depth, including assignment of fourth-degree burns despite their absence in the dataset. Error direction analysis revealed model-specific and task-dependent biases, including opposing patterns within the same model. Internal consistency between numeric and text classifications was near-perfect (99.6-100%), indicating format-invariant but systematically biased outputs. These findings demonstrate that MLLM performance is characterized by stochastic response instability invisible to single-query evaluations. Such inconsistency for identical inputs represents a fundamental limitation for workflows requiring consistent outputs across repeated evaluations.

Authors 4

1. Ibrahim Güler corresponding

   Friedrich-Alexander-Universität Erlangen-Nürnberg · University Hospital Magdeburg

   Affiliation as printed

   Department of Plastic, Aesthetic and Hand Surgery, University Hospital Magdeburg, Leipziger Strasse 44, 39120 Magdeburg, Germany; Department of Health Management, Friedrich-Alexander-Universität Erlangen-Nürnberg FAU, Lange Gasse 20, 90403 Nürnberg, Germany. Electronic address: ibrahim.gueler@med.ovgu.de

2. Armin Kraus corresponding

   University Hospital Magdeburg

   Affiliation as printed

   Department of Plastic, Aesthetic and Hand Surgery, University Hospital Magdeburg, Leipziger Strasse 44, 39120 Magdeburg, Germany. Electronic address: armin.kraus@med.ovgu.de

3. Gerrit Grieb corresponding Aachen Department of Plastic Surgery and Hand Surgery Medical Faculty

   Gemeinschaftskrankenhaus Havelhöhe · RWTH Aachen University

   Affiliation as printed

   Department of Plastic Surgery and Hand Surgery, Gemeinschaftskrankenhaus Havelhöhe, Berlin, Germany; Department of Plastic Surgery and Hand Surgery, Medical Faculty, RWTH Aachen University, Aachen, Germany. Electronic address: gerritgrieb@gmx.de

4. Henrik Stelling corresponding

   Friedrich-Alexander-Universität Erlangen-Nürnberg

   Affiliation as printed

   Department of Health Management, Friedrich-Alexander-Universität Erlangen-Nürnberg FAU, Lange Gasse 20, 90403 Nürnberg, Germany; Practices for Nuclear Medicine, Rubensstraße 125, 12157 Berlin, Germany. Electronic address: henrik.stelling@fau.de