Benchmarking AI Vibe Coding for Clinical Statistical Analysis: A Structured Evaluation in Pulmonary Hypertension Research

Studies in health technology and informatics, vol. 340, pp. 234–241

Abstract

Statistical analysis of clinical data requires expertise in medical statistics. Large language models (LLMs) are increasingly used for code generation and may support both descriptive and advanced analyses, but their reliability remains uncertain. This study evaluated whether five current LLMs (GPT 5.3, Claude Sonnet 4.6, Gemini 2.5 Flash, Perplexity, and Grok) could reproduce expert-validated statistical analyses from a published clinical workflow across three statistical tasks: descriptive table generation, Kaplan-Meier survival analysis, and Cox proportional hazards modelling. All models received identical datasets and standardised prompts. Their outputs were compared with analyses performed by two experts trained in mathematical statistics and evaluated for grouping correctness, numerical accuracy, missing-value handling, and quality of generated R code. All five models produced correct descriptive statistics once dataset variables were specified explicitly. Two models failed the initial descriptive benchmark because of variable-name ambiguity, but both recovered after prompt clarification. All models reproduced the correct Kaplan-Meier p-value, although figure completeness differed. In the Cox benchmark, four models reproduced all required hazard-ratio terms, while one omitted the interaction results. These findings suggest that LLMs may support clinical research workflows, but their outputs still require careful validation before use.

Authors 8

1. Jeeva Sam

   Universities of Giessen and Marburg Lung Center · Cardio-Pulmonary Institute

   Affiliation as printed

   Cardio-Pulmonary Institute (CPI), Institute for Lung Health (ILH), Gießen, Germany

   Universities of Giessen and Marburg Lung Center, Department of Internal Medicine, Gießen, Germany

2. T. Spreuer

   Universities of Giessen and Marburg Lung Center · Cardio-Pulmonary Institute

   Affiliation as printed

   Cardio-Pulmonary Institute (CPI), Institute for Lung Health (ILH), Gießen, Germany

   Universities of Giessen and Marburg Lung Center, Department of Internal Medicine, Gießen, Germany

3. Athiththan Yogeswaran

   Universities of Giessen and Marburg Lung Center · Cardio-Pulmonary Institute

   Affiliation as printed

   Cardio-Pulmonary Institute (CPI), Institute for Lung Health (ILH), Gießen, Germany

   Universities of Giessen and Marburg Lung Center, Department of Internal Medicine, Gießen, Germany

4. Khodr Tello

   Universities of Giessen and Marburg Lung Center · Cardio-Pulmonary Institute

   Affiliation as printed

   Cardio-Pulmonary Institute (CPI), Institute for Lung Health (ILH), Gießen, Germany

   Universities of Giessen and Marburg Lung Center, Department of Internal Medicine, Gießen, Germany

5. Patrick Janetzko

   Universities of Giessen and Marburg Lung Center · Cardio-Pulmonary Institute

   Affiliation as printed

   Cardio-Pulmonary Institute (CPI), Institute for Lung Health (ILH), Gießen, Germany

   Universities of Giessen and Marburg Lung Center, Department of Internal Medicine, Gießen, Germany

6. Werner Seeger

   Universities of Giessen and Marburg Lung Center · Cardio-Pulmonary Institute

   Affiliation as printed

   Cardio-Pulmonary Institute (CPI), Institute for Lung Health (ILH), Gießen, Germany

   Universities of Giessen and Marburg Lung Center, Department of Internal Medicine, Gießen, Germany

7. Jonas Bienzeisler Aachen Institute of Medical Informatics

   RWTH Aachen University

   Affiliation as printed

   Institute of Medical Informatics, RWTH Aachen University Medical Faculty, Aachen, Germany

8. Raphael W. Majeed Aachen Institute of Medical Informatics

   RWTH Aachen University · Universities of Giessen and Marburg Lung Center · Cardio-Pulmonary Institute

   Affiliation as printed

   Cardio-Pulmonary Institute (CPI), Institute for Lung Health (ILH), Gießen, Germany

   Institute of Medical Informatics, RWTH Aachen University Medical Faculty, Aachen, Germany

   Universities of Giessen and Marburg Lung Center, Department of Internal Medicine, Gießen, Germany