Validating large language model–assisted data extraction from clinical notes

ESMO Real World Data and Digital Oncology, vol. 12, pp. 100718

Abstract

Background Health care professionals face increasing documentation burdens, which can compromise efficiency and patient safety. Large language models (LLMs) may offer a scalable solution by automating data extraction from unstructured clinical notes. This study evaluates the accuracy and clinical impact of structured data extraction by an LLM compared with manual extraction by physicians in the context of head and neck oncology consultations. Patients and methods This was a prospective validation study comparing LLM-powered data extraction with manual extraction. Clinical documentation from 60 patients (1482 pages) was analyzed. Data were extracted for 29 clinically relevant categories by two physicians and a pretrained open-source LLM. Six clinical experts evaluated 2555 extracted values in a two-step procedure: first, a blinded binary Match/Non-Match assessment between LLM output and human consensus; and second, categorization of all Non-Matches into predefined error types ( Incorrect , Incomplete , Missing , Hallucination , or Overcomplete ). These expert-assigned labels formed the basis for accuracy, precision, recall, and F1 scores. Extraction times were compared using a paired Student's t -test, and error impact was scored on a 5-point Likert scale. Results LLM-powered extraction achieved accuracies between 74% [pathology: 95% confidence interval (CI) 65% to 82%] and 90% (patient characteristics: 95% CI 87% to 94%). Manual extraction showed 29% interobserver disagreement (95% CI 25.19% to 32.74%). Of 2555 extracted values, 68 were rated as high-impact errors, although evaluator assessments varied widely. Hallucinations were rare (0.16%) and low impact. LLM extraction reduced average time per case from 8.6 minutes to 1.9 minutes ( P < 0.001). Conclusion LLMs can support clinical workflows by reducing documentation time and maintaining acceptable accuracy, provided that human oversight is ensured. These findings support further exploration of AI-assisted documentation tools in clinical practice.

Authors 13

1. J.W. Van Koevorden corresponding Aachen Department of Biomedical Data Sciences

   Leiden University Medical Center · The Netherlands Cancer Institute

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands

   Department of Head and Neck Surgery, Antoni van Leeuwenhoek-Netherlands Cancer Institute, Amsterdam, the Netherlands

2. N. Aben
   Affiliation as printed

   Kaiko.ai, Amsterdam, the Netherlands

3. V. Struben
   Affiliation as printed

   Kaiko.ai, Amsterdam, the Netherlands

4. A. Rollings
   Affiliation as printed

   Kaiko.ai, Amsterdam, the Netherlands

5. M. Kurban
   Affiliation as printed

   Kaiko.ai, Amsterdam, the Netherlands

6. L. Wessels

   The Netherlands Cancer Institute

   Affiliation as printed

   Department of Molecular Carcinogenesis, Antoni van Leeuwenhoek-Netherlands Cancer Institute, Amsterdam, the Netherlands

7. V. van der Noort

   The Netherlands Cancer Institute

   Affiliation as printed

   Department of Biometrics, Antoni van Leeuwenhoek-Netherlands Cancer Institute, Amsterdam, the Netherlands

8. A.V.M. Burger

   The Netherlands Cancer Institute

   Affiliation as printed

   Department of Head and Neck Surgery, Antoni van Leeuwenhoek-Netherlands Cancer Institute, Amsterdam, the Netherlands

9. S.W. Dijk

   The Netherlands Cancer Institute

   Affiliation as printed

   Department of Head and Neck Surgery, Antoni van Leeuwenhoek-Netherlands Cancer Institute, Amsterdam, the Netherlands

10. L. Karssemakers

    The Netherlands Cancer Institute

    Affiliation as printed

    Department of Head and Neck Surgery, Antoni van Leeuwenhoek-Netherlands Cancer Institute, Amsterdam, the Netherlands

11. L. Smeele

    The Netherlands Cancer Institute

    Affiliation as printed

    Department of Head and Neck Surgery, Antoni van Leeuwenhoek-Netherlands Cancer Institute, Amsterdam, the Netherlands

12. M.W.J.M. WOUTERS Aachen Department of Biomedical Data Sciences Dutch Institute for Clinical Auditing

    Leiden University Medical Center · The Netherlands Cancer Institute · Dutch Institute for Clinical Auditing

    Affiliation as printed

    Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands

    Department of Surgical Oncology, Antoni van Leeuwenhoek-Netherlands Cancer Institute, Amsterdam, the Netherlands

    Dutch Institute for Clinical Auditing, Leiden, the Netherlands

13. R. Dirven

    Radboud University Nijmegen · Radboud University Medical Center · The Netherlands Cancer Institute

    Affiliation as printed

    Department of Head and Neck Surgery, Antoni van Leeuwenhoek-Netherlands Cancer Institute, Amsterdam, the Netherlands

    Department of Otorhinolaryngology and Head and Neck Surgery, Radboud University Medical Center, Nijmegen, the Netherlands