Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology

Nature Cancer, vol. 6, pp. 1337–1349

Abstract

Clinical decision-making in oncology is complex, requiring the integration of multimodal data and multidomain expertise. We developed and evaluated an autonomous clinical artificial intelligence (AI) agent leveraging GPT-4 with multimodal precision oncology tools to support personalized clinical decision-making. The system incorporates vision transformers for detecting microsatellite instability and KRAS and BRAF mutations from histopathology slides, MedSAM for radiological image segmentation and web-based search tools such as OncoKB, PubMed and Google. Evaluated on 20 realistic multimodal patient cases, the AI agent autonomously used appropriate tools with 87.5% accuracy, reached correct clinical conclusions in 91.0% of cases and accurately cited relevant oncology guidelines 75.5% of the time. Compared to GPT-4 alone, the integrated AI agent drastically improved decision-making accuracy from 30.3% to 87.2%. These findings demonstrate that integrating language models with precision oncology and search tools substantially enhances clinical accuracy, establishing a robust foundation for deploying AI-driven personalized oncology support systems.

Authors 14

1. Dyke Ferber

   Heidelberg University · University Hospital Heidelberg · National Center for Tumor Diseases · Else Kröner Fresenius Center for Digital Health

   Affiliation as printed

   Department of Medical Oncology, National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany

   Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany

2. Omar S. M. El Nahhas

   Else Kröner Fresenius Center for Digital Health

   Affiliation as printed

   Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany

3. Georg Wölflein

   University of St Andrews

   Affiliation as printed

   School of Computer Science, University of St Andrews, St Andrews, UK

4. Isabella C. Wiest

   Heidelberg University · Medizinische Fakultät Mannheim · Else Kröner Fresenius Center for Digital Health

   Affiliation as printed

   Department of Medicine II, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany

   Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany

5. Jan Clusmann Aachen Department of Medicine III

   Universitätsklinikum Aachen · Else Kröner Fresenius Center for Digital Health · RWTH Aachen University

   Affiliation as printed

   Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany

   Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany

6. Marie-Elisabeth Leßmann

   University Hospital Carl Gustav Carus · Else Kröner Fresenius Center for Digital Health

   Affiliation as printed

   Department of Medicine I, University Hospital Dresden, Dresden, Germany

   Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany

7. Sebastian Foersch

   University Medical Center of the Johannes Gutenberg University Mainz

   Affiliation as printed

   Institute of Pathology, University Medical Center Mainz, Mainz, Germany

8. Jacqueline Lammert

   TUM Klinikum · Deutsches Konsortium für Translationale Krebsforschung · Technical University of Munich

   Affiliation as printed

   Center for Personalized Medicine (ZPM), University Hospital rechts der Isar, Technical University of Munich (TUM), Munich, Germany

   Department of Gynecology and Center for Hereditary Breast and Ovarian Cancer, University Hospital rechts der Isar, Technical University of Munich (TUM), Munich, Germany

   European Network for Rare Cancers (EURACAN) Initiative, Munich, Germany

   German Cancer Consortium (DKTK), Munich, Germany

9. Maximilian Tschochohei

   Ludwig-Maximilians-Universität München

   Affiliation as printed

   Google Cloud, Munich, Germany

10. Dirk Jaeger

    Heidelberg University · University Hospital Heidelberg · National Center for Tumor Diseases

    Affiliation as printed

    Department of Medical Oncology, National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany

11. Manuel Salto‐Tellez

    Royal Marsden Hospital

    Affiliation as printed

    Integrated Pathology Unit, Institute for Cancer Research and Royal Marsden Hospital, London, UK

12. Nikolaus D. Schultz

    Memorial Sloan Kettering Cancer Center

    Affiliation as printed

    Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, NY, USA

13. Daniel Truhn Aachen Department of Diagnostic and Interventional Radiology

    Universitätsklinikum Aachen

    Affiliation as printed

    Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany

14. Jakob Nikolas Kather corresponding

    Heidelberg University · University Hospital Heidelberg · National Center for Tumor Diseases · Else Kröner Fresenius Center for Digital Health · Technische Universität Dresden

    Affiliation as printed

    Department of Medical Oncology, National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany. jakob-nikolas.kather@alumni.dkfz.de

    Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany. jakob-nikolas.kather@alumni.dkfz.de