Prompt Injection Attacks on Large Language Models in Oncology

arXiv (Cornell University)

Abstract

Vision-language artificial intelligence models (VLMs) possess medical knowledge and can be employed in healthcare in numerous ways, including as image interpreters, virtual scribes, and general decision support systems. However, here, we demonstrate that current VLMs applied to medical tasks exhibit a fundamental security flaw: they can be attacked by prompt injection attacks, which can be used to output harmful information just by interacting with the VLM, without any access to its parameters. We performed a quantitative study to evaluate the vulnerabilities to these attacks in four state of the art VLMs which have been proposed to be of utility in healthcare: Claude 3 Opus, Claude 3.5 Sonnet, Reka Core, and GPT-4o. Using a set of N=297 attacks, we show that all of these models are susceptible. Specifically, we show that embedding sub-visual prompts in medical imaging data can cause the model to provide harmful output, and that these prompts are non-obvious to human observers. Thus, our study demonstrates a key vulnerability in medical VLMs which should be mitigated before widespread clinical adoption.

Authors 8

1. Jan Clusmann Aachen Department of Medicine III

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   . Department of Medicine III , University Hospital RWTH Aachen , Aachen , Germany

   . Else Kroener Fresenius Center for Digital Health , Technical University Dresden , Dresden , Germany

2. Dyke Ferber
3. Isabella C. Wiest

   Heidelberg University · Medizinische Fakultät Mannheim

   Affiliation as printed

   . Department of Medicine II , Medical Faculty Mannheim , Heidelberg Uni- versity , Mannheim , Germany

   . Else Kroener Fresenius Center for Digital Health , Technical University Dresden , Dresden , Germany

4. Carolin Victoria Schneider Aachen Department of Medicine III

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   . Department of Medicine III , University Hospital RWTH Aachen , Aachen , Germany

5. Titus Josef Brinker

   German Cancer Research Center · Heidelberg University

   Affiliation as printed

   . Digital Biomarkers for Oncology Group , German Cancer Research Center , Heidelberg , Germany

6. Sebastian Foersch

   University Medical Center of the Johannes Gutenberg University Mainz

   Affiliation as printed

   . Institute of Pathology , University Medical Center Mainz , Mainz , Germany

7. Daniel Truhn Aachen Department of Diagnostic and Interventional Radiology

   Universitätsklinikum Aachen

   Affiliation as printed

   . Department of Diagnostic and Interventional Radiology , University Hospital Aachen , Germany

8. Jakob Nikolas Kather

   Artificial Intelligence in Medicine (Canada) · Else Kröner Fresenius Center for Digital Health · Technische Universität Dresden

   Affiliation as printed

   Fresenius Center for Digital Health Technische Universität Dresden DE-01062 Dresden

   Professor of Clinical Artificial Intelligence