In-context learning enables multimodal large language models to classify cancer pathology images
Nature Communications, vol. 15, pp. 10104
Abstract
Medical image classification requires labeled, task-specific datasets which are used to train deep learning networks de novo, or to fine-tune foundation models. However, this process is computationally and technically demanding. In language processing, in-context learning provides an alternative, where models learn from within prompts, bypassing the need for parameter updates. Yet, in-context learning remains underexplored in medical image analysis. Here, we systematically evaluate the model Generative Pretrained Transformer 4 with Vision capabilities (GPT-4V) on cancer image processing with in-context learning on three cancer histopathology tasks of high importance: Classification of tissue subtypes in colorectal cancer, colon polyp subtyping and breast tumor detection in lymph node sections. Our results show that in-context learning is sufficient to match or even outperform specialized neural networks trained for particular tasks, while only requiring a minimal number of samples. In summary, this study demonstrates that large vision language models trained on non-domain specific data can be applied out-of-the box to solve medical image-processing tasks in histopathology. This democratizes access of generalist AI models to medical experts without technical background especially for areas where annotated data is scarce.
Authors 11
-
Heidelberg University · University Hospital Heidelberg · National Center for Tumor Diseases · Else Kröner Fresenius Center for Digital Health
Affiliation as printed
Department of Medical Oncology, Heidelberg University Hospital, Heidelberg, Germany
Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany
-
Affiliation as printed
School of Computer Science, University of St Andrews, St Andrews, UK
-
Heidelberg University · Medizinische Fakultät Mannheim · Else Kröner Fresenius Center for Digital Health
Affiliation as printed
Department of Medicine II, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
-
Else Kröner Fresenius Center for Digital Health
Affiliation as printed
Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
-
Else Kröner Fresenius Center for Digital Health
Affiliation as printed
Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
-
Else Kröner Fresenius Center for Digital Health
Affiliation as printed
Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
-
Else Kröner Fresenius Center for Digital Health
Affiliation as printed
Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
-
Affiliation as printed
Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
-
Heidelberg University · University Hospital Heidelberg · National Center for Tumor Diseases
Affiliation as printed
Department of Medical Oncology, Heidelberg University Hospital, Heidelberg, Germany
National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany
-
Affiliation as printed
Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
-
Jakob Nikolas Kather corresponding
Heidelberg University · University Hospital Heidelberg · National Center for Tumor Diseases · University Hospital Carl Gustav Carus · Else Kröner Fresenius Center for Digital Health · Technische Universität Dresden
Affiliation as printed
Department of Medical Oncology, Heidelberg University Hospital, Heidelberg, Germany. jakob_nikolas.kather@tu-dresden.de
Department of Medicine I, University Hospital Dresden, Dresden, Germany. jakob_nikolas.kather@tu-dresden.de
Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany. jakob_nikolas.kather@tu-dresden.de
National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany. jakob_nikolas.kather@tu-dresden.de
Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
Cited by 156 stored of 156
No patents citing this paper on Lens.org (checked 2026-10-06).