Vision-Language Models as Zero-Annotation Oracles in Histopathology

arXiv (Cornell University)

Abstract

Foreground segmentation is the critical first step of every computational pathology pipeline, yet existing methods rely on hand-tuned heuristics or supervised models that overfit to narrow stain and scanner distributions, failing silently on specialised stains such as Jones silver or Elastica van Gieson. We propose a coarse-to-fine approach that recasts foreground segmentation as a visual perception task and leverages general-purpose vision-language models (VLMs) as zero-annotation oracles. Our key insight is that tissue-versus-background discrimination is a natural-image recognition problem, not a histopathological one, so VLMs trained on internet-scale corpora generalise where domain-specific models cannot. We introduce Leica-75, a benchmark of 75 renal transplant whole-slide images spanning three stain families. On Leica-75, our method achieves the highest segmentation quality on out-of-distribution stains (Dice 0.858 +/- 0.027 on Jones, 0.853 +/- 0.041 on EVG) with 7x lower cross-stain variance than the best supervised baseline, while remaining competitive on in-distribution H&E. Few-shot prompting with automatically curated exemplars (Auto-context) rescues hard cases on Stress-32 (n=32), a curated stress-test subset (Dice 0.470 to 0.819 for the 2B model). VLM-based annotation review matches human expert consensus (kappa=0.989 for blur detection; mean precision/recall grading accuracy 0.708 vs. human 0.646 for segmentation mask review). The resulting pseudo-labels are used to distil lightweight student models that are as performant as the teacher model while running for a fraction of the cost. Our framework provides a principled, scalable solution to a persistent infrastructure bottleneck in digital pathology.

Authors 9

1. Vishal Jain

   Imperial College London

   Affiliation as printed

   Imperial College London , London , UK

2. Giorgio Buzzanca Aachen

   Leiden University Medical Center

   Affiliation as printed

   Leiden University Medical Center , Leiden , The Netherlands

3. Sarah Cechnicka

   Imperial College London

   Affiliation as printed

   Imperial College London , London , UK

4. Maarten Naesens

   KU Leuven

   Affiliation as printed

   KU Leuven , Leuven , Belgium

5. Priyanka Joseph Koshy

   Universitair Ziekenhuis Leuven

   Affiliation as printed

   University Hospitals Leuven , Leuven , Belgium

6. Tri Nguyen

   University Medical Center Utrecht

   Affiliation as printed

   University Medical Center Utrecht , Utrecht , The Netherlands

7. Jesper Kers Aachen

   Leiden University Medical Center

   Affiliation as printed

   Leiden University Medical Center , Leiden , The Netherlands

8. Candice Roufosse

   Imperial College London

   Affiliation as printed

   Imperial College London , London , UK

9. Bernhard Kainz

   Friedrich-Alexander-Universität Erlangen-Nürnberg · Imperial College London

   Affiliation as printed

   Dept. AIBE , Friedrich-Alexander University Erlangen- Nürnberg , Germany

   Imperial College London , London , UK