Benchmarking foundation models as feature extractors for weakly-supervised computational pathology

arXiv (Cornell University)

Abstract

Advancements in artificial intelligence have driven the development of numerous pathology foundation models capable of extracting clinically relevant information. However, there is currently limited literature independently evaluating these foundation models on truly external cohorts and clinically-relevant tasks to uncover adjustments for future improvements. In this study, we benchmarked 19 histopathology foundation models on 13 patient cohorts with 6,818 patients and 9,528 slides from lung, colorectal, gastric, and breast cancers. The models were evaluated on weakly-supervised tasks related to biomarkers, morphological properties, and prognostic outcomes. We show that a vision-language foundation model, CONCH, yielded the highest performance when compared to vision-only foundation models, with Virchow2 as close second. The experiments reveal that foundation models trained on distinct cohorts learn complementary features to predict the same label, and can be fused to outperform the current state of the art. An ensemble combining CONCH and Virchow2 predictions outperformed individual models in 55% of tasks, leveraging their complementary strengths in classification scenarios. Moreover, our findings suggest that data diversity outweighs data volume for foundation models. Our work highlights actionable adjustments to improve pathology foundation models.

Authors 16

1. Peter Neidlinger
   Affiliation as printed

   . Else Kroener Fresenius Center for Digital Health , Technical University Dresden , Dresden , Germany

2. Omar S. M. El Nahhas
   Affiliation as printed

   . Else Kroener Fresenius Center for Digital Health , Technical University Dresden , Dresden , Germany

   . StratifAI GmbH , Dresden , Germany

3. Hannah Sophie Muti
4. Tim Lenz
   Affiliation as printed

   . Else Kroener Fresenius Center for Digital Health , Technical University Dresden , Dresden , Germany

5. Michael Hoffmeister

   German Cancer Research Center

   Affiliation as printed

   . Division of Clinical Epidemiology and Aging Research , German Cancer Research Center (DKFZ) , Heidelberg , Germany

6. Hermann Brenner
7. Marko van Treeck
   Affiliation as printed

   . Else Kroener Fresenius Center for Digital Health , Technical University Dresden , Dresden , Germany

8. Rupert Langer

   Johannes Kepler University of Linz · Kepler Universitätsklinikum

   Affiliation as printed

   . Institute of Pathology and Molecular Pathology , Kepler University Hospital , Johannes Kepler University Linz , Linz , Austria

9. Bastian Dislich

   University of Bern

   Affiliation as printed

   . Institute of Tissue Medicine and Pathology , University of Bern , Bern , Switzerland

10. H.-M. Behrens

    University Hospital Schleswig-Holstein · University of Lübeck

    Affiliation as printed

    . Department of Pathology , University Hospital Schleswig-Holstein , Kiel , Germany

11. Christoph Röcken
12. Sebastian Foersch

    University Medical Center of the Johannes Gutenberg University Mainz

    Affiliation as printed

    . Institute of Pathology , University Medical Center Mainz , Mainz , Germany

13. Daniel Truhn Aachen Department of Diagnostic and Interventional Radiology

    Universitätsklinikum Aachen

    Affiliation as printed

    . Department of Diagnostic and Interventional Radiology , University Hospital Aachen

    . StratifAI GmbH , Dresden , Germany

14. Antonio Marra
15. Oliver Lester Saldanha
    Affiliation as printed

    . Else Kroener Fresenius Center for Digital Health , Technical University Dresden , Dresden , Germany

16. Jakob Nikolas Kather