Recommendations on compiling test datasets for evaluating artificial intelligence solutions in pathology

Modern Pathology, vol. 35, pp. 1759–1769

Abstract

Artificial intelligence (AI) solutions that automatically extract information from digital histology images have shown great promise for improving pathological diagnosis. Prior to routine use, it is important to evaluate their predictive performance and obtain regulatory approval. This assessment requires appropriate test datasets. However, compiling such datasets is challenging and specific recommendations are missing. A committee of various stakeholders, including commercial AI developers, pathologists, and researchers, discussed key aspects and conducted extensive literature reviews on test datasets in pathology. Here, we summarize the results and derive general recommendations on compiling test datasets. We address several questions: Which and how many images are needed? How to deal with low-prevalence subsets? How can potential bias be detected? How should datasets be reported? What are the regulatory requirements in different countries? The recommendations are intended to help AI developers demonstrate the utility of their products and to help pathologists and regulatory agencies verify reported performance measures. Further research is needed to formulate criteria for sufficiently representative test datasets so that AI solutions can operate with less user intervention and better support diagnostic workflows in the future.

Authors 30

1. André Homeyer corresponding

   Fraunhofer Institute for Digital Medicine

   Affiliation as printed

   Fraunhofer Institute for Digital Medicine MEVIS, Max-von-Laue-Straße 2, 28359, Bremen, Germany. andre.homeyer@mevis.fraunhofer.de

   Fraunhofer Institute for Digital Medicine MEVIS, Max-von-Laue-Straße 2, 28359, Bremen, Germany

2. Christian Geißler

   Technische Universität Berlin

   Affiliation as printed

   Technische Universität Berlin, DAI-Labor, Ernst-Reuter-Platz 7, 10587, Berlin, Germany

3. Lars Ole Schwen

   Fraunhofer Institute for Digital Medicine

   Affiliation as printed

   Fraunhofer Institute for Digital Medicine MEVIS, Max-von-Laue-Straße 2, 28359, Bremen, Germany

4. Falk Zakrzewski

   University Hospital Carl Gustav Carus · Technische Universität Dresden

   Affiliation as printed

   Institute of Pathology, Carl Gustav Carus University Hospital Dresden (UKD), TU Dresden (TUD), Fetscherstrasse 74, 01307, Dresden, Germany

5. Theodore Evans

   Technische Universität Berlin

   Affiliation as printed

   Technische Universität Berlin, DAI-Labor, Ernst-Reuter-Platz 7, 10587, Berlin, Germany

6. Klaus Strohmenger

   Humboldt-Universität zu Berlin · Charité - Universitätsmedizin Berlin

   Affiliation as printed

   Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117, Berlin, Germany

   Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117, Berlin, Germany

7. Max Westphal

   Fraunhofer Institute for Digital Medicine

   Affiliation as printed

   Fraunhofer Institute for Digital Medicine MEVIS, Max-von-Laue-Straße 2, 28359, Bremen, Germany

8. Roman David Bülow Aachen Institute of Pathology

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Institute of Pathology, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074, Aachen, Germany

9. Michaela Kargl

   Medical University of Graz

   Affiliation as printed

   Medical University of Graz, Diagnostic and Research Center for Molecular BioMedicine, Diagnostic & Research Institute of Pathology, Neue Stiftingtalstrasse 6, 8010, Graz, Austria

10. Aray Karjauv

    Technische Universität Berlin

    Affiliation as printed

    Technische Universität Berlin, DAI-Labor, Ernst-Reuter-Platz 7, 10587, Berlin, Germany

11. Isidre Munné-Bertran
    Affiliation as printed

    MoticEurope, S.L.U., C. Les Corts, 12 Poligono Industrial, 08349, Barcelona, Spain

12. Carl Orge Retzlaff

    Technische Universität Berlin

    Affiliation as printed

    Technische Universität Berlin, DAI-Labor, Ernst-Reuter-Platz 7, 10587, Berlin, Germany

13. Adrià Romero-López
    Affiliation as printed

    Lakera AI AG, Zelgstrasse 7, 8003, Zürich, Switzerland

14. Tomasz Sołtysiński
    Affiliation as printed

    QuIP GmbH, Reinhardtstraße 1, 10117, Berlin, Germany

15. Markus Plass

    Medical University of Graz

    Affiliation as printed

    Medical University of Graz, Diagnostic and Research Center for Molecular BioMedicine, Diagnostic & Research Institute of Pathology, Neue Stiftingtalstrasse 6, 8010, Graz, Austria

16. Rita Carvalho

    Humboldt-Universität zu Berlin · Charité - Universitätsmedizin Berlin

    Affiliation as printed

    Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117, Berlin, Germany

    Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117, Berlin, Germany

17. Peter Steinbach

    Helmholtz-Zentrum Dresden-Rossendorf

    Affiliation as printed

    Helmholtz-Zentrum Dresden Rossendorf, Bautzner Landstraße 400, 01328, Dresden, Germany

18. Yu-Chia Lan Aachen Institute of Pathology

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Institute of Pathology, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074, Aachen, Germany

19. Nassim Bouteldja Aachen Institute of Pathology

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Institute of Pathology, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074, Aachen, Germany

20. David Haber
    Affiliation as printed

    Lakera AI AG, Zelgstrasse 7, 8003, Zürich, Switzerland

21. Mateo Rojas-Carulla
    Affiliation as printed

    Lakera AI AG, Zelgstrasse 7, 8003, Zürich, Switzerland

22. Alireza Vafaei Sadr Aachen Institute of Pathology

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Institute of Pathology, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074, Aachen, Germany

23. Matthias Kraft
    Affiliation as printed

    Lakera AI AG, Zelgstrasse 7, 8003, Zürich, Switzerland

24. Daniel Krüger

    Olympus (Germany)

    Affiliation as printed

    Olympus Soft Imaging Solutions GmbH, Johann-Krane-Weg 39, 48149, Münster, Germany

25. Rutger Fick
    Affiliation as printed

    Tribun Health, 2 Rue du Capitaine Scott, 75015, Paris, France

26. Tobias Lang
    Affiliation as printed

    Mindpeak GmbH, Zirkusweg 2, 20359, Hamburg, Germany

27. Peter Boor Aachen Institute of Pathology

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Institute of Pathology, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074, Aachen, Germany

28. Heimo Müller

    Medical University of Graz

    Affiliation as printed

    Medical University of Graz, Diagnostic and Research Center for Molecular BioMedicine, Diagnostic & Research Institute of Pathology, Neue Stiftingtalstrasse 6, 8010, Graz, Austria

29. Peter Hufnagl

    Humboldt-Universität zu Berlin · Charité - Universitätsmedizin Berlin

    Affiliation as printed

    Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117, Berlin, Germany

    Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117, Berlin, Germany

30. Norman Zerbe

    Humboldt-Universität zu Berlin · Charité - Universitätsmedizin Berlin

    Affiliation as printed

    Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117, Berlin, Germany

    Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117, Berlin, Germany