Hidden Variables in Deep Learning Digital Pathology and Their Potential to Cause Batch Effects: Prediction Model Study (Preprint)

Abstract

BACKGROUND An increasing number of studies within digital pathology show the potential of artificial intelligence (AI) to diagnose cancer using histological whole slide images, which requires large and diverse data sets. While diversification may result in more generalizable AI-based systems, it can also introduce hidden variables. If neural networks are able to distinguish/learn hidden variables, these variables can introduce batch effects that compromise the accuracy of classification systems. OBJECTIVE The objective of the study was to analyze the learnability of an exemplary selection of hidden variables (patient age, slide preparation date, slide origin, and scanner type) that are commonly found in whole slide image data sets in digital pathology and could create batch effects. METHODS We trained four separate convolutional neural networks (CNNs) to learn four variables using a data set of digitized whole slide melanoma images from five different institutes. For robustness, each CNN training and evaluation run was repeated multiple times, and a variable was only considered learnable if the lower bound of the 95% confidence interval of its mean balanced accuracy was above 50.0%. RESULTS A mean balanced accuracy above 50.0% was achieved for all four tasks, even when considering the lower bound of the 95% confidence interval. Performance between tasks showed wide variation, ranging from 56.1% (slide preparation date) to 100% (slide origin). CONCLUSIONS Because all of the analyzed hidden variables are learnable, they have the potential to create batch effects in dermatopathology data sets, which negatively affect AI-based classification systems. Practitioners should be aware of these and similar pitfalls when developing and evaluating such systems and address these and potentially other batch effect variables in their data sets through sufficient data set stratification.

Authors 17

1. Max Schmitt

   German Cancer Research Center · National Center for Tumor Diseases

   Affiliation as printed

   Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center (DKFZ), Heidelberg, Germany

2. Roman Christoph Maron

   German Cancer Research Center · National Center for Tumor Diseases

   Affiliation as printed

   Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center (DKFZ), Heidelberg, Germany

3. Achim Hekler

   German Cancer Research Center · National Center for Tumor Diseases

   Affiliation as printed

   Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center (DKFZ), Heidelberg, Germany

4. Albrecht Stenzinger

   Heidelberg University · University Hospital Heidelberg

   Affiliation as printed

   Institute of Pathology, University Hospital Heidelberg, University of Heidelberg, Heidelberg, Germany

5. Axel Hauschild

   Christian-Albrechts-Universität zu Kiel

   Affiliation as printed

   Department of Dermatology, University Hospital Kiel, University of Kiel, Kiel, Germany

6. Michael Weichenthal

   Christian-Albrechts-Universität zu Kiel

   Affiliation as printed

   Department of Dermatology, University Hospital Kiel, University of Kiel, Kiel, Germany

7. Markus Tiemann

   Institut für Hämatopathologie Hamburg

   Affiliation as printed

   Institute for Hematopathology Hamburg, Hamburg, Germany

8. Dieter Krahl
   Affiliation as printed

   Private Institute of Dermatopathology, Heidelberg, Germany

9. Heinz Kutzner

   Dermatopathologie Friedrichshafen

   Affiliation as printed

   Private Institute of Dermatopathology, Friedrichshafen, Germany

10. Jochen Sven Utikal

    German Cancer Research Center · Heidelberg University · University Medical Centre Mannheim

    Affiliation as printed

    Department of Dermatology, University Medical Center Mannheim, University of Heidelberg, Mannheim, Germany

    Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany

11. Sebastian Haferkamp

    University Hospital Regensburg

    Affiliation as printed

    Department of Dermatology, University Hospital of Regensburg, Regensburg, Germany

12. Jakob Nikolas Kather Aachen Department of Medicine III

    Universitätsklinikum Aachen · RWTH Aachen University

    Affiliation as printed

    Department of Medicine III, RWTH University Hospital Aachen, Aachen, Germany

13. Frederick Klauschen

    Charité - Universitätsmedizin Berlin

    Affiliation as printed

    Institute of Pathology, Charité University Hospital Berlin, Berlin, Germany

14. Eva Krieghoff‐Henning

    German Cancer Research Center · National Center for Tumor Diseases

    Affiliation as printed

    Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center (DKFZ), Heidelberg, Germany

15. Stefan Fröhling

    German Cancer Research Center · National Center for Tumor Diseases

    Affiliation as printed

    National Center for Tumor Diseases, German Cancer Research Center (DKFZ), Heidelberg, Germany

16. Christof V. Kalle

    Berlin Institute of Health at Charité - Universitätsmedizin Berlin · Charité - Universitätsmedizin Berlin

    Affiliation as printed

    Department of Clinical-Translational Sciences, Charité and Berlin Institute of Health, Berlin, Germany

17. Titus Josef Brinker corresponding

    German Cancer Research Center · National Center for Tumor Diseases

    Affiliation as printed

    Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center (DKFZ), Heidelberg, Germany