Artificial Intelligence and Its Effect on Dermatologists’ Accuracy in Dermoscopic Melanoma Image Classification: Web-Based Survey Study

Journal of Medical Internet Research, vol. 22, pp. e18091

Abstract

BACKGROUND: Early detection of melanoma can be lifesaving but this remains a challenge. Recent diagnostic studies have revealed the superiority of artificial intelligence (AI) in classifying dermoscopic images of melanoma and nevi, concluding that these algorithms should assist a dermatologist's diagnoses. OBJECTIVE: The aim of this study was to investigate whether AI support improves the accuracy and overall diagnostic performance of dermatologists in the dichotomous image-based discrimination between melanoma and nevus. METHODS: Twelve board-certified dermatologists were presented disjoint sets of 100 unique dermoscopic images of melanomas and nevi (total of 1200 unique images), and they had to classify the images based on personal experience alone (part I) and with the support of a trained convolutional neural network (CNN, part II). Additionally, dermatologists were asked to rate their confidence in their final decision for each image. RESULTS: While the mean specificity of the dermatologists based on personal experience alone remained almost unchanged (70.6% vs 72.4%; P=.54) with AI support, the mean sensitivity and mean accuracy increased significantly (59.4% vs 74.6%; P=.003 and 65.0% vs 73.6%; P=.002, respectively) with AI support. Out of the 10% (10/94; 95% CI 8.4%-11.8%) of cases where dermatologists were correct and AI was incorrect, dermatologists on average changed to the incorrect answer for 39% (4/10; 95% CI 23.2%-55.6%) of cases. When dermatologists were incorrect and AI was correct (25/94, 27%; 95% CI 24.0%-30.1%), dermatologists changed their answers to the correct answer for 46% (11/25; 95% CI 33.1%-58.4%) of cases. Additionally, the dermatologists' average confidence in their decisions increased when the CNN confirmed their decision and decreased when the CNN disagreed, even when the dermatologists were correct. Reported values are based on the mean of all participants. Whenever absolute values are shown, the denominator and numerator are approximations as every dermatologist ended up rating a varying number of images due to a quality control step. CONCLUSIONS: The findings of our study show that AI support can improve the overall accuracy of the dermatologists in the dichotomous image-based discrimination between melanoma and nevus. This supports the argument for AI-based tools to aid clinicians in skin lesion classification and provides a rationale for studies of such classifiers in real-life settings, wherein clinicians can integrate additional information such as patient age and medical history into their decisions.

Authors 20

1. Roman Christoph Maron

   German Cancer Research Center · National Center for Tumor Diseases

   Affiliation as printed

   Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany

2. Jochen Sven Utikal

   German Cancer Research Center · Heidelberg University · University Medical Centre Mannheim

   Affiliation as printed

   Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, University of Heidelberg, Heidelberg, Germany

   Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany

3. Achim Hekler

   German Cancer Research Center · National Center for Tumor Diseases

   Affiliation as printed

   Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany

4. Axel Hauschild

   Christian-Albrechts-Universität zu Kiel

   Affiliation as printed

   Department of Dermatology, University Hospital Kiel, University of Kiel, Kiel, Germany

5. Elke Sattler

   LMU Klinikum · Ludwig-Maximilians-Universität München

   Affiliation as printed

   Department of Dermatology, University Hospital Munich (LMU), Munich, Germany

6. Wiebke Sondermann

   Essen University Hospital · University of Duisburg-Essen

   Affiliation as printed

   Department of Dermatology, University Hospital Essen, University of Duisburg-Essen, Essen, Germany

7. Sebastian Haferkamp

   University Hospital Regensburg

   Affiliation as printed

   Department of Dermatology, University Hospital Regensburg, Regensburg, Germany

8. Bastian Schilling

   University of Würzburg · Universitätsklinikum Würzburg

   Affiliation as printed

   Department of Dermatology, University Hospital Würzburg, University of Würzburg, Würzburg, Germany

9. Markus Vincent Heppt

   Universitätsklinikum Erlangen

   Affiliation as printed

   Department of Dermatology, University Hospital Erlangen, University of Erlangen, Erlangen, Germany

10. Philipp Jansen

    Essen University Hospital · University of Duisburg-Essen

    Affiliation as printed

    Department of Dermatology, University Hospital Essen, University of Duisburg-Essen, Essen, Germany

11. Markus Reinholz

    LMU Klinikum · Ludwig-Maximilians-Universität München

    Affiliation as printed

    Department of Dermatology, University Hospital Munich (LMU), Munich, Germany

12. Cindy Franklin

    University Hospital Cologne

    Affiliation as printed

    Department of Dermatology, University Hospital Cologne, Cologne, Germany

13. Laurenz Schmitt Aachen Department of Dermatology

    Universitätsklinikum Aachen

    Affiliation as printed

    Department of Dermatology, University Hospital Aachen, Aachen, Germany

14. Daniela Hartmann

    LMU Klinikum · Ludwig-Maximilians-Universität München

    Affiliation as printed

    Department of Dermatology, University Hospital Munich (LMU), Munich, Germany

15. Eva Krieghoff‐Henning

    German Cancer Research Center · National Center for Tumor Diseases

    Affiliation as printed

    Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany

16. Max Schmitt

    German Cancer Research Center · National Center for Tumor Diseases

    Affiliation as printed

    Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany

17. Michael Weichenthal

    Christian-Albrechts-Universität zu Kiel

    Affiliation as printed

    Department of Dermatology, University Hospital Kiel, University of Kiel, Kiel, Germany

18. Christof V. Kalle

    Berlin Institute of Health at Charité - Universitätsmedizin Berlin · Charité - Universitätsmedizin Berlin

    Affiliation as printed

    Department of Clinical-Translational Sciences, Charité University Medicine and Berlin Institute of Health (BIH), Berlin, Germany

19. Stefan Fröhling

    German Cancer Research Center · National Center for Tumor Diseases

    Affiliation as printed

    National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany

20. Titus Josef Brinker corresponding

    German Cancer Research Center · National Center for Tumor Diseases

    Affiliation as printed

    Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany