What Does DALL-E 2 Know About Radiology?

Journal of Medical Internet Research, vol. 25, pp. e43110

Abstract

Generative models, such as DALL-E 2 (OpenAI), could represent promising future tools for image generation, augmentation, and manipulation for artificial intelligence research in radiology, provided that these models have sufficient medical domain knowledge. Herein, we show that DALL-E 2 has learned relevant representations of x-ray images, with promising capabilities in terms of zero-shot text-to-image generation of new images, the continuation of an image beyond its original boundaries, and the removal of elements; however, its capabilities for the generation of images with pathological abnormalities (eg, tumors, fractures, and inflammation) or computed tomography, magnetic resonance imaging, or ultrasound images are still limited. The use of generative models for augmenting and generating radiological data thus seems feasible, even if the further fine-tuning and adaptation of these models to their respective domains are required first.

Authors 6

1. Lisa C. Adams

   Humboldt-Universität zu Berlin · Freie Universität Berlin · Charité - Universitätsmedizin Berlin · Stanford University

   Affiliation as printed

   Department of Radiology, Charité Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany

   Department of Radiology, Stanford University, Stanford, CA, United States

2. Felix Busch

   Humboldt-Universität zu Berlin · Freie Universität Berlin · Charité - Universitätsmedizin Berlin

   Affiliation as printed

   Department of Radiology, Charité Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany

3. Daniel Truhn Aachen Department of Diagnostic and Interventional Radiology

   Universitätsklinikum Aachen

   Affiliation as printed

   Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany

4. Marcus R. Makowski

   Technical University of Munich

   Affiliation as printed

   Department of Diagnostic and Interventional Radiology, Technical University of Munich, Munich, Germany

5. Hugo J.W.L. Aerts

   Harvard University · Maastricht University · Mass General Brigham

   Affiliation as printed

   Artificial Intelligence in Medicine Program (AIM), Mass General Brigham, Harvard Medical School, Boston, MA, United States

   Radiology and Nuclear Medicine, CARIM & GROW, Maastricht University, Maastricht, Netherlands

6. Keno K. Bressem corresponding

   Harvard University · Maastricht University · Humboldt-Universität zu Berlin · Mass General Brigham · Freie Universität Berlin · Charité - Universitätsmedizin Berlin

   Affiliation as printed

   Artificial Intelligence in Medicine Program (AIM), Mass General Brigham, Harvard Medical School, Boston, MA, United States

   Department of Radiology, Charité Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany

   Radiology and Nuclear Medicine, CARIM & GROW, Maastricht University, Maastricht, Netherlands