Abnormality-Driven Representation Learning for Radiology Imaging

Lecture notes in computer science, pp. 14–24

Abstract

Radiology deep learning pipelines predominantly employ end-to-end 3D networks based on models pre-trained on other tasks, which are then fine-tuned on the task at hand. In contrast, adjacent medical fields such as pathology, which focus on 2D images, have effectively adopted task-agnostic foundational models based on self-supervised learning (SSL), combined with weakly-supervised deep learning (DL). However, the field of radiology still lacks task-agnostic representation models due to the computational and data demands of 3D imaging and the anatomical complexity inherent to radiology scans. To address this gap, we propose Clear, a framework for 3D radiology images that uses extracted embeddings from 2D slices along with attention-based aggregation to efficiently predict clinical endpoints. As part of this framework, we introduce Lecl, a novel approach to obtain visual representations driven by abnormalities in 2D axial slices across different locations of the CT scans. Specifically, we trained single-domain contrastive learning approaches using three different architectures: Vision Transformers, Vision State Space Models and Gated Convolutional Neural Networks. We evaluate our approach across three clinical tasks: tumor lesion location, lung disease detection, and patient staging, benchmarking against four state-of-the-art foundation models, including BiomedCLIP. Our findings demonstrate that Clear, using representations learned through Lecl, outperforms existing foundation models, while being substantially more compute- and data-efficient. The code is available at https://github.com/KatherLab/CLEAR .

Authors 6

1. Marta Ligero

   Else Kröner Fresenius Center for Digital Health · Technische Universität Dresden

   Affiliation as printed

   EKFZ for Digital Health TU Dresden, Dresden, Germany

2. Tim Lenz

   Else Kröner Fresenius Center for Digital Health · Technische Universität Dresden

   Affiliation as printed

   EKFZ for Digital Health TU Dresden, Dresden, Germany

3. Georg Wölflein

   University of St Andrews · Else Kröner Fresenius Center for Digital Health · Technische Universität Dresden

   Affiliation as printed

   EKFZ for Digital Health TU Dresden, Dresden, Germany

   University of St Andrews, St Andrews, UK

4. Omar S. M. El Nahhas

   Else Kröner Fresenius Center for Digital Health · Technische Universität Dresden

   Affiliation as printed

   EKFZ for Digital Health TU Dresden, Dresden, Germany

   StratifAI GmbH, Dresden, Germany

5. Daniel Truhn Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University, Aachen, Germany

6. Jakob Nikolas Kather corresponding Aachen

   Heidelberg University · University Hospital Heidelberg · University Hospital Carl Gustav Carus · Else Kröner Fresenius Center for Digital Health · Technische Universität Dresden · RWTH Aachen University

   Affiliation as printed

   EKFZ for Digital Health TU Dresden, Dresden, Germany

   Heidelberg University Hospital, Heidelberg, Germany

   RWTH Aachen University, Aachen, Germany

   StratifAI GmbH, Dresden, Germany

   University Hospital Dresden, Dresden, Germany