HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation

IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 6191–6201

Abstract

Convolutional neural networks (CNNs) have been the consensus for medical image segmentation tasks. However, they suffer from the limitation in modeling long-range dependencies and spatial correlations due to the nature of convolution operation. Although transformers were first developed to address this issue, they fail to capture low-level features. In contrast, it is demonstrated that both local and global features are crucial for dense prediction, such as segmenting in challenging contexts. In this paper, we propose HiFormer, a novel method that efficiently bridges a CNN and a transformer for medical image segmentation. Specifically, we design two multi-scale feature representations using the seminal Swin Transformer module and a CNN-based encoder. To secure a fine fusion of global and local features obtained from the two aforementioned representations, we propose a Double-Level Fusion (DLF) module in the skip connection of the encoder-decoder structure. Extensive experiments on various medical image segmentation datasets demonstrate the effectiveness of HiFormer over other CNN-based, transformer-based, and hybrid methods in terms of computational complexity, quantitative and qualitative results. Our code is publicly available at GitHub.

Authors 7

1. Moein Heidari

   Iran University of Science and Technology

   Affiliation as printed

   Iran University of Science and Technology,School of Electrical Engineering,Tehran,Iran

   School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran

2. Amirhossein Kazerouni

   Iran University of Science and Technology

   Affiliation as printed

   Iran University of Science and Technology,School of Electrical Engineering,Tehran,Iran

   School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran

3. Milad Soltany

   Iran University of Science and Technology

   Affiliation as printed

   Iran University of Science and Technology,School of Electrical Engineering,Tehran,Iran

   School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran

4. Reza Azad Aachen Institute of Imaging & Computer Vision

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Institute of Imaging and Computer Vision,Aachen,Germany

   Institute of Imaging and Computer Vision, RWTH Aachen University, Aachen, Germany

5. Ehsan Khodapanah Aghdam

   Shahid Beheshti University

   Affiliation as printed

   Shahid Beheshti University,Department of Electrical Engineering,Tehran,Iran

   Department of Electrical Engineering, Shahid Beheshti University, Tehran, Iran

6. Julien Cohen‐Adad

   Mila - Quebec Artificial Intelligence Institute

   Affiliation as printed

   Quebec AI Institute,MILA,Montreal,Canada

   MILA, Quebec AI Institute, Montreal, Canada

7. Dorit Merhof

   Fraunhofer Institute for Digital Medicine · University of Regensburg

   Affiliation as printed

   University of Regensburg,Faculty of Informatics and Data Science,Regensburg,Germany

   Faculty of Informatics and Data Science, University of Regensburg, Regensburg, Germany

   Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany