Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insights

arXiv (Cornell University)

Abstract

Intrigued by the inherent ability of the human visual system to identify salient regions in complex scenes, attention mechanisms have been seamlessly integrated into various Computer Vision (CV) tasks. Building upon this paradigm, Vision Transformer (ViT) networks exploit attention mechanisms for improved efficiency. This review navigates the landscape of redesigned attention mechanisms within ViTs, aiming to enhance their performance. This paper provides a comprehensive exploration of techniques and insights for designing attention mechanisms, systematically reviewing recent literature in the field of CV. This survey begins with an introduction to the theoretical foundations and fundamental concepts underlying attention mechanisms. We then present a systematic taxonomy of various attention mechanisms within ViTs, employing redesigned approaches. A multi-perspective categorization is proposed based on their application, objectives, and the type of attention applied. The analysis includes an exploration of the novelty, strengths, weaknesses, and an in-depth evaluation of the different proposed strategies. This culminates in the development of taxonomies that highlight key properties and contributions. Finally, we gather the reviewed studies along with their available open-source implementations at our \href{https://github.com/mindflow-institue/Awesome-Attention-Mechanism-in-Medical-Imaging}{GitHub}\footnote{\url{https://github.com/xmindflow/Awesome-Attention-Mechanism-in-Medical-Imaging}}. We aim to regularly update it with the most recent relevant papers.

Authors 11

1. Moein Heidari

   University of British Columbia

   Affiliation as printed

   School of Biomedical Engineering , University of British Columbia , British Columbia , Canada

2. Reza Azad Aachen Faculty of Electrical Engineering and Information Technology

   RWTH Aachen University

   Affiliation as printed

   Faculty of Electrical Engineering and Information Technology , RWTH Aachen University , Aachen , Germany

3. Sina Ghorbani Kolahi

   Tarbiat Modares University

   Affiliation as printed

   Department of Industrial and Systems Engineering , Tarbiat Modares University , Tehran , Iran

4. René Arimond Aachen Faculty of Electrical Engineering and Information Technology

   RWTH Aachen University

   Affiliation as printed

   Faculty of Electrical Engineering and Information Technology , RWTH Aachen University , Aachen , Germany

5. Leon Niggemeier Aachen Faculty of Electrical Engineering and Information Technology

   RWTH Aachen University

   Affiliation as printed

   Faculty of Electrical Engineering and Information Technology , RWTH Aachen University , Aachen , Germany

6. Alaa Sulaiman

   National University of Malaysia

   Affiliation as printed

   Faculty of Information Science and Technology , Universiti Ke- bangsaan , Bangi , Malaysia

7. Afshin Bozorgpour

   University of Regensburg

   Affiliation as printed

   Faculty of Informatics and Data Science , University of Regens- burg , Regensburg , Germany

8. Ehsan Khodapanah Aghdam

   Shahid Beheshti University

   Affiliation as printed

   Department of Electrical Engineering , Shahid Beheshti Univer- sity , Tehran , Iran

9. Amirhossein Kazerouni

   University of Toronto

   Affiliation as printed

   Department of Computer Science , University of Toronto , Toronto , Canada

10. Ilker Hacihaliloglu

    University of British Columbia

    Affiliation as printed

    Department of Radiology , Department of Medicine , Univer- sity of British Columbia , British Columbia , Canada

11. Dorit Merhof

    Fraunhofer Institute for Digital Medicine · University of Regensburg

    Affiliation as printed

    Faculty of Informatics and Data Science , University of Regens- burg , Regensburg , Germany

    Fraunhofer Institute for Digital Medicine MEVIS , Bremen , Ger- many