Radiopaths: Deep Multimodal Analysis on Chest Radiographs
IEEE International Conference on Big Data (BigData), vol. 30, pp. 3613–3621
Abstract
Each year, pneumonia causes harm to millions of people worldwide. The chest radiograph remains the most important tool for diagnosing pneumonia and many other thoracic diseases. Many deep-learning approaches have been advocated in recent years, taking advantage of large public medical datasets and powerful computational resources to support radiologists in their examinations. However, the vast amount of data available is rarely utilized. In this work, we propose two novel attention-based mechanisms for fusing multimodal and multi-view data in a medical context, respectively. A unimodal multi-view classifier for chest X-rays and a multimodal classifier are developed, inter alia, operating these mechanisms to classify pneumonia using chest radiographs in combination with other clinically derived modalities. Our unimodal multi-view model outperforms other state-of-the-art approaches on the MIMIC-IV dataset in the AUROC metric by at least 9.2%, achieving an AUROC score of 81.5%. Our attention-based multimodal model, namely Radiopaths, achieved an AUROC score of 87.3%, further increasing the multi-view model's performance by 5.8%, while acquiring robustness to missing modalities and scalability.
Authors 5
-
Affiliation as printed
RWTH Aachen University,Chair of Information Theory and Data Analytics,Aachen,Germany,D-52074
-
Affiliation as printed
RWTH Aachen University,Chair of Information Theory and Data Analytics,Aachen,Germany,D-52074
-
Affiliation as printed
RWTH Aachen University,Chair of Information Theory and Data Analytics,Aachen,Germany,D-52074
-
Affiliation as printed
RWTH Aachen University,Chair of Computer Vision,Aachen,Germany,D-52074
-
Affiliation as printed
RWTH Aachen University,Chair of Information Theory and Data Analytics,Aachen,Germany,D-52074
Cited by 3 stored of 3
3 results
No patents citing this paper on Lens.org (checked 2026-10-06).