CoNeS: Conditional neural fields with shift modulation for multi-sequence MRI translation

The Journal of Machine Learning for Biomedical Imaging, vol. 2024, pp. 657–685

Abstract

Multi-sequence magnetic resonance imaging (MRI) has found wide applications in both modern clinical studies and deep learning research. However, in clinical practice, it frequently occurs that one or more of the MRI sequences are missing due to different image acquisition protocols or contrast agent contraindications of patients, limiting the utilization of deep learning models trained on multi-sequence data. One promising approach is to leverage generative models to synthesize the missing sequences, which can serve as a surrogate acquisition. State-of-the-art methods tackling this problem are based on convolutional neural networks (CNN) which usually suffer from spectral biases, resulting in poor reconstruction of high-frequency fine details. In this paper, we propose Conditional Neural fields with Shift modulation (CoNeS), a model that takes voxel coordinates as input and learns a representation of the target images for multi-sequence MRI translation. The proposed model uses a multi-layer perceptron (MLP) instead of a CNN as the decoder for pixel-to-pixel mapping. Hence, each target image is represented as a neural field that is conditioned on the source image via shift modulation with a learned latent code. Experiments on BraTS 2018 and an in-house clinical dataset of vestibular schwannoma patients showed that the proposed method outperformed state-of-the-art methods for multi-sequence MRI translation both visually and quantitatively. Moreover, we conducted spectral analysis, showing that CoNeS was able to overcome the spectral bias issue common in conventional CNN models. To further evaluate the usage of synthesized images in clinical downstream tasks, we tested a segmentation network using the synthesized images at inference. The results showed that CoNeS improved the segmentation performance when some MRI sequences were missing and outperformed other synthesis models. We concluded that neural fields are a promising technique for multi-sequence MRI translation.

Authors 8

1. Yunjie Chen Aachen Department of Radiology, Leiden University Medical Center Department of Otorhinolaryngology and Head & Neck Surgery

   Leiden University Medical Center · University of Twente · Delft University of Technology

   Affiliation as printed

   Department of Radiology, Leiden University Medical Center

   Department of Applied Mathematics, Technical Medical Center, University of Twente, Enschede, the Netherlands

   Department of Imaging Physics, Delft University of Technology, Delft, the Netherlands

   Department of Otorhinolaryngology and Head & Neck Surgery, Leiden University Medical Center, Leiden, the Netherlands

   Department of Radiology, Leiden University Medical Center, Leiden, the Netherlands

2. Marius Staring Aachen Department of Radiology, Leiden University Medical Center Department of Otorhinolaryngology and Head & Neck Surgery

   Leiden University Medical Center · University of Twente · Delft University of Technology

   Affiliation as printed

   Department of Radiology, Leiden University Medical Center

   Department of Applied Mathematics, Technical Medical Center, University of Twente, Enschede, the Netherlands

   Department of Imaging Physics, Delft University of Technology, Delft, the Netherlands

   Department of Otorhinolaryngology and Head & Neck Surgery, Leiden University Medical Center, Leiden, the Netherlands

   Department of Radiology, Leiden University Medical Center, Leiden, the Netherlands

3. Olaf M. Neve Aachen Department of Otorhinolaryngology and Head & Neck Surgery Department of Radiology, Leiden University Medical Center

   Leiden University Medical Center · University of Twente · Delft University of Technology

   Affiliation as printed

   Department of Otorhinolaryngology and Head & Neck Surgery, Leiden University Medical Center

   Department of Applied Mathematics, Technical Medical Center, University of Twente, Enschede, the Netherlands

   Department of Imaging Physics, Delft University of Technology, Delft, the Netherlands

   Department of Otorhinolaryngology and Head & Neck Surgery, Leiden University Medical Center, Leiden, the Netherlands

   Department of Radiology, Leiden University Medical Center, Leiden, the Netherlands

4. Stephan R. Romeijn Aachen Department of Radiology, Leiden University Medical Center Department of Otorhinolaryngology and Head & Neck Surgery

   Leiden University Medical Center · University of Twente · Delft University of Technology

   Affiliation as printed

   Department of Radiology, Leiden University Medical Center

   Department of Applied Mathematics, Technical Medical Center, University of Twente, Enschede, the Netherlands

   Department of Imaging Physics, Delft University of Technology, Delft, the Netherlands

   Department of Otorhinolaryngology and Head & Neck Surgery, Leiden University Medical Center, Leiden, the Netherlands

   Department of Radiology, Leiden University Medical Center, Leiden, the Netherlands

5. Erik Frans Hensen Aachen Department of Otorhinolaryngology and Head & Neck Surgery Department of Radiology, Leiden University Medical Center

   Leiden University Medical Center · University of Twente · Delft University of Technology

   Affiliation as printed

   Department of Otorhinolaryngology and Head & Neck Surgery, Leiden University Medical Center

   Department of Applied Mathematics, Technical Medical Center, University of Twente, Enschede, the Netherlands

   Department of Imaging Physics, Delft University of Technology, Delft, the Netherlands

   Department of Otorhinolaryngology and Head & Neck Surgery, Leiden University Medical Center, Leiden, the Netherlands

   Department of Radiology, Leiden University Medical Center, Leiden, the Netherlands

6. Berit M. Verbist Aachen Department of Radiology, Leiden University Medical Center Department of Otorhinolaryngology and Head & Neck Surgery

   Leiden University Medical Center · University of Twente · Delft University of Technology

   Affiliation as printed

   Department of Radiology, Leiden University Medical Center

   Department of Applied Mathematics, Technical Medical Center, University of Twente, Enschede, the Netherlands

   Department of Imaging Physics, Delft University of Technology, Delft, the Netherlands

   Department of Otorhinolaryngology and Head & Neck Surgery, Leiden University Medical Center, Leiden, the Netherlands

   Department of Radiology, Leiden University Medical Center, Leiden, the Netherlands

7. Jelmer M. Wolterink Aachen Department of Otorhinolaryngology and Head & Neck Surgery Department of Radiology, Leiden University Medical Center

   Leiden University Medical Center · University of Twente · Delft University of Technology

   Affiliation as printed

   Department of Applied Mathematics, Technical Medical Center, University of Twente

   Department of Applied Mathematics, Technical Medical Center, University of Twente, Enschede, the Netherlands

   Department of Imaging Physics, Delft University of Technology, Delft, the Netherlands

   Department of Otorhinolaryngology and Head & Neck Surgery, Leiden University Medical Center, Leiden, the Netherlands

   Department of Radiology, Leiden University Medical Center, Leiden, the Netherlands

8. Qian Tao

   Delft University of Technology

   Affiliation as printed

   Department of Imaging Physics, Delft University of Technology