Hierarchical Prediction of Registration Misalignment Using a Convolutional LSTM: Application to Chest CT Scans

IEEE Access, vol. 9, pp. 62008–62020

Abstract

In this paper we propose a supervised method to predict registration misalignment using convolutional neural networks (CNNs). This task is casted to a classification problem with multiple classes of misalignment: “correct” 0-3 mm, “poor” 3-6 mm and “wrong” over 6 mm. Rather than a direct prediction, we propose a hierarchical approach, where the prediction is gradually refined from coarse to fine. Our solution is based on a convolutional Long Short-Term Memory (LSTM), using hierarchical misalignment predictions on three resolutions of the image pair, leveraging the intrinsic strengths of an LSTM for this problem. The convolutional LSTM is trained on a set of artificially generated image pairs obtained from artificial displacement vector fields (DVFs). Results on chest CT scans show that incorporating multi-resolution information, and the hierarchical use via an LSTM for this, leads to overall better F1 scores, with fewer misclassifications in a well-tuned registration setup. The final system yields an accuracy of 87.1%, and an average F1 score of 66.4% aggregated in two independent chest CT scan studies.

Authors 5

1. Hessam Sokooti Aachen Division of Image Processing (LKEB)

   Leiden University Medical Center

   Affiliation as printed

   Division of Image Processing, Leiden University Medical Center, ZA Leiden, The Netherlands

2. Sahar Yousefi Aachen Division of Image Processing (LKEB)

   Leiden University Medical Center

   Affiliation as printed

   Division of Image Processing, Leiden University Medical Center, ZA Leiden, The Netherlands

3. Mohamed S. Elmahdy Aachen Division of Image Processing (LKEB)

   Leiden University Medical Center

   Affiliation as printed

   Division of Image Processing, Leiden University Medical Center, ZA Leiden, The Netherlands

4. Boudewijn P. F. Lelieveldt Aachen Division of Image Processing (LKEB)

   Leiden University Medical Center · Delft University of Technology

   Affiliation as printed

   Delft University of Technology, CD Delft, The Netherlands

   Division of Image Processing, Leiden University Medical Center, ZA Leiden, The Netherlands

5. Marius Staring Aachen Division of Image Processing (LKEB)

   Leiden University Medical Center · Delft University of Technology

   Affiliation as printed

   Delft University of Technology, CD Delft, The Netherlands

   Division of Image Processing, Leiden University Medical Center, ZA Leiden, The Netherlands