Training-free thick cloud removal for Sentinel-2 imagery using value propagation interpolation

ISPRS Journal of Photogrammetry and Remote Sensing, vol. 216, pp. 168–184

Abstract

Remote sensing imagery has an ever-increasing impact on important downstream applications, such as vegetation monitoring and climate change modelling. Clouds obscuring parts of the images create a substantial bottleneck in most machine learning tasks that use remote sensing data, and being robust to this issue is an important technical challenge. In many cases, cloudy images cannot be used in a machine learning pipeline, leading to either the removal of the images altogether, or to using suboptimal solutions reliant on recent cloud-free imagery or the availability of pre-trained models for the exact use case. In this work, we propose VPint2, a cloud removal method built upon the VPint algorithm, an easy-to-apply data-driven spatial interpolation method requiring no prior training, to address the problem of cloud removal. This method leverages previously sensed cloud-free images to represent the spatial structure of a region, which is then used to propagate up-to-date information from non-cloudy pixels to cloudy ones. We also created a benchmark dataset called SEN2-MSI-T, composed of 20 scenes with 5 full-sized images each, belonging to five common land cover classes. We used this dataset to evaluate our method against three alternatives: mosaicking, an AutoML-based regression method, and the nearest similar pixel interpolator. Additionally, we compared against two previously published neural network-based methods on SEN2-MSI-T, and evaluate our method on a subset of the popular SEN12MS-CR-TS benchmark dataset. The methods are compared using several performance metrics, including the structural similarity index, mean absolute error, and error rates on a downstream NDVI derivation task. Our experimental results show that VPint2 performed significantly better than competing methods over 20 experimental conditions, improving performance by 2.4% to 34.3% depending on the condition. We also found that the performance of VPint2 only decreases marginally as the temporal distance of its reference image increases, and that, unlike typical interpolation methods, the performance of VPint2 remains strong for larger percentages of cloud cover. Our findings furthermore support a cloud removal evaluation approach founded on the transfer of cloud masks over the use of cloud-free previous acquisitions as ground truth.

Authors 7

1. Laurens Arp corresponding Aachen Leiden Institute of Advanced Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science, Leiden University, The Netherlands

2. Holger H. Hoos Aachen Chair for Artificial Intelligence Methodology Leiden Institute of Advanced Computer Science

   Leiden University · RWTH Aachen University

   Affiliation as printed

   Chair for Artificial Intelligence Methodology, RTWH Aachen University, Germany

   Leiden Institute of Advanced Computer Science, Leiden University, The Netherlands

3. Peter M. van Bodegom Aachen Institute of Environmental Sciences (CML)

   Leiden University

   Affiliation as printed

   Institute of Environmental Sciences, Leiden University, The Netherlands

4. Alistair Francis

   European Space Agency · European Space Research Institute

   Affiliation as printed

   Φ-lab, European Space Research Institute, European Space Agency, Italy

5. James Wheeler

   European Space Agency · European Space Research Institute

   Affiliation as printed

   Φ-lab, European Space Research Institute, European Space Agency, Italy

6. Dean van Laar Aachen Leiden Institute of Advanced Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science, Leiden University, The Netherlands

7. Mitra Baratchi Aachen Leiden Institute of Advanced Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science, Leiden University, The Netherlands