Machine learning-based conditional mean filter: A generalization of the ensemble Kalman filter for nonlinear data assimilation

Foundations of Data Science, vol. 5, pp. 56–80

Abstract

This paper presents the machine learning-based ensemble conditional mean filter (ML-EnCMF) — a filtering method based on the conditional mean filter (CMF) previously introduced in the literature. The updated mean of the CMF matches that of the posterior, obtained by applying Bayes' rule on the filter's forecast distribution. Moreover, we show that the CMF's updated covariance coincides with the expected conditional covariance. Implementing the EnCMF requires computing the conditional mean (CM). A likelihood-based estimator is prone to significant errors for small ensemble sizes, causing the filter divergence. We develop a systematical methodology for integrating machine learning into the EnCMF based on the CM's orthogonal projection property. First, we use a combination of an artificial neural network (ANN) and a linear function, obtained based on the ensemble Kalman filter (EnKF), to approximate the CM, enabling the ML-EnCMF to inherit EnKF's advantages. Secondly, we apply a suitable variance reduction technique to reduce statistical errors when estimating loss function. Lastly, we propose a model selection procedure for element-wisely selecting the applied filter, i.e., either the EnKF or ML-EnCMF, at each updating step. We demonstrate the ML-EnCMF performance using the Lorenz-63 and Lorenz-96 systems and show that the ML-EnCMF outperforms the EnKF and the likelihood-based EnCMF.

Authors 4

1. Truong‐Vinh Hoang corresponding Aachen Chair of Mathematics for Uncertainty Quantification

   RWTH Aachen University

   Affiliation as printed

   Chair of Mathematics for Uncertainty Quantification, RWTH Aachen University, Germany,

2. Sebastian Krumscheid Aachen Chair of Mathematics for Uncertainty Quantification

   RWTH Aachen University

   Affiliation as printed

   Chair of Mathematics for Uncertainty Quantification, RWTH Aachen University, Germany,

3. Hermann Georg Matthies

   Technische Universität Braunschweig

   Affiliation as printed

   Technische Universität Braunschweig, Germany

4. Raul F. Tempone Aachen

   King Abdullah University of Science and Technology · RWTH Aachen University

   Affiliation as printed

   Alexander von Humboldt professor in Mathematics of Uncertainty Quantification, RWTH Aachen University

   Computer, Electrical and Mathematical Sciences and Engineering, KAUST, and