Sample-efficient reinforcement learning of Koopman eNMPC

Computers & Chemical Engineering, vol. 201, pp. 109240

Abstract

Reinforcement learning (RL) can be used to tune data-driven (economic) nonlinear model predictive controllers ((e)NMPCs) for optimal performance in a specific control task by optimizing the dynamic model or parameters in the policy’s objective function or constraints, such as state bounds. However, the sample efficiency of RL is crucial, and to improve it, we combine a model-based RL algorithm with our published method that turns Koopman (e)NMPCs into automatically differentiable policies. We apply our approach to an eNMPC case study of a continuous stirred-tank reactor (CSTR) model from the literature. The approach outperforms benchmark methods, i.e., data-driven eNMPCs using models based on system identification without further RL tuning of the resulting policy, and neural network controllers trained with model-based RL, by achieving superior control performance and higher sample efficiency. Furthermore, utilizing partial prior knowledge about the system dynamics via physics-informed learning further increases sample efficiency.

Authors 4

1. Daniel Mayfrank Aachen

   Forschungszentrum Jülich · RWTH Aachen University

   Affiliation as printed

   Forschungszentrum Jülich GmbH, Institute of Climate and Energy Systems, Energy Systems Engineering (ICE-1), Jülich 52425, Germany

   RWTH Aachen University, Aachen 52062, Germany

2. Mehmet Velioglu Aachen

   Forschungszentrum Jülich · RWTH Aachen University

   Affiliation as printed

   Forschungszentrum Jülich GmbH, Institute of Climate and Energy Systems, Energy Systems Engineering (ICE-1), Jülich 52425, Germany

   RWTH Aachen University, Aachen 52062, Germany

3. Alexander Mitsos Aachen

   Forschungszentrum Jülich · Jülich Aachen Research Alliance · RWTH Aachen University

   Affiliation as printed

   Forschungszentrum Jülich GmbH, Institute of Climate and Energy Systems, Energy Systems Engineering (ICE-1), Jülich 52425, Germany

   JARA-ENERGY, Jülich 52425, Germany

   RWTH Aachen University, Process Systems Engineering (AVT.SVT), Aachen 52074, Germany

4. Manuel Dahmen corresponding

   Forschungszentrum Jülich

   Affiliation as printed

   Forschungszentrum Jülich GmbH, Institute of Climate and Energy Systems, Energy Systems Engineering (ICE-1), Jülich 52425, Germany