Model-based reinforcement learning for hydraulic systems: a PILCO-based control strategy for controlling an inverted hydraulic pendulum

International Journal of Dynamics and Control, vol. 13

Abstract

Abstract In fluid power systems, hydraulic drives are widely used but often rely on linear controllers that require precise tuning to specific operational contexts. Traditional control methods, while effective, can be inflexible and struggle with the nonlinear aspects of hydraulic systems, such as fluid compressibility and friction. Experimental investigations to understand these systems are often impractical due to cost and safety concerns, making simulations a valuable alternative. However, accurate simulations are computationally intensive and not always feasible for real-time control. This work introduces a Probabilistic Inference for Learning and Control (PILCO) algorithm to address these challenges. PILCO is a model-based reinforcement learning (RL) algorithm that constructs a dynamics model from observed data, incorporating uncertainties to predict future system behavior with minimal environmental interactions. The algorithm was applied to control a physical inverted hydraulic pendulum, a complex task due to the system’s nonlinearities. PILCO successfully learned precise control policies with only 64 seconds of interaction time, demonstrating the effectiveness of reward shaping and offline data initialization in accelerating learning. The results highlight PILCO’s potential to advance optimal control in real-world hydraulic systems through efficient and adaptable RL strategies.

Authors 4

1. Faras Brumand‐Poor corresponding Aachen Institute for Fluid Power Drives and Systems

   RWTH Aachen University

   Affiliation as printed

   Institute for Fluid Power Drives and Systems (ifas), RWTH Aachen University, Aachen, Germany

2. Paul Wagemann

   Bucher Industries (Switzerland)

   Affiliation as printed

   Bucher Hydraulics AG, Bucher Industries AG, Neuheim, Switzerland

3. Timm Geibel

   Bucher Industries (Switzerland)

   Affiliation as printed

   Bucher Hydraulics AG, Bucher Industries AG, Neuheim, Switzerland

4. Katharina Schmitz Aachen Institute for Fluid Power Drives and Systems

   RWTH Aachen University

   Affiliation as printed

   Institute for Fluid Power Drives and Systems (ifas), RWTH Aachen University, Aachen, Germany