Reinforcement learning for real-world production systems

RWTH Publications (RWTH Aachen)

Abstract

This dissertation envisions a future of robotic production capable of solving complex tasks in changing and uncertain environments. Such capabilities are critical as production trends shift toward smaller lot sizes, individualised products, and human-robot collaboration. Reinforcement learning (RL) methods hold promises for solving such challenges as they learn behaviours through interaction with their surrounding environments and ideally generalise to new unseen scenarios. However, realising this promise in real-world settings has been challenging. Key barriers include data collection efficiency and safety, the complexity and unpredictability of real-world dynamics, and usability challenges for domain experts with limited machine learning experience. The central challenge addressed in this dissertation lies in bridging the gap between existing RL methods and the challenges posed by real-world production. It proposes novel methods to improve data collection regarding safety and efficiency, consider changing and uncertain environments, and improve user interpretability and acceptance. As a first contribution, we introduce a novel metric for evaluating the performance of spatial autoencoders, a neural network extracting spatial features from images. By quantifying how well such autoencoders track objects in a 3D space, we address key challenges in using image-based state representations in RL, particularly in unstructured environments. The proposed metric is applied to assess various spatial autoencoder architectures and their ability to track object positions. We show that this ability correlates with downstream RL performance. As a result, the developed evaluation metric offers a lightweight, computationally efficient method for predicting RL performance, providing a practical tool for selecting trained models in robotic RL tasks. Next, we develop the Contextualized Hybrid Ensemble Q-learning (CHEQ) algorithm, a novel adaptive hybrid RL method. The algorithm blends the action from an RL agent with a classical model-based controller and dynamically adjusts the weighting between both based on the RL agent's current capabilities. CHEQ is evaluated on a simulated car racing task that requires precise control near stability limits. Compared to state-of-the-art adaptive hybrid RL methods, CHEQ demonstrates (i) enhanced data efficiency, (ii) safer exploration, and (iii) improved zero-shot transferability to unseen scenarios. Lastly, this thesis explores the application of RL to robotic polishing, a contact-rich and dynamic task that exemplifies the complexities of manipulation and other contact-rich problems in industrial automation. In this context, the CHEQ algorithm is applied to learn variable impedance gains and path-following for polishing a 3D workpiece. In simulation, we demonstrate the potential of CHEQ to learn suitable gains while maintaining safe exploration behaviour. Building on this, we successfully deploy CHEQ on hardware. Direct hardware training introduces environment uncertainties and nonstationarities, as well as practical requirements for sample efficiency and safety. CHEQ effectively addresses these challenges, confirming its suitability for real-world industrial automation tasks.

Authors 0

1. Author list not loaded yet.