On the use of evolutionary-algorithm-driven reinforcement learning in underwater robotics

RWTH Publications (RWTH Aachen)

Abstract

Motivation, Goal and Task of the Dissertation: The underwater environment is inherently dynamic, uncertain, and difficult to capture through precise mathematical models. These characteristics pose substantial challenges to the design and development of underwater robots and autonomous vehicles, both in terms of morphology and control. Moreover, the absence of comprehensive and effective optimization frameworks has hindered systematic progress in this field. Bio-inspired approaches offer a promising alternative for addressing these limitations. This thesis is motivated by research indicating that success in nature arises from the synergy of two fundamental mechanisms: the inheritance of traits refined over evolutionary timescales, and the ability of individuals to adaptively exploit these traits through learning during their lifetime. Translated into the computational domain, this perspective motivates the integration of evolutionary algorithms with reinforcement learning to form hybrid methodologies capable of addressing highly complex engineering problems. Such an approach is particularly well suited to underwater environments, which are characterized by strong nonlinearities, uncertainty, and limited observability. Moreover, the co-optimization of morphology and control presents significant potential for improving system performance and design efficiency. The ultimate goal is to propose computational methods that are not only effective but also more efficient and adaptable than existing state-of-the-art techniques in underwater robotics. Within this context, the thesis seeks to address the following research questions: - Given the pivotal role of morphology in determining the performance of underwater vehicles, can evolutionary algorithms be effectively employed for morphological optimization in this domain? - What are the benefits and limitations of co-optimizing both morphology and motion control in underwater vehicles, as opposed to treating them separately? - How can reinforcement learning be systematically applied to design and optimize motion controllers for autonomous underwater vehicles, and under what conditions does it yield robust and reliable performance? - What are the practical trade-offs, advantages, and limitations of integrating evolutionary algorithms with reinforcement learning in this application domain? - Can such hybrid approaches generalize across different underwater tasks and environmental conditions, thereby demonstrating broader applicability? - Can insights from this hybrid optimization framework be extended to other domains of robotics and complex systems beyond underwater applications? Major Scientific Contributions: The primary scientific contributions of this thesis can be summarized as follows: - A novel open-source framework for AUV research was developed. It integrates evolutionary algorithms with reinforcement learning, supported by a physics-based simulator for realistic modeling in fluid environments. This platform lowers the entry barrier for optimization research in the field, enabling rapid prototyping, evaluation, and extension of new design approaches. [151] - In this framework, evolutionary algorithms were applied to optimize vehicle morphologies, while reinforcement learning was used to train their controllers. The results provided strong evidence for the central hypothesis that morphology and behavior are fundamentally interdependent: different reinforcement learning algorithms performed variably across evolved morphologies, and certain morphologies only revealed their potential when paired with suitable controllers. This mirrors biological systems, where form and function co-adapt rather than evolve in isolation. [151] - The framework was further enhanced to address complex control tasks, particularly path-following under realistic hydrodynamic conditions. A new methodology was introduced that couples morphological evolution with reinforcement learning in custom environments, using a step-wise training strategy to decompose the task into manageable stages. This approach overcame the limitations of direct training, achieving consistent success in realistic underwater scenarios. Optimized morphologies revealed recurring structural motifs among top-performing solutions, underscoring the importance of concurrent optimization of form and control. [150] - The applicability of evolutionary algorithms to the joint optimization of morphology and behavior in underwater vehicles was validated through a case study on pollution source localization in a fluid pipe system. The framework showed that evolutionary multi-objective optimization can effectively and simultaneously guide physical design choices and sensory strategy development. Simulation results confirmed the feasibility of the approach and established a solid foundation for more advanced research on the co-design of morphology and control systems. [147, 66] - The applicability of reinforcement learning for controller optimization in AUVs was demonstrated across navigation, obstacle avoidance, and underwater pipeline inspection tasks, with simulation results confirming the feasibility of the proposed method. [148, 149] - Collectively, these contributions validate the central thesis: effective underwater robotics requires the synergy of morphological evolution and learning. Morphological optimization alone is insufficient without the adaptive capabilities provided by learning, while reinforcement learning alone cannot compensate for poorly designed morphologies. It is the integration of both processes that enables robust performance in complex environments. [147, 150, 151]

Authors 1

1. Cagatay Sariman corresponding Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen