UR-EARL: A framework for designing underwater robots using evolutionary algorithm-driven reinforcement learning

Ocean Engineering, vol. 321, pp. 120402

Abstract

Designing an autonomous underwater vehicle (AUV) is a significant challenge, which involves two primary tasks: shaping the body and developing the control system. Both elements are essential for the AUV’s effective performance. This study introduces a novel framework that integrates evolutionary algorithms with reinforcement learning to simultaneously optimize both the shape and behavior of AUVs toward specific objectives. Our framework is designed to be highly expandable, making it easy to incorporate and test new design algorithms alongside existing methods. Additionally, we provide an underwater simulation platform based on the Pybullet engine to realistically evaluate the performance of evolved/trained AUVs. To demonstrate the capabilities of our methodology, we present a case study focused on navigating and maintaining positions within diverse underwater environments with different flow dynamics. We use a Lindenmayer-System-based evolutionary algorithm for optimizing the AUV’s morphology and utilize Stable Baselines 3 algorithms to train the controllers within our proposed framework. Being entirely open-source, we expect this framework to provide researchers with a convenient platform for efficiently testing their innovative methods and accelerating progress in the field.

Authors 3

1. Cagatay Sariman corresponding Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University - Chair of Information Theory and Data Analytics, Kopernikusstraße 16, Aachen, 52074, NRW, Germany

2. Ahmed Hallawa Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University - Institute for Communication Technologies and Embedded Systems, Kopernikusstraße 16, Aachen, 52074, NRW, Germany

3. Anke Schmeink Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University - Chair of Information Theory and Data Analytics, Kopernikusstraße 16, Aachen, 52074, NRW, Germany