Simulation-Aided Policy Tuning for Black-Box Robot Learning

IEEE Transactions on Robotics, vol. 41, pp. 2533–2548

Abstract

How can robots learn and adapt to new tasks and situations with little data? Systematic exploration and simulation are crucial tools for efficient robot learning. We present a novel black-box policy search algorithm focused on data-efficient policy improvements. The algorithm learns directly on the robot and treats simulation as an additional information source to speed up the learning process. At the core of the algorithm, a probabilistic model learns the dependence between the policy parameters and the robot learning objective not only by performing experiments on the robot, but also by leveraging data from a simulator. This substantially reduces interaction time with the robot. Using the model, we can guarantee improvements with high probability for each policy update, thereby facilitating fast, goal-oriented learning. We evaluate our algorithm on simulated fine-tuning tasks and demonstrate the data-efficiency of the proposed dual-information source optimization algorithm. In a real robot learning experiment, we show fast and successful task learning on a robot manipulator with the aid of an imperfect simulator.

Authors 4

1. Alexander von Rohr Aachen Institute for Data Science in Mechanical Engineering

   RWTH Aachen University

   Affiliation as printed

   Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Aachen, Germany

2. Dominik Baumann

   Aalto University

   Affiliation as printed

   Department of Electrical Engineering and Automation, Aalto University, Espoo, Finland

3. Xiang Ji

   Zhejiang University

   Affiliation as printed

   College of Electrical Engineering, Zhejiang University, Hangzhou, China

4. Sebastian Trimpe Aachen Institute for Data Science in Mechanical Engineering

   RWTH Aachen University

   Affiliation as printed

   Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Aachen, Germany