ExploRLLM: Guiding Exploration in Reinforcement Learning with Large Language Models

Proceedings - IEEE International Conference on Robotics and Automation/Proceedings, pp. 9011–9017

Abstract

In robot manipulation, Reinforcement Learning (RL) often suffers from low sample efficiency and uncertain convergence, especially in large observation and action spaces. Foundation Models (FMs) offer an alternative, demonstrating promise in zero-shot and few-shot settings. However, they can be unreliable due to limited physical and spatial understanding. We introduce ExploRLLM, a method that combines the strengths of both paradigms. In our approach, FMs improve RL convergence by generating policy code and efficient representations, while a residual RL agent compensates for the FMs' limited physical understanding. We show that Explorllm outperforms both policies derived from FMs and RL baselines in table-top manipulation tasks. Additionally, real-world experiments show that the policies exhibit promising zero-shot sim-to-real transfer. Supplementary material is available at https://explorllm.github.io.

Authors 4

1. Runyu Ma

   Delft University of Technology

   Affiliation as printed

   Cognitive Robotics, Delft University of Technology,The Netherlands

2. Jelle Luijkx

   Delft University of Technology

   Affiliation as printed

   Cognitive Robotics, Delft University of Technology,The Netherlands

3. Zlatan Ajanović Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Germany

4. Jens Kober

   Delft University of Technology

   Affiliation as printed

   Cognitive Robotics, Delft University of Technology,The Netherlands