Hybrid Zero-Shot Interactive Navigation with LLMs: Path Planning Under Dual Constraints of Speech and Environment

Robotics, vol. 15, pp. 167

Abstract

This study proposes a novel hybrid interactive navigation framework for mobile robots, designed to enable robots to operate under dual constraints imposed by both the environment and the speech of accompanying humans in challenging future collaborative working scenarios. By leveraging human perceptual capabilities, the proposed framework significantly enhances the obstacle-avoidance capabilities and flexibility of robots. Specifically, an innovative composition algorithm is introduced to integrate traditional costmap-based navigation with a newly proposed LLM-assisted voice-based interaction method, thereby achieving real-time human–robot collaborative navigation with complementary advantages. Within this framework, robots can not only rely on spatial sensors to avoid obstacles but also follow verbal instructions from humans to bypass hazards that are difficult to detect. Moreover, the volume of speech is innovatively incorporated as a fusion weight, allowing the accompanying human to naturally guide the robot through voice volume modulation. To validate the feasibility and performance of the proposed framework and algorithm, we conducted both simulation and real-world experiments. A series of ablation and comparative studies was conducted to evaluate the merits and limitations of various configurations, ultimately providing optimal configurations based on the results. This work expands the scope of real-time human–robot interaction in navigation, offering new perspectives for future research.

Authors 4

1. Fan Yang corresponding Aachen Institute of Construction Management

   RWTH Aachen University

   Affiliation as printed

   Institute of Construction Management, Digital Engineering and Robotics in Construction, RWTH Aachen University, 52062 Aachen, Germany

2. Jing Wu Aachen Department of Computer Science

   RWTH Aachen University

   Affiliation as printed

   Department of Computer Science, RWTH Aachen University, 52062 Aachen, Germany

3. Timur Kuzu Aachen Institute of Construction Management

   RWTH Aachen University

   Affiliation as printed

   Institute of Construction Management, Digital Engineering and Robotics in Construction, RWTH Aachen University, 52062 Aachen, Germany

4. Katharina Klemt-Albert Aachen Institute of Construction Management

   RWTH Aachen University

   Affiliation as printed

   Institute of Construction Management, Digital Engineering and Robotics in Construction, RWTH Aachen University, 52062 Aachen, Germany