Flexible voice control for collaborative robotics

at - Automatisierungstechnik, vol. 73, pp. 341–351

Abstract

Abstract In this paper, we present a novel combination of an artificial intelligence (AI)-based and a grammar-based approach for a voice control system (VCS) for collaborative robots in industrial environments. A behavior tree (BT) serves as a high-level control system. A zero-shot object detection model is utilized to ensure the detection of different objects in the workspace. The grammar-based VCS is used for predefined actions. If a command does not trigger any action, a fine-tuned large-language model (LLM) is utilized for task planning. In order to further improve the model, interactions with the LLM are stored and processed to collect additional training data. The system is evaluated in real-world experiments, using a collaborative robotic arm in conjunction with a static calibrated depth camera that provides a birds-eye view.

Authors 5

1. David Kötter Aachen Laboratory for Machine Tools and Production Engineering WZL

   RWTH Aachen University

   Affiliation as printed

   Laboratory for Machine Tools and Production Engineering WZL , RWTH Aachen University , Steinbachstraße 19, 52074 Aachen , Germany

2. Denis Teknetas Aachen Laboratory for Machine Tools and Production Engineering WZL

   RWTH Aachen University

   Affiliation as printed

   Laboratory for Machine Tools and Production Engineering WZL , RWTH Aachen University , Aachen , Germany

3. Melanie Kolditz Aachen Faculty of Electrical Engineering and Information Technology

   FH Aachen

   Affiliation as printed

   Faculty of Electrical Engineering and Information Technology , FH Aachen University of Applied Science , Aachen , Germany

4. Oliver Petrović Aachen Laboratory for Machine Tools and Production Engineering WZL

   RWTH Aachen University

   Affiliation as printed

   Laboratory for Machine Tools and Production Engineering WZL , RWTH Aachen University , Aachen , Germany

5. Christian Brecher Aachen Laboratory for Machine Tools and Production Engineering WZL

   RWTH Aachen University

   Affiliation as printed

   Laboratory for Machine Tools and Production Engineering WZL , RWTH Aachen University , Aachen , Germany