Embodied AI Agent Framework for (Re)Programming Robotic Tasks in Flexible Assembly using LLMs and VLMs

Procedia CIRP, vol. 139, pp. 59–65

Abstract

The increasing product variability and skilled labor shortages in manufacturing intensify the need for more flexible and adaptive automation solutions, particularly in assembly systems. However, existing robotic automation typically requires expert-level programming, preventing adaptability to new tasks or product variants. This work presents a modular embodied AI agent framework that enables non-programming operators to implement and reconfigure robotic tasks through intuitive natural language (NL) commands in text or voice format. By integrating open-source Large Language Models (LLMs) and Visual Language Models (VLMs) with a Robot Operating System 2 (ROS2)-based stack, the agent translates user instructions into perception, grasp generation, and motion planning actions. The framework is evaluated in real-world experiments across four scenarios: low-level motion control, vision-guided cable grasping, visual scene feedback, and task-cycle recording with replay for scalable execution. Results show 100% command parsing accuracy for basic motions, reliable visual feedback, and 70% success in dual-cable manipulation, with failures mainly from trajectory planning or self-collisions. VLM inference latency proved highly hardware-dependent, with near real-time performance on high-end GPUs but significant slowdowns on consumer devices, highlighting deployment challenges. These findings show the potential of embodied AI agents to bridge NL interaction with robotic execution, lowering barriers to deploying adaptive, operator-friendly robotic systems in dynamic manufacturing.

Authors 6

1. Lucas Manassés Pinheiro de Souza corresponding Aachen Laboratory for Machine Tools and Production Engineering (WZL)

   RWTH Aachen University

   Affiliation as printed

   Laboratory for Machine Tools and Production Engineering (WZL), RWTH Aachen University, Campus Boulevard 30, 52074 Aachen, Germany

2. Abhishek J. Deshmukh Aachen Laboratory for Machine Tools and Production Engineering (WZL)

   RWTH Aachen University

   Affiliation as printed

   Laboratory for Machine Tools and Production Engineering (WZL), RWTH Aachen University, Campus Boulevard 30, 52074 Aachen, Germany

3. Kedar Rajpathak Kedar Rajpathak Aachen Laboratory for Machine Tools and Production Engineering (WZL)

   RWTH Aachen University

   Affiliation as printed

   Laboratory for Machine Tools and Production Engineering (WZL), RWTH Aachen University, Campus Boulevard 30, 52074 Aachen, Germany

4. Alexander Moriz Aachen Laboratory for Machine Tools and Production Engineering (WZL)

   RWTH Aachen University

   Affiliation as printed

   Laboratory for Machine Tools and Production Engineering (WZL), RWTH Aachen University, Campus Boulevard 30, 52074 Aachen, Germany

5. Amon Göppert Aachen Laboratory for Machine Tools and Production Engineering (WZL)

   RWTH Aachen University

   Affiliation as printed

   Laboratory for Machine Tools and Production Engineering (WZL), RWTH Aachen University, Campus Boulevard 30, 52074 Aachen, Germany

6. Robert Heinrich Schmitt Aachen Fraunhofer Institute for Production Technology Laboratory for Machine Tools and Production Engineering (WZL)

   Fraunhofer Institute for Production Technology IPT · RWTH Aachen University

   Affiliation as printed

   Fraunhofer Institute for Production Technology (IPT), 52074 Aachen, Germany

   Laboratory for Machine Tools and Production Engineering (WZL), RWTH Aachen University, Campus Boulevard 30, 52074 Aachen, Germany