Tool and Interactivity Detection for Automatic Assembly Instruction Generation

Procedia CIRP, vol. 130, pp. 611–618

Abstract

Manually generating instructions for industrial assembly processes demands considerable effort. This applies in particular to processes characterized by a high variance of products at low volumes. This challenge prevents companies from generating detailed assembly instructions. As a result, assembly operators are required to rely on implicit knowledge for product assembly. Due to demographic changes, capturing this implicit knowledge becomes increasingly relevant to prevent expertise loss. In order to reduce the effort for assembly instruction generation, in prior work, the authors presented an approach for automatic generation of assembly instructions. The approach analyzes video recordings of assembly processes with several independent computer vision models for object and activity recognition. The detected objects and activities are then converted into assembly instructions. While the previous work outlined the approach, the development of the individual models remains to be done. This paper presents a computer vision approach for detecting assembly-related objects, with a focus on tools. Moreover, the interactions of all assembly-related objects, i.e. tools, parts, and hands, are analyzed. Our methodology employs an experimental setup with an assembly workstation and a 2D RGB camera that is used to record assembly processes. Specific training videos were recorded for the training of the deep learning models. Rigorous evaluation demonstrates the effectiveness of the tool detector by achieving an average precision rate exceeding 90 % across our case studies involving 11 different types of tools. The primary contribution of this work is the development of a robust tool and interactivity detector, contributing to the implicit knowledge extraction for the generation of assembly instructions. In a broader context, our findings offer practical solutions for industries dependent on manual assembly processes, advancing automation and laying the foundation for future advancements in automatic assembly instruction generation.

Authors 5

1. Michael Riesener Aachen Laboratory for Machine Tools and Production Engineering (WZL)

   RWTH Aachen University

   Affiliation as printed

   Laboratory for Machine Tools and Production Engineering (WZL), RWTH Aachen University, Campus-Boulevard 30, 52074 Aachen, Germany

2. Esben Schukat Aachen Laboratory for Machine Tools and Production Engineering (WZL)

   RWTH Aachen University

   Affiliation as printed

   Laboratory for Machine Tools and Production Engineering (WZL), RWTH Aachen University, Campus-Boulevard 30, 52074 Aachen, Germany

3. Luis A. Curiel-Ramirez corresponding Aachen Laboratory for Machine Tools and Production Engineering (WZL)

   RWTH Aachen University

   Affiliation as printed

   Laboratory for Machine Tools and Production Engineering (WZL), RWTH Aachen University, Campus-Boulevard 30, 52074 Aachen, Germany

4. Florian Bröhl Aachen Laboratory for Machine Tools and Production Engineering (WZL)

   RWTH Aachen University

   Affiliation as printed

   Laboratory for Machine Tools and Production Engineering (WZL), RWTH Aachen University, Campus-Boulevard 30, 52074 Aachen, Germany

5. Nelvin R. Paul-Jayaraj Aachen Laboratory for Machine Tools and Production Engineering (WZL)

   RWTH Aachen University

   Affiliation as printed

   Laboratory for Machine Tools and Production Engineering (WZL), RWTH Aachen University, Campus-Boulevard 30, 52074 Aachen, Germany