LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal Assembly Assistants
INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION, pp. 87–98
Abstract
Vision-language models (VLMs) are facing the challenges of understanding and following multimodal assembly instructions, particularly when fine-grained spatial reasoning and precise object state detection are required. In this work, we explore LEGO Co-builder, a hybrid benchmark combining real-world LEGO assembly logic with programmatically generated multimodal scenes. The dataset captures stepwise visual states and procedural instructions, allowing controlled evaluation of instruction-following, object detection, and state detection. We introduce a unified framework and assess leading VLMs such as GPT-4o, Gemini, and Qwen-VL, under zero-shot and fine-tuned settings. We also evaluated the framework using a reasoning-focused model, GLM-4.1-thinking. Our results show that while object detection achieved high performance (98.16% with fine-tuned InstructBLIP), fine-grained scene understanding and assembly state detection remain challenging: Fine-tuned MiniGPT-v2 reached only 37.52% F1 for identifying theme entities, and even advanced models such as GPT-4o achieved just 40.54% F1 on state detection. This highlights gaps in fine-grained visual understanding among existing models. We release the benchmark, codebase, and generation pipeline to support future research on multimodal assembly assistants grounded in real-world workflows.
Authors 14
-
Centrum Wiskunde & Informatica
Affiliation as printed
Vrije University Amsterdam, Amsterdam, Netherlands; University of Amsterdam, Amsterdam, Netherlands and Centrum Wiskunde & Informatica, Amsterdam, Netherlands
-
Affiliation as printed
Vrije University Amsterdam, Amsterdam, Netherlands
-
National Institute of Informatics
Affiliation as printed
National Institute of Informatics, Tokyo, Japan and University of Amsterdam, Amsterdam, Netherlands
-
Centrum Wiskunde & Informatica
Affiliation as printed
Centrum Wiskunde & Informatica, Amsterdam, Netherlands
-
Affiliation as printed
University of Amsterdam, Amsterdam, Netherlands
-
Centrum Wiskunde & Informatica
Affiliation as printed
Centrum Wiskunde en Informatica, Amsterdam, Netherlands
-
Zhaochun Ren Aachen
Affiliation as printed
Leiden University, Leiden, Netherlands
-
Affiliation as printed
UCL Interaction Centre, University College London, London, United Kingdom
-
Affiliation as printed
Vrije Universiteit Amsterdam, Amsterdam, Netherlands
-
Affiliation as printed
Vrije Universiteit Amterdam, Amsterdam, Netherlands
-
Affiliation as printed
Vrije Universiteit Amterdam, Amsterdam, Netherlands
-
Centrum Wiskunde & Informatica · Delft University of Technology
Affiliation as printed
Centrum Wiskunde & Informatica, Amsterdam, Netherlands and Delft University of Technology, Delft, Netherlands
-
Affiliation as printed
Guangzhou University, Guangzhou, China
-
Affiliation as printed
Vrije Universiteit Amsterdam, Amsterdam, Netherlands
Cited by 0 stored of 0
No patents citing this paper on Lens.org (checked 2026-10-11).