LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistants

arXiv (Cornell University)

Abstract

Vision-language models (VLMs) are facing the challenges of understanding and following multimodal assembly instructions, particularly when fine-grained spatial reasoning and precise object state detection are required. In this work, we explore LEGO Co-builder, a hybrid benchmark combining real-world LEGO assembly logic with programmatically generated multimodal scenes. The dataset captures stepwise visual states and procedural instructions, allowing controlled evaluation of instruction-following, object detection, and state detection. We introduce a unified framework and assess leading VLMs such as GPT-4o, Gemini, and Qwen-VL, under zero-shot and fine-tuned settings. We also evaluated the framework using a reasoning-focused model, GLM-4.1-thinking. Our results show that while object detection achieved high performance (98.16% with fine-tuned InstructBLIP), fine-grained scene understanding and assembly state detection remain challenging: Fine-tuned MiniGPT-v2 reached only 37.52% F1 for identifying theme entities, and even advanced models such as GPT-4o achieved just 40.54% F1 on state detection. This highlights gaps in fine-grained visual understanding among existing models. We release the benchmark, codebase, and generation pipeline to support future research on multimodal assembly assistants grounded in real-world workflows.

Authors 14

1. Hao‐Chen Huang

   University College London

   Affiliation as printed

   University College London

2. Y W Su
3. Xin Sun

   Centrum Wiskunde & Informatica · National Institute of Informatics

   Affiliation as printed

   National Institute of Informatics ¶ Centrum Wiskunde & Informatic

4. Moonisa Ahsan

   Centrum Wiskunde & Informatica · National Institute of Informatics

   Affiliation as printed

   National Institute of Informatics ¶ Centrum Wiskunde & Informatic

5. Mohammad Aliannejadi

   University of Amsterdam

   Affiliation as printed

   University of Amsterdam

6. Irene Viola Aachen

   Leiden University · Centrum Wiskunde & Informatica

   Affiliation as printed

   Centrum Wiskunde & Informatic ∥ Leiden University *

7. Zhaochun Ren Aachen

   Leiden University · Centrum Wiskunde & Informatica

   Affiliation as printed

   Centrum Wiskunde & Informatic ∥ Leiden University *

8. C Shijia Yu

   University College London

   Affiliation as printed

   University College London

9. Aneta Lisowska

   Vrije Universiteit Amsterdam

   Affiliation as printed

   Vrije University of Amsterdam

10. Artem Belopolsky

    Vrije Universiteit Amsterdam

    Affiliation as printed

    Vrije University of Amsterdam

11. Koen Hindriks

    Vrije Universiteit Amsterdam

    Affiliation as printed

    Vrije University of Amsterdam

12. Pablo S. Cesar

    Delft University of Technology

    Affiliation as printed

    Technische Universiteit Delft

13. Junxiao Wang
14. Jiahuan Pei

    Centrum Wiskunde & Informatica · Vrije Universiteit Amsterdam · Delft University of Technology

    Affiliation as printed

    Technische Universiteit Delft

    Vrije University of Amsterdam Centrum Wiskunde & Informatic