Assessment of Simulated Data for Training Semantic Segmentation-based Perception Models in Construction

Procedia Computer Science, vol. 274, pp. 220–229

Abstract

This research investigates the effectiveness of simulated data for training semantic segmentation-based perception models for object detection on construction sites, a context severely limited by the scarcity of relevant annotated real-world data. We explore purely simulated datasets, purely real datasets, and various hybrid combinations to quantify their respective performance and data efficiency. Using metrics for data distribution (FID, KID), learning performance (Accuracy, IoU), and efficiency (normalized Real-Data Efficiency, nRDE), we show that purely simulated data results in substantial performance degradation due to domain gaps. However, hybrid datasets significantly reduce the required volume of real annotated images while achieving acceptable accuracy. Our results highlight the detrimental role of catastrophic forgetting in sequential fine-tuning approaches, emphasizing instead the advantage of combining simulated and real data simultaneously. We further outline potential simulation improvements to reduce the domain gap. Our findings demonstrate that enhanced hybrid datasets hold considerable promise for practical model deployment in the Of-Highway Twins 2 (OHT-2) research project, especially given real annotated data collection constraints inherent in construction environments.

Authors 5

1. Jorge Luis Jiménez Aparicio corresponding
2. David Böken Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University - Institute for Man-Machine Interaction (MMI), Im Süsterfeld 9, Aachen, 52072, Germany

3. Johannes Sprink Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University – Institute for Fluid Power Drives and Systems (ifas), Campus-Boulevard 30, Aachen, 52074, Germany

4. Xiaoting Wang Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University - Institute for Man-Machine Interaction (MMI), Im Süsterfeld 9, Aachen, 52072, Germany

5. Jürgen Roßmann Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University - Institute for Man-Machine Interaction (MMI), Im Süsterfeld 9, Aachen, 52072, Germany