Mask3D: Mask Transformer for 3D Semantic Instance Segmentation

Proceedings - IEEE International Conference on Robotics and Automation/Proceedings, pp. 8216–8223

Abstract

Modern 3D semantic instance segmentation approaches predominantly rely on specialized voting mechanisms followed by carefully designed geometric clustering techniques. Building on the successes of recent Transformer-based methods for object detection and image segmentation, we propose the first Transformer-based approach for 3D semantic instance segmentation. We show that we can leverage generic Transformer building blocks to directly predict instance masks from 3D point clouds. In our model - called Mask3D - each object instance is represented as an instance query. Using Transformer decoders, the instance queries are learned by iteratively attending to point cloud features at multiple scales. Combined with point features, the instance queries directly yield all instance masks in parallel. Mask3D has several advantages over current state-of-the-art approaches, since it neither relies on (1) voting schemes which require hand-selected geometric properties (such as centers) nor (2) geometric grouping mechanisms requiring manually-tuned hyper-parameters (e.g. radii) and (3) enables a loss that directly optimizes instance masks. Mask3D sets a new state-of-the-art on ScanNet test (+6.2mAP), S3DIS 6-fold (+10.1 mAP), STPLS3D (+11.2 mAP) and ScanNet200 test (+12.4 mAP).

Authors 6

1. Jonas Schult Aachen Computer Vision Group

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Computer Vision Group,Germany

   Computer Vision Group, RWTH Aachen University, Germany

2. Francis Engelmann

   ETH Zurich

   Affiliation as printed

   ETH Zürich,Computer Vision and Learning Group,Switzerland

3. Alexander Hermans Aachen Computer Vision Group

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Computer Vision Group,Germany

   Computer Vision Group, RWTH Aachen University, Germany

4. Or Litany

   Nvidia (United States)

   Affiliation as printed

   NVIDIA,Santa Clara,USA

   NVIDIA, Santa Clara, USA

5. Siyu Tang

   ETH Zurich

   Affiliation as printed

   ETH Zürich,Computer Vision and Learning Group,Switzerland

6. Bastian Leibe Aachen Computer Vision Group

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Computer Vision Group,Germany

   Computer Vision Group, RWTH Aachen University, Germany