OpenSplat3D: Open-Vocabulary 3D Instance Segmentation using Gaussian Splatting

IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 5246–5255

Abstract

3D Gaussian Splatting (3DGS) has emerged as a powerful representation for neural scene reconstruction, offering high-quality novel view synthesis while maintaining computational efficiency. In this paper, we extend the capabilities of 3DGS beyond pure scene representation by introducing an approach for open-vocabulary 3D instance segmentation without requiring manual labeling, termed OpenSplat3D. Our method leverages feature-splatting techniques to associate semantic information with individual Gaussians, enabling fine-grained scene understanding. We incorporate Segment Anything Model instance masks with a contrastive loss formulation as guidance for the instance features to achieve accurate instance-level segmentation. Furthermore, we utilize language embeddings of a visionlanguage model, allowing for flexible, text-driven instance identification. This combination enables our system to identify and segment arbitrary objects in 3D scenes based on natural language descriptions. We show results on LERFmask and LERF-OVS as well as the full ScanNet++ validation set, demonstrating the effectiveness of our approach.

Authors 6

1. Jens Piekenbrinck Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

2. Christian Schmidt Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

3. Alexander Hermans Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

4. Narunas Vaškevičius

   Robert Bosch (Germany)

   Affiliation as printed

   Robert Bosch GmbH

5. Timm Linder

   Robert Bosch (Germany)

   Affiliation as printed

   Robert Bosch GmbH

6. Bastian Leibe Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University