DINO in the Room: Leveraging 2D Foundation Models for 3D Segmentation

International Conference on 3D Vision (3DV), pp. 1884–1894

Abstract

Vision foundation models (VFMs) trained on large-scale image datasets provide high-quality features that have significantly advanced$2 D$visual recognition. However, their potential in 3D scene segmentation remains largely untapped, despite the common availability of$2 D$images alongside 3D point cloud datasets. While significant research has been dedicated to 2D-3D fusion, recent state-of-the-art 3D methods predominantly focus on 3D data, leaving the integration of VFMs into 3D models underexplored. In this work, we challenge this trend by introducing DITR, a generally applicable approach that extracts$2 D$foundation model features, projects them to 3D, and finally injects them into a 3D point cloud segmentation model. DITR achieves state-of-the-art results on both indoor and outdoor 3D semantic segmentation benchmarks. To enable the use of VFMs even when images are unavailable during inference, we additionally propose to pretrain 3D models by distilling 2D foundation models. By initializing the 3D backbone with knowledge distilled from 2D VFMs, we create a strong basis for downstream 3D segmentation tasks, ultimately boosting performance across various datasets.

Authors 7

1. Karim Knaebel Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

2. Kadir Yilmaz Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

3. Daan de Geus Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

4. Alexander Hermans Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

5. David B. Adrian

   Robert Bosch (India) · Robert Bosch (United States) · Robert Bosch (Taiwan)

   Affiliation as printed

   Bosch Center for AI

6. Timm Linder

   Robert Bosch (India) · Robert Bosch (United States) · Robert Bosch (Taiwan)

   Affiliation as printed

   Bosch Center for AI

7. Bastian Leibe Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University