Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis

International Conference on 3D Vision (3DV), pp. 800–809

Abstract

We present a method that simultaneously addresses the tasks of dynamic scene novel-view synthesis and six degree-of-freedom (6-DOF) tracking of all dense scene elements. We follow an analysis-by-synthesis framework, inspired by recent work that models scenes as a collection of 3D Gaussians which are optimized to reconstruct input images via differentiable rendering. To model dynamic scenes, we allow Gaussians to move and rotate over time while enforcing that they have persistent color, opacity, and size. By regularizing Gaussians’ motion and rotation with local-rigidity constraints, we show that our Dynamic 3D Gaussians correctly model the same area of physical space over time, including the rotation of that space. Dense 6-DOF tracking and dynamic reconstruction emerges naturally from persistent dynamic view synthesis, without requiring any correspondence or flow as input. We demonstrate a large number of downstream applications enabled by our representation, including first-person view synthesis, dynamic compositional scene synthesis, and 4D video editing.111. Project Website: dynamic3dgaussians.github.io

Authors 4

1. Jonathon Luiten Aachen

   Carnegie Mellon University · RWTH Aachen University

   Affiliation as printed

   Carnegie Mellon University,USA

   Carnegie Mellon University, USA

   RWTH Aachen University, Germany

2. Georgios Kopanas

   Institut national de recherche en sciences et technologies du numérique

   Affiliation as printed

   Inria & Université Côte d’Azur,France

3. Bastian Leibe Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Germany

   RWTH Aachen University, Germany

4. Deva Ramanan

   Carnegie Mellon University

   Affiliation as printed

   Carnegie Mellon University,USA

   Carnegie Mellon University, USA