Interactive NeRF Geometry Editing With Shape Priors

IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, pp. 14821–14837

Abstract

Neural Radiance Fields (NeRFs) have shown great potential for tasks like novel view synthesis of static 3D scenes. Since NeRFs are trained on a large number of input images, it is not trivial to change their content afterwards. Previous methods to modify NeRFs provide some control but they do not support direct shape deformation which is common for geometry representations like triangle meshes. In this paper, we present a NeRF geometry editing method that first extracts a triangle mesh representation of the geometry inside a NeRF. This mesh can be modified by any 3D modeling tool (we use ARAP mesh deformation). The mesh deformation is then extended into a volume deformation around the shape which establishes a mapping between ray queries to the deformed NeRF and the corresponding queries to the original NeRF. The basic shape editing mechanism is extended towards more powerful and more meaningful editing handles by generating box abstractions of the NeRF shapes which provide an intuitive interface to the user. By additionally assigning semantic labels, we can even identify and combine parts from different objects. We demonstrate the performance and quality of our method in a number of experiments on synthetic data as well as real captured scenes.

Authors 7

1. Yu-Jie Yuan

   Chinese Academy of Sciences · Institute of Computing Technology · University of Chinese Academy of Sciences

   Affiliation as printed

   Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

   University of Chinese Academy of Sciences, Beijing, China

2. Yang-Tian Sun

   Chinese Academy of Sciences · Institute of Computing Technology · University of Chinese Academy of Sciences

   Affiliation as printed

   Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

   University of Chinese Academy of Sciences, Beijing, China

3. Yu‐Kun Lai

   Cardiff University

   Affiliation as printed

   School of Computer Science & Informatics, Cardiff University, Cardiff, U.K

4. Yuewen Ma

   ByteDance (China)

   Affiliation as printed

   PICO, ByteDance, Beijing, China

5. Rongfei Jia
   Affiliation as printed

   Beijing ZaoWuHuiJing Technology Company Ltd., Beijing, China

6. Leif P. Kobbelt Aachen Computer Graphics Group

   RWTH Aachen University

   Affiliation as printed

   Computer Graphics Group, RWTH Aachen University, Aachen, Germany

7. Lin Gao

   Chinese Academy of Sciences · Institute of Computing Technology · University of Chinese Academy of Sciences

   Affiliation as printed

   Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

   University of Chinese Academy of Sciences, Beijing, China