DeferredGS: Decoupled and Relightable Gaussian Splatting With Deferred Shading

IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, pp. 6307–6319

Abstract

Reconstructing and editing 3D objects and scenes both play crucial roles in computer graphics and computer vision. Neural radiance fields (NeRFs) can achieve realistic reconstruction and editing results but suffer from inefficiency in rendering. Gaussian splatting significantly accelerates rendering by rasterizing Gaussian ellipsoids. However, Gaussian splatting utilizes a single Spherical Harmonic (SH) function to model both texture and lighting, limiting independent editing capabilities of these components. Recently, attempts have been made to decouple texture and lighting with the Gaussian splatting representation but may fail to produce plausible geometry and decomposition results on reflective scenes. Additionally, the forward shading technique they employ introduces noticeable blending artifacts during relighting, as the geometry attributes of Gaussians are optimized under the original illumination and may not be suitable for novel lighting conditions. To address these issues, we introduce DeferredGS, a method for decoupling and relighting the Gaussian splatting representation using deferred shading. To achieve successful decoupling, we model the illumination with a learnable environment map and define additional attributes such as texture parameters and normal direction on Gaussians, where the normal is distilled from a jointly trained signed distance function. More importantly, we apply deferred shading, resulting in more realistic relighting effects compared to previous methods. Both qualitative and quantitative experiments demonstrate the superior performance of DeferredGSin novel view synthesis and relighting tasks.

Authors 6

1. Tong Wu

   Chinese Academy of Sciences · Institute of Computing Technology

   Affiliation as printed

   Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

   Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, China

2. Jia-Mu Sun

   Chinese Academy of Sciences · Institute of Computing Technology

   Affiliation as printed

   Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

   Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, China

3. Yu‐Kun Lai

   Cardiff University

   Affiliation as printed

   School of Computer Science & Informatics, Cardiff University, Cardiff, U.K

   School of Computer Science & Informatics, Cardiff University, U.K

4. Yuewen Ma

   ByteDance (China)

   Affiliation as printed

   ByteDance Pico, Beijing, China

   ByteDance Pico, China

5. Leif P. Kobbelt Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University, Aachen, Germany

   RWTH Aachen University, Germany

6. Lin Gao

   Chinese Academy of Sciences · Institute of Computing Technology

   Affiliation as printed

   Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

   Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, China