3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with
  View-consistent 2D Diffusion Priors
Novel-view synthesis aims to generate novel views of a scene from multiple
input images or videos, and recent advancements like 3D Gaussian splatting
(3DGS) have achieved notable success in producing photorealistic renderings
with efficient pipelines. However, generating high-quality novel views under
challenging settings, such as sparse input views, remains difficult due to
insufficient information in under-sampled areas, often resulting in noticeable
artifacts. This paper presents 3DGS-Enhancer, a novel pipeline for enhancing
the representation quality of 3DGS representations. We leverage 2D video
diffusion priors to address the challenging 3D view consistency problem,
reformulating it as achieving temporal consistency within a video generation
process. 3DGS-Enhancer restores view-consistent latent features of rendered
novel views and integrates them with the input views through a spatial-temporal
decoder. The enhanced views are then used to fine-tune the initial 3DGS model,
significantly improving its rendering performance. Extensive experiments on
large-scale datasets of unbounded scenes demonstrate that 3DGS-Enhancer yields
superior reconstruction performance and high-fidelity rendering results
compared to state-of-the-art methods. The project webpage is
https://xiliu8006.github.io/3DGS-Enhancer-project .
67181fa03ad5d8e375bbbb21
Xi Liu
claimed_verified
2024-10-23T07:33:17.542Z
648b15305461ecec939fb9c9
/avatars/c099e3847cc82d0b3230435feacb0319.svg
Liu
user
Xi8006
67181fa03ad5d8e375bbbb22
Chaoyi Zhou
admin_assigned
2024-10-24T08:39:11.650Z
65495d1008775ce78e43e77d
/avatars/172678e8187acfec0aa8e647479cbb81.svg
Chaoyi Zhou
user
chaoyizh
67181fa03ad5d8e375bbbb23
Siyu Huang
3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with
  View-consistent 2D Diffusion Priors
Novel-view synthesis aims to generate novel views of a scene from multiple
input images or videos, and recent advancements like 3D Gaussian splatting
(3DGS) have achieved notable success in producing photorealistic renderings
with efficient pipelines. However, generating high-quality novel views under
challenging settings, such as sparse input views, remains difficult due to
insufficient information in under-sampled areas, often resulting in noticeable
artifacts. This paper presents 3DGS-Enhancer, a novel pipeline for enhancing
the representation quality of 3DGS representations. We leverage 2D video
diffusion priors to address the challenging 3D view consistency problem,
reformulating it as achieving temporal consistency within a video generation
process. 3DGS-Enhancer restores view-consistent latent features of rendered
novel views and integrates them with the input views through a spatial-temporal
decoder. The enhanced views are then used to fine-tune the initial 3DGS model,
significantly improving its rendering performance. Extensive experiments on
large-scale datasets of unbounded scenes demonstrate that 3DGS-Enhancer yields
superior reconstruction performance and high-fidelity rendering results
compared to state-of-the-art methods. The project webpage is
https://xiliu8006.github.io/3DGS-Enhancer-project .
3DGS-Enhancer improves novel-view synthesis by using 2D video diffusion priors to address view consistency and refine 3DGS representations, achieving superior results on large-scale datasets.
3D Gaussian splatting 3DGS novel-view synthesis sparse input views view-consistent latent features 2D video diffusion spatial-temporal decoder