title: Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric
    Videos
summary: Volumetric video represents a transformative advancement in visual media,
  enabling users to freely navigate immersive virtual experiences and narrowing
  the gap between digital and real worlds. However, the need for extensive manual
  intervention to stabilize mesh sequences and the generation of excessively
  large assets in existing workflows impedes broader adoption. In this paper, we
  present a novel Gaussian-based approach, dubbed DualGS, for real-time
  and high-fidelity playback of complex human performance with excellent
  compression ratios. Our key idea in DualGS is to separately represent motion
  and appearance using the corresponding skin and joint Gaussians. Such an
  explicit disentanglement can significantly reduce motion redundancy and enhance
  temporal coherence. We begin by initializing the DualGS and anchoring skin
  Gaussians to joint Gaussians at the first frame. Subsequently, we employ a
  coarse-to-fine training strategy for frame-by-frame human performance modeling.
  It includes a coarse alignment phase for overall motion prediction as well as a
  fine-grained optimization for robust tracking and high-fidelity rendering. To
  integrate volumetric video seamlessly into VR environments, we efficiently
  compress motion using entropy encoding and appearance using codec compression
  coupled with a persistent codebook. Our approach achieves a compression ratio
  of up to 120 times, only requiring approximately 350KB of storage per frame. We
  demonstrate the efficacy of our representation through photo-realistic,
  free-view experiences on VR headsets, enabling users to immersively watch
  musicians in performance and feel the rhythm of the notes at the performers'
  fingertips.
paper_authors._id: 66e79c7195bb7553ed7600f4
paper_authors.name: Yuheng Jiang
paper_authors._id: 66e79c7195bb7553ed7600f5
paper_authors.name: Zhehao Shen
paper_authors._id: 66e79c7195bb7553ed7600f6
paper_authors.name: Yu Hong
paper_authors._id: 66e79c7195bb7553ed7600f7
paper_authors.name: Chengcheng Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-16T07:36:08.414Z
paper_authors._id: 64e4a9e6b78bc92221b2909e
paper_authors.avatarUrl: /avatars/b57b879d43fafa457cbad77dc4d387b1.svg
paper_authors.fullname: fy
paper_authors.type: user
paper_authors.user: chengchengguo123
paper_authors._id: 66e79c7195bb7553ed7600f8
paper_authors.name: Yize Wu
paper_authors._id: 66e79c7195bb7553ed7600f9
paper_authors.name: Yingliang Zhang
paper_authors._id: 66e79c7195bb7553ed7600fa
paper_authors.name: Jingyi Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-16T07:34:41.916Z
paper_authors._id: 63ffcec6e7767a89533b8f80
paper_authors.avatarUrl: /avatars/b9e8673943741c2e7b80b15a061bb951.svg
paper_authors.fullname: JINGYI Yu
paper_authors.type: user
paper_authors.user: Daluuu
paper_authors._id: 66e79c7195bb7553ed7600fb
paper_authors.name: Lan Xu
paper_title: Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric
    Videos
paper_summary: Volumetric video represents a transformative advancement in visual media,
  enabling users to freely navigate immersive virtual experiences and narrowing
  the gap between digital and real worlds. However, the need for extensive manual
  intervention to stabilize mesh sequences and the generation of excessively
  large assets in existing workflows impedes broader adoption. In this paper, we
  present a novel Gaussian-based approach, dubbed DualGS, for real-time
  and high-fidelity playback of complex human performance with excellent
  compression ratios. Our key idea in DualGS is to separately represent motion
  and appearance using the corresponding skin and joint Gaussians. Such an
  explicit disentanglement can significantly reduce motion redundancy and enhance
  temporal coherence. We begin by initializing the DualGS and anchoring skin
  Gaussians to joint Gaussians at the first frame. Subsequently, we employ a
  coarse-to-fine training strategy for frame-by-frame human performance modeling.
  It includes a coarse alignment phase for overall motion prediction as well as a
  fine-grained optimization for robust tracking and high-fidelity rendering. To
  integrate volumetric video seamlessly into VR environments, we efficiently
  compress motion using entropy encoding and appearance using codec compression
  coupled with a persistent codebook. Our approach achieves a compression ratio
  of up to 120 times, only requiring approximately 350KB of storage per frame. We
  demonstrate the efficacy of our representation through photo-realistic,
  free-view experiences on VR headsets, enabling users to immersively watch
  musicians in performance and feel the rhythm of the notes at the performers'
  fingertips.
paper_ai_keywords: Gaussian-based approach DualGS real-time playback high-fidelity playback complex human performance compression ratios motion representation appearance representation skin Gaussians joint Gaussians explicit disentanglement motion redundancy temporal coherence coarse-to-fine training strategy overall motion prediction fine-grained optimization robust tracking high-fidelity rendering volumetric video VR environments entropy encoding codec compression persistent codebook photo-realistic free-view experiences VR headsets