title: Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with
    Dynamic Gaussian Surfels
summary: Video generative models are receiving particular attention given their
  ability to generate realistic and imaginative frames. Besides, these models are
  also observed to exhibit strong 3D consistency, significantly enhancing their
  potential to act as world simulators. In this work, we present Vidu4D, a novel
  reconstruction model that excels in accurately reconstructing 4D (i.e.,
  sequential 3D) representations from single generated videos, addressing
  challenges associated with non-rigidity and frame distortion. This capability
  is pivotal for creating high-fidelity virtual contents that maintain both
  spatial and temporal coherence. At the core of Vidu4D is our proposed Dynamic
  Gaussian Surfels (DGS) technique. DGS optimizes time-varying warping functions
  to transform Gaussian surfels (surface elements) from a static state to a
  dynamically warped state. This transformation enables a precise depiction of
  motion and deformation over time. To preserve the structural integrity of
  surface-aligned Gaussian surfels, we design the warped-state geometric
  regularization based on continuous warping fields for estimating normals.
  Additionally, we learn refinements on rotation and scaling parameters of
  Gaussian surfels, which greatly alleviates texture flickering during the
  warping process and enhances the capture of fine-grained appearance details.
  Vidu4D also contains a novel initialization state that provides a proper start
  for the warping fields in DGS. Equipping Vidu4D with an existing video
  generative model, the overall framework demonstrates high-fidelity text-to-4D
  generation in both appearance and geometry.
paper_authors._id: 6655503d43da5d2435dfd03d
paper_authors.name: Yikai Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-05-28T05:46:08.794Z
paper_authors._id: 64206b35e40f66bcd1e3c985
paper_authors.avatarUrl: /avatars/a8f830d1c47702f642045399e59c9fe8.svg
paper_authors.fullname: Yikai Wang
paper_authors.type: user
paper_authors.user: yikaiw
paper_authors._id: 6655503d43da5d2435dfd03e
paper_authors.name: Xinzhou Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:00:30.795Z
paper_authors._id: 6459c45b5a263a4c87b7c639
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/t3iK1wPBOP-h_kLBzZA_h.png
paper_authors.fullname: Xinzhou Wang
paper_authors.type: user
paper_authors.user: zz7379
paper_authors._id: 6655503d43da5d2435dfd03f
paper_authors.name: Zilong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:03:46.000Z
paper_authors._id: 644e3e5f030210812f413073
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uW8TKV2sds97lFBwnt6JK.jpeg
paper_authors.fullname: Zilong Chen
paper_authors.type: user
paper_authors.user: heheyas
paper_authors._id: 6655503d43da5d2435dfd040
paper_authors.name: Zhengyi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:03:52.691Z
paper_authors._id: 634e15aec1ce28f1de91c470
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634e15aec1ce28f1de91c470/MdtJKgE_fbHydhqESr-DH.png
paper_authors.fullname: Zhengyi Wang
paper_authors.type: user
paper_authors.user: Zhengyi
paper_authors._id: 6655503d43da5d2435dfd041
paper_authors.name: Fuchun Sun
paper_authors._id: 6655503d43da5d2435dfd042
paper_authors.name: Jun Zhu
paper_title: Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with
    Dynamic Gaussian Surfels
paper_summary: Video generative models are receiving particular attention given their
  ability to generate realistic and imaginative frames. Besides, these models are
  also observed to exhibit strong 3D consistency, significantly enhancing their
  potential to act as world simulators. In this work, we present Vidu4D, a novel
  reconstruction model that excels in accurately reconstructing 4D (i.e.,
  sequential 3D) representations from single generated videos, addressing
  challenges associated with non-rigidity and frame distortion. This capability
  is pivotal for creating high-fidelity virtual contents that maintain both
  spatial and temporal coherence. At the core of Vidu4D is our proposed Dynamic
  Gaussian Surfels (DGS) technique. DGS optimizes time-varying warping functions
  to transform Gaussian surfels (surface elements) from a static state to a
  dynamically warped state. This transformation enables a precise depiction of
  motion and deformation over time. To preserve the structural integrity of
  surface-aligned Gaussian surfels, we design the warped-state geometric
  regularization based on continuous warping fields for estimating normals.
  Additionally, we learn refinements on rotation and scaling parameters of
  Gaussian surfels, which greatly alleviates texture flickering during the
  warping process and enhances the capture of fine-grained appearance details.
  Vidu4D also contains a novel initialization state that provides a proper start
  for the warping fields in DGS. Equipping Vidu4D with an existing video
  generative model, the overall framework demonstrates high-fidelity text-to-4D
  generation in both appearance and geometry.
paper_ai_summary: Vidu4D, a novel reconstruction model using Dynamic Gaussian Surfels, accurately reconstructs 4D representations from single generated videos, addressing non-rigidity and frame distortion for high-fidelity virtual contents.
paper_ai_keywords: reconstruction model Dynamic Gaussian Surfels Gaussian surfels time-varying warping functions warped-state geometric regularization continuous warping fields rotation parameters scaling parameters texture flickering high-fidelity text-to-4D generation