Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with
  Dynamic Gaussian Surfels
Video generative models are receiving particular attention given their
ability to generate realistic and imaginative frames. Besides, these models are
also observed to exhibit strong 3D consistency, significantly enhancing their
potential to act as world simulators. In this work, we present Vidu4D, a novel
reconstruction model that excels in accurately reconstructing 4D (i.e.,
sequential 3D) representations from single generated videos, addressing
challenges associated with non-rigidity and frame distortion. This capability
is pivotal for creating high-fidelity virtual contents that maintain both
spatial and temporal coherence. At the core of Vidu4D is our proposed Dynamic
Gaussian Surfels (DGS) technique. DGS optimizes time-varying warping functions
to transform Gaussian surfels (surface elements) from a static state to a
dynamically warped state. This transformation enables a precise depiction of
motion and deformation over time. To preserve the structural integrity of
surface-aligned Gaussian surfels, we design the warped-state geometric
regularization based on continuous warping fields for estimating normals.
Additionally, we learn refinements on rotation and scaling parameters of
Gaussian surfels, which greatly alleviates texture flickering during the
warping process and enhances the capture of fine-grained appearance details.
Vidu4D also contains a novel initialization state that provides a proper start
for the warping fields in DGS. Equipping Vidu4D with an existing video
generative model, the overall framework demonstrates high-fidelity text-to-4D
generation in both appearance and geometry.
6655503d43da5d2435dfd03d
Yikai Wang
extracted_confirmed
2024-05-28T05:46:08.794Z
64206b35e40f66bcd1e3c985
/avatars/a8f830d1c47702f642045399e59c9fe8.svg
Yikai Wang
user
yikaiw
6655503d43da5d2435dfd03e
Xinzhou Wang
admin_assigned
2024-05-28T10:00:30.795Z
6459c45b5a263a4c87b7c639
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/t3iK1wPBOP-h_kLBzZA_h.png
Xinzhou Wang
user
zz7379
6655503d43da5d2435dfd03f
Zilong Chen
admin_assigned
2024-05-28T10:03:46.000Z
644e3e5f030210812f413073
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uW8TKV2sds97lFBwnt6JK.jpeg
Zilong Chen
user
heheyas
6655503d43da5d2435dfd040
Zhengyi Wang
admin_assigned
2024-05-28T10:03:52.691Z
634e15aec1ce28f1de91c470
https://cdn-avatars.huggingface.co/v1/production/uploads/634e15aec1ce28f1de91c470/MdtJKgE_fbHydhqESr-DH.png
Zhengyi Wang
user
Zhengyi
6655503d43da5d2435dfd041
Fuchun Sun
6655503d43da5d2435dfd042
Jun Zhu
Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with
  Dynamic Gaussian Surfels
Video generative models are receiving particular attention given their
ability to generate realistic and imaginative frames. Besides, these models are
also observed to exhibit strong 3D consistency, significantly enhancing their
potential to act as world simulators. In this work, we present Vidu4D, a novel
reconstruction model that excels in accurately reconstructing 4D (i.e.,
sequential 3D) representations from single generated videos, addressing
challenges associated with non-rigidity and frame distortion. This capability
is pivotal for creating high-fidelity virtual contents that maintain both
spatial and temporal coherence. At the core of Vidu4D is our proposed Dynamic
Gaussian Surfels (DGS) technique. DGS optimizes time-varying warping functions
to transform Gaussian surfels (surface elements) from a static state to a
dynamically warped state. This transformation enables a precise depiction of
motion and deformation over time. To preserve the structural integrity of
surface-aligned Gaussian surfels, we design the warped-state geometric
regularization based on continuous warping fields for estimating normals.
Additionally, we learn refinements on rotation and scaling parameters of
Gaussian surfels, which greatly alleviates texture flickering during the
warping process and enhances the capture of fine-grained appearance details.
Vidu4D also contains a novel initialization state that provides a proper start
for the warping fields in DGS. Equipping Vidu4D with an existing video
generative model, the overall framework demonstrates high-fidelity text-to-4D
generation in both appearance and geometry.
Vidu4D, a novel reconstruction model using Dynamic Gaussian Surfels, accurately reconstructs 4D representations from single generated videos, addressing non-rigidity and frame distortion for high-fidelity virtual contents.
reconstruction model Dynamic Gaussian Surfels Gaussian surfels time-varying warping functions warped-state geometric regularization continuous warping fields rotation parameters scaling parameters texture flickering high-fidelity text-to-4D generation