Vid3D: Synthesis of Dynamic 3D Scenes using 2D Video Diffusion
A recent frontier in computer vision has been the task of 3D video
generation, which consists of generating a time-varying 3D representation of a
scene. To generate dynamic 3D scenes, current methods explicitly model 3D
temporal dynamics by jointly optimizing for consistency across both time and
views of the scene. In this paper, we instead investigate whether it is
necessary to explicitly enforce multiview consistency over time, as current
approaches do, or if it is sufficient for a model to generate 3D
representations of each timestep independently. We hence propose a model,
Vid3D, that leverages 2D video diffusion to generate 3D videos by first
generating a 2D "seed" of the video's temporal dynamics and then independently
generating a 3D representation for each timestep in the seed video. We evaluate
Vid3D against two state-of-the-art 3D video generation methods and find that
Vid3D is achieves comparable results despite not explicitly modeling 3D
temporal dynamics. We further ablate how the quality of Vid3D depends on the
number of views generated per frame. While we observe some degradation with
fewer views, performance degradation remains minor. Our results thus suggest
that 3D temporal knowledge may not be necessary to generate high-quality
dynamic 3D scenes, potentially enabling simpler generative algorithms for this
task.
667108b7938266072cceef58
Rishab Parthasarathy
claimed_verified
2024-06-18T07:17:51.901Z
648a05fc15fcd0bd3500afc6
/avatars/325fac90c66920a9d8cb5a2ff37099f4.svg
Rishab Parthasarathy
user
rishab-partha
667108b7938266072cceef59
Zack Ankner
admin_assigned
2024-06-18T15:30:37.171Z
5ffde8127581ab61acbeb4e2
https://cdn-avatars.huggingface.co/v1/production/uploads/5ffde8127581ab61acbeb4e2/l-UyUmSc23zLPXT5LGteh.jpeg
Zack Ankner
user
ankner
667108b7938266072cceef5a
Aaron Gokaslan
admin_assigned
2024-06-18T15:30:43.632Z
620573d0522e40b4a18d8763
/avatars/9353c064ef8ccac84d0397411d38fa90.svg
Aaron Gokaslan
user
Skylion007
Vid3D: Synthesis of Dynamic 3D Scenes using 2D Video Diffusion
A recent frontier in computer vision has been the task of 3D video
generation, which consists of generating a time-varying 3D representation of a
scene. To generate dynamic 3D scenes, current methods explicitly model 3D
temporal dynamics by jointly optimizing for consistency across both time and
views of the scene. In this paper, we instead investigate whether it is
necessary to explicitly enforce multiview consistency over time, as current
approaches do, or if it is sufficient for a model to generate 3D
representations of each timestep independently. We hence propose a model,
Vid3D, that leverages 2D video diffusion to generate 3D videos by first
generating a 2D "seed" of the video's temporal dynamics and then independently
generating a 3D representation for each timestep in the seed video. We evaluate
Vid3D against two state-of-the-art 3D video generation methods and find that
Vid3D is achieves comparable results despite not explicitly modeling 3D
temporal dynamics. We further ablate how the quality of Vid3D depends on the
number of views generated per frame. While we observe some degradation with
fewer views, performance degradation remains minor. Our results thus suggest
that 3D temporal knowledge may not be necessary to generate high-quality
dynamic 3D scenes, potentially enabling simpler generative algorithms for this
task.
Vid3D generates 3D videos using 2D video diffusion and independent 3D representations per timestep, demonstrating comparable performance to methods explicitly modeling 3D temporal dynamics.
Vid3D 3D video generation multiview consistency 2D video diffusion 3D representations temporal dynamics