title: Vid3D: Synthesis of Dynamic 3D Scenes using 2D Video Diffusion
summary: A recent frontier in computer vision has been the task of 3D video
  generation, which consists of generating a time-varying 3D representation of a
  scene. To generate dynamic 3D scenes, current methods explicitly model 3D
  temporal dynamics by jointly optimizing for consistency across both time and
  views of the scene. In this paper, we instead investigate whether it is
  necessary to explicitly enforce multiview consistency over time, as current
  approaches do, or if it is sufficient for a model to generate 3D
  representations of each timestep independently. We hence propose a model,
  Vid3D, that leverages 2D video diffusion to generate 3D videos by first
  generating a 2D "seed" of the video's temporal dynamics and then independently
  generating a 3D representation for each timestep in the seed video. We evaluate
  Vid3D against two state-of-the-art 3D video generation methods and find that
  Vid3D is achieves comparable results despite not explicitly modeling 3D
  temporal dynamics. We further ablate how the quality of Vid3D depends on the
  number of views generated per frame. While we observe some degradation with
  fewer views, performance degradation remains minor. Our results thus suggest
  that 3D temporal knowledge may not be necessary to generate high-quality
  dynamic 3D scenes, potentially enabling simpler generative algorithms for this
  task.
paper_authors._id: 667108b7938266072cceef58
paper_authors.name: Rishab Parthasarathy
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:17:51.901Z
paper_authors._id: 648a05fc15fcd0bd3500afc6
paper_authors.avatarUrl: /avatars/325fac90c66920a9d8cb5a2ff37099f4.svg
paper_authors.fullname: Rishab Parthasarathy
paper_authors.type: user
paper_authors.user: rishab-partha
paper_authors._id: 667108b7938266072cceef59
paper_authors.name: Zack Ankner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:30:37.171Z
paper_authors._id: 5ffde8127581ab61acbeb4e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5ffde8127581ab61acbeb4e2/l-UyUmSc23zLPXT5LGteh.jpeg
paper_authors.fullname: Zack Ankner
paper_authors.type: user
paper_authors.user: ankner
paper_authors._id: 667108b7938266072cceef5a
paper_authors.name: Aaron Gokaslan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:30:43.632Z
paper_authors._id: 620573d0522e40b4a18d8763
paper_authors.avatarUrl: /avatars/9353c064ef8ccac84d0397411d38fa90.svg
paper_authors.fullname: Aaron Gokaslan
paper_authors.type: user
paper_authors.user: Skylion007
paper_title: Vid3D: Synthesis of Dynamic 3D Scenes using 2D Video Diffusion
paper_summary: A recent frontier in computer vision has been the task of 3D video
  generation, which consists of generating a time-varying 3D representation of a
  scene. To generate dynamic 3D scenes, current methods explicitly model 3D
  temporal dynamics by jointly optimizing for consistency across both time and
  views of the scene. In this paper, we instead investigate whether it is
  necessary to explicitly enforce multiview consistency over time, as current
  approaches do, or if it is sufficient for a model to generate 3D
  representations of each timestep independently. We hence propose a model,
  Vid3D, that leverages 2D video diffusion to generate 3D videos by first
  generating a 2D "seed" of the video's temporal dynamics and then independently
  generating a 3D representation for each timestep in the seed video. We evaluate
  Vid3D against two state-of-the-art 3D video generation methods and find that
  Vid3D is achieves comparable results despite not explicitly modeling 3D
  temporal dynamics. We further ablate how the quality of Vid3D depends on the
  number of views generated per frame. While we observe some degradation with
  fewer views, performance degradation remains minor. Our results thus suggest
  that 3D temporal knowledge may not be necessary to generate high-quality
  dynamic 3D scenes, potentially enabling simpler generative algorithms for this
  task.
paper_ai_keywords: 3D video generation 3D temporal dynamics multiview consistency 2D video diffusion 2D "seed" 3D representation timestep performance degradation