title: World-consistent Video Diffusion with Explicit 3D Modeling
summary: Recent advancements in diffusion models have set new benchmarks in image and
  video generation, enabling realistic visual synthesis across single- and
  multi-frame contexts. However, these models still struggle with efficiently and
  explicitly generating 3D-consistent content. To address this, we propose
  World-consistent Video Diffusion (WVD), a novel framework that incorporates
  explicit 3D supervision using XYZ images, which encode global 3D coordinates
  for each image pixel. More specifically, we train a diffusion transformer to
  learn the joint distribution of RGB and XYZ frames. This approach supports
  multi-task adaptability via a flexible inpainting strategy. For example, WVD
  can estimate XYZ frames from ground-truth RGB or generate novel RGB frames
  using XYZ projections along a specified camera trajectory. In doing so, WVD
  unifies tasks like single-image-to-3D generation, multi-view stereo, and
  camera-controlled video generation. Our approach demonstrates competitive
  performance across multiple benchmarks, providing a scalable solution for
  3D-consistent video and image generation with a single pretrained model.
paper_authors._id: 674f26532836633b518dd279
paper_authors.name: Qihang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:05:47.586Z
paper_authors._id: 64da979f3a7ab21ea7d499c3
paper_authors.avatarUrl: /avatars/84fd959092e5a5d3b29720f6f7464b25.svg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: QihangZhang
paper_authors._id: 674f26532836633b518dd27a
paper_authors.name: Shuangfei Zhai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:23:17.582Z
paper_authors._id: 675bf04240726aac79ae8e7b
paper_authors.avatarUrl: /avatars/8ae95255dfce91a6f5e76f14eae9e9e5.svg
paper_authors.fullname: Shuangfei Zhai
paper_authors.type: user
paper_authors.user: shuangfei
paper_authors._id: 674f26532836633b518dd27b
paper_authors.name: Miguel Angel Bautista
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:05:28.650Z
paper_authors._id: 65dd58aa29b761fff4bd3a4c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/SD7Vh0i5HwJJsQxXpSTFa.jpeg
paper_authors.fullname: Miguel Angel Bautista
paper_authors.type: user
paper_authors.user: FlippyCode
paper_authors._id: 674f26532836633b518dd27c
paper_authors.name: Kevin Miao
paper_authors._id: 674f26532836633b518dd27d
paper_authors.name: Alexander Toshev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:04:58.423Z
paper_authors._id: 6541636c4f939214d3a0c742
paper_authors.avatarUrl: /avatars/c9507d6ec2b81681461837c8261703c7.svg
paper_authors.fullname: Alexander Toshev
paper_authors.type: user
paper_authors.user: toshev
paper_authors._id: 674f26532836633b518dd27e
paper_authors.name: Joshua Susskind
paper_authors._id: 674f26532836633b518dd27f
paper_authors.name: Jiatao Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:04:43.809Z
paper_authors._id: 6164e72d73996c363c52e66d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1634002684894-noauth.png
paper_authors.fullname: Jiatao Gu
paper_authors.type: user
paper_authors.user: thomagram
paper_title: World-consistent Video Diffusion with Explicit 3D Modeling
paper_summary: Recent advancements in diffusion models have set new benchmarks in image and
  video generation, enabling realistic visual synthesis across single- and
  multi-frame contexts. However, these models still struggle with efficiently and
  explicitly generating 3D-consistent content. To address this, we propose
  World-consistent Video Diffusion (WVD), a novel framework that incorporates
  explicit 3D supervision using XYZ images, which encode global 3D coordinates
  for each image pixel. More specifically, we train a diffusion transformer to
  learn the joint distribution of RGB and XYZ frames. This approach supports
  multi-task adaptability via a flexible inpainting strategy. For example, WVD
  can estimate XYZ frames from ground-truth RGB or generate novel RGB frames
  using XYZ projections along a specified camera trajectory. In doing so, WVD
  unifies tasks like single-image-to-3D generation, multi-view stereo, and
  camera-controlled video generation. Our approach demonstrates competitive
  performance across multiple benchmarks, providing a scalable solution for
  3D-consistent video and image generation with a single pretrained model.
paper_ai_summary: A new framework, World-consistent Video Diffusion (WVD), uses XYZ images for 3D supervision to improve the generation of 3D-consistent content in videos and images.
paper_ai_keywords: diffusion models World-consistent Video Diffusion (WVD) XYZ images global 3D coordinates diffusion transformer joint distribution RGB flexible inpainting strategy single-image-to-3D generation multi-view stereo camera-controlled video generation