title: Vivid-ZOO: Multi-View Video Generation with Diffusion Model
summary: While diffusion models have shown impressive performance in 2D image/video
  generation, diffusion-based Text-to-Multi-view-Video (T2MVid) generation
  remains underexplored. The new challenges posed by T2MVid generation lie in the
  lack of massive captioned multi-view videos and the complexity of modeling such
  multi-dimensional distribution. To this end, we propose a novel diffusion-based
  pipeline that generates high-quality multi-view videos centered around a
  dynamic 3D object from text. Specifically, we factor the T2MVid problem into
  viewpoint-space and time components. Such factorization allows us to combine
  and reuse layers of advanced pre-trained multi-view image and 2D video
  diffusion models to ensure multi-view consistency as well as temporal coherence
  for the generated multi-view videos, largely reducing the training cost. We
  further introduce alignment modules to align the latent spaces of layers from
  the pre-trained multi-view and the 2D video diffusion models, addressing the
  reused layers' incompatibility that arises from the domain gap between 2D and
  multi-view data. In support of this and future research, we further contribute
  a captioned multi-view video dataset. Experimental results demonstrate that our
  method generates high-quality multi-view videos, exhibiting vivid motions,
  temporal coherence, and multi-view consistency, given a variety of text
  prompts.
paper_authors._id: 666ddc3b2c11d1cf29d2aa91
paper_authors.name: Bing Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-15T21:10:03.603Z
paper_authors._id: 666ddb45c0f3d5afc27e85ba
paper_authors.avatarUrl: /avatars/dce98fc77bd8cf9e348f2d91bc3c0225.svg
paper_authors.fullname: Bing Li
paper_authors.type: user
paper_authors.user: bing-li-ai
paper_authors._id: 666ddc3b2c11d1cf29d2aa92
paper_authors.name: Cheng Zheng
paper_authors._id: 666ddc3b2c11d1cf29d2aa93
paper_authors.name: Wenxuan Zhu
paper_authors._id: 666ddc3b2c11d1cf29d2aa94
paper_authors.name: Jinjie Mai
paper_authors._id: 666ddc3b2c11d1cf29d2aa95
paper_authors.name: Biao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:21:35.943Z
paper_authors._id: 63f89d53b0ae1748524c1b8c
paper_authors.avatarUrl: /avatars/850ffb2724869dbb0deba50f6c71cb95.svg
paper_authors.fullname: Biao Zhang
paper_authors.type: user
paper_authors.user: Zbalpha
paper_authors._id: 666ddc3b2c11d1cf29d2aa96
paper_authors.name: Peter Wonka
paper_authors._id: 666ddc3b2c11d1cf29d2aa97
paper_authors.name: Bernard Ghanem
paper_title: Vivid-ZOO: Multi-View Video Generation with Diffusion Model
paper_summary: While diffusion models have shown impressive performance in 2D image/video
  generation, diffusion-based Text-to-Multi-view-Video (T2MVid) generation
  remains underexplored. The new challenges posed by T2MVid generation lie in the
  lack of massive captioned multi-view videos and the complexity of modeling such
  multi-dimensional distribution. To this end, we propose a novel diffusion-based
  pipeline that generates high-quality multi-view videos centered around a
  dynamic 3D object from text. Specifically, we factor the T2MVid problem into
  viewpoint-space and time components. Such factorization allows us to combine
  and reuse layers of advanced pre-trained multi-view image and 2D video
  diffusion models to ensure multi-view consistency as well as temporal coherence
  for the generated multi-view videos, largely reducing the training cost. We
  further introduce alignment modules to align the latent spaces of layers from
  the pre-trained multi-view and the 2D video diffusion models, addressing the
  reused layers' incompatibility that arises from the domain gap between 2D and
  multi-view data. In support of this and future research, we further contribute
  a captioned multi-view video dataset. Experimental results demonstrate that our
  method generates high-quality multi-view videos, exhibiting vivid motions,
  temporal coherence, and multi-view consistency, given a variety of text
  prompts.
paper_ai_keywords: diffusion-based Text-to-Multi-view-Video (T2MVid) multi-view videos viewpoint-space time components pre-trained multi-view image 2D video diffusion models latent spaces domain gap captioned multi-view video dataset