title: SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix
summary: Video generation models have demonstrated great capabilities of producing
  impressive monocular videos, however, the generation of 3D stereoscopic video
  remains under-explored. We propose a pose-free and training-free approach for
  generating 3D stereoscopic videos using an off-the-shelf monocular video
  generation model. Our method warps a generated monocular video into camera
  views on stereoscopic baseline using estimated video depth, and employs a novel
  frame matrix video inpainting framework. The framework leverages the video
  generation model to inpaint frames observed from different timestamps and
  views. This effective approach generates consistent and semantically coherent
  stereoscopic videos without scene optimization or model fine-tuning. Moreover,
  we develop a disocclusion boundary re-injection scheme that further improves
  the quality of video inpainting by alleviating the negative effects propagated
  from disoccluded areas in the latent space. We validate the efficacy of our
  proposed method by conducting experiments on videos from various generative
  models, including Sora [4 ], Lumiere [2], WALT [8 ], and Zeroscope [ 42]. The
  experiments demonstrate that our method has a significant improvement over
  previous methods. The code will be released at
  https://daipengwa.github.io/SVG_ProjectPage.
paper_authors._id: 66837a5b2d89090894bf8fee
paper_authors.name: Peng Dai
paper_authors._id: 66837a5b2d89090894bf8fef
paper_authors.name: Feitong Tan
paper_authors._id: 66837a5b2d89090894bf8ff0
paper_authors.name: Qiangeng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:16:23.602Z
paper_authors._id: 64c1a214c84cd12dd79c1530
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/CSK0AyQhXs3Izrnt9s6_l.png
paper_authors.fullname: qiangeng xu
paper_authors.type: user
paper_authors.user: xharlie
paper_authors._id: 66837a5b2d89090894bf8ff1
paper_authors.name: David Futschik
paper_authors._id: 66837a5b2d89090894bf8ff2
paper_authors.name: Ruofei Du
paper_authors._id: 66837a5b2d89090894bf8ff3
paper_authors.name: Sean Fanello
paper_authors._id: 66837a5b2d89090894bf8ff4
paper_authors.name: Xiaojuan Qi
paper_authors._id: 66837a5b2d89090894bf8ff5
paper_authors.name: Yinda Zhang
paper_title: SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix
paper_summary: Video generation models have demonstrated great capabilities of producing
  impressive monocular videos, however, the generation of 3D stereoscopic video
  remains under-explored. We propose a pose-free and training-free approach for
  generating 3D stereoscopic videos using an off-the-shelf monocular video
  generation model. Our method warps a generated monocular video into camera
  views on stereoscopic baseline using estimated video depth, and employs a novel
  frame matrix video inpainting framework. The framework leverages the video
  generation model to inpaint frames observed from different timestamps and
  views. This effective approach generates consistent and semantically coherent
  stereoscopic videos without scene optimization or model fine-tuning. Moreover,
  we develop a disocclusion boundary re-injection scheme that further improves
  the quality of video inpainting by alleviating the negative effects propagated
  from disoccluded areas in the latent space. We validate the efficacy of our
  proposed method by conducting experiments on videos from various generative
  models, including Sora [4 ], Lumiere [2], WALT [8 ], and Zeroscope [ 42]. The
  experiments demonstrate that our method has a significant improvement over
  previous methods. The code will be released at
  https://daipengwa.github.io/SVG_ProjectPage.
paper_ai_keywords: 3D stereoscopic video pose-free training-free off-the-shelf monocular video generation model camera views stereoscopic baseline video depth frame matrix video inpainting framework video inpainting semantic coherence disocclusion boundary re-injection scheme latent space disoccluded areas Sora Lumiere WALT Zeroscope