title: Collaborative Video Diffusion: Consistent Multi-video Generation with
    Camera Control
summary: Research on video generation has recently made tremendous progress, enabling
  high-quality videos to be generated from text prompts or images. Adding control
  to the video generation process is an important goal moving forward and recent
  approaches that condition video generation models on camera trajectories make
  strides towards it. Yet, it remains challenging to generate a video of the same
  scene from multiple different camera trajectories. Solutions to this
  multi-video generation problem could enable large-scale 3D scene generation
  with editable camera trajectories, among other applications. We introduce
  collaborative video diffusion (CVD) as an important step towards this vision.
  The CVD framework includes a novel cross-video synchronization module that
  promotes consistency between corresponding frames of the same video rendered
  from different camera poses using an epipolar attention mechanism. Trained on
  top of a state-of-the-art camera-control module for video generation, CVD
  generates multiple videos rendered from different camera trajectories with
  significantly better consistency than baselines, as shown in extensive
  experiments. Project page: https://collaborativevideodiffusion.github.io/.
paper_authors._id: 665549127473f3c25745ced4
paper_authors.name: Zhengfei Kuang
paper_authors._id: 665549127473f3c25745ced5
paper_authors.name: Shengqu Cai
paper_authors._id: 665549127473f3c25745ced6
paper_authors.name: Hao He
paper_authors._id: 665549127473f3c25745ced7
paper_authors.name: Yinghao Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:38:34.487Z
paper_authors._id: 6555da9adcf410fd0753569c
paper_authors.avatarUrl: /avatars/ac58a796bb54f334fdb475a4b75c4d27.svg
paper_authors.fullname: Yinghao Xu
paper_authors.type: user
paper_authors.user: justimyhxu
paper_authors._id: 665549127473f3c25745ced8
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:38:41.353Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 665549127473f3c25745ced9
paper_authors.name: Leonidas Guibas
paper_authors._id: 665549127473f3c25745ceda
paper_authors.name: Gordon Wetzstein
paper_title: Collaborative Video Diffusion: Consistent Multi-video Generation with
    Camera Control
paper_summary: Research on video generation has recently made tremendous progress, enabling
  high-quality videos to be generated from text prompts or images. Adding control
  to the video generation process is an important goal moving forward and recent
  approaches that condition video generation models on camera trajectories make
  strides towards it. Yet, it remains challenging to generate a video of the same
  scene from multiple different camera trajectories. Solutions to this
  multi-video generation problem could enable large-scale 3D scene generation
  with editable camera trajectories, among other applications. We introduce
  collaborative video diffusion (CVD) as an important step towards this vision.
  The CVD framework includes a novel cross-video synchronization module that
  promotes consistency between corresponding frames of the same video rendered
  from different camera poses using an epipolar attention mechanism. Trained on
  top of a state-of-the-art camera-control module for video generation, CVD
  generates multiple videos rendered from different camera trajectories with
  significantly better consistency than baselines, as shown in extensive
  experiments. Project page: https://collaborativevideodiffusion.github.io/.
paper_ai_keywords: collaborative video diffusion (CVD) cross-video synchronization module epipolar attention mechanism camera-control module video generation camera trajectories multi-video generation 3D scene generation frame consistency