title: SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse
    Viewpoints
summary: Recent advancements in video diffusion models have shown exceptional
  abilities in simulating real-world dynamics and maintaining 3D consistency.
  This progress inspires us to investigate the potential of these models to
  ensure dynamic consistency across various viewpoints, a highly desirable
  feature for applications such as virtual filming. Unlike existing methods
  focused on multi-view generation of single objects for 4D reconstruction, our
  interest lies in generating open-world videos from arbitrary viewpoints,
  incorporating 6 DoF camera poses. To achieve this, we propose a plug-and-play
  module that enhances a pre-trained text-to-video model for multi-camera video
  generation, ensuring consistent content across different viewpoints.
  Specifically, we introduce a multi-view synchronization module to maintain
  appearance and geometry consistency across these viewpoints. Given the scarcity
  of high-quality training data, we design a hybrid training scheme that
  leverages multi-camera images and monocular videos to supplement Unreal
  Engine-rendered multi-camera videos. Furthermore, our method enables intriguing
  extensions, such as re-rendering a video from novel viewpoints. We also release
  a multi-view synchronized video dataset, named SynCamVideo-Dataset. Project
  page: https://jianhongbai.github.io/SynCamMaster/.
paper_authors._id: 67591885536995a5bc59a89e
paper_authors.name: Jianhong Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:19:50.897Z
paper_authors._id: 6530bf50f145530101ec03a2
paper_authors.avatarUrl: /avatars/c61c00c314cf202b64968e51e855694d.svg
paper_authors.fullname: Jianhong Bai
paper_authors.type: user
paper_authors.user: jianhongbai
paper_authors._id: 67591885536995a5bc59a89f
paper_authors.name: Menghan Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T08:24:51.075Z
paper_authors._id: 63401c89f81b9d101361f712
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
paper_authors.fullname: Richard
paper_authors.type: user
paper_authors.user: menghanxia
paper_authors._id: 67591885536995a5bc59a8a0
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T08:24:43.182Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 67591885536995a5bc59a8a1
paper_authors.name: Ziyang Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T08:24:57.808Z
paper_authors._id: 66c15837b186e4f6a0dac80c
paper_authors.avatarUrl: /avatars/a3b75d6945f1608e64a2fcff887a5024.svg
paper_authors.fullname: Ziyang Yuan
paper_authors.type: user
paper_authors.user: ziyangy
paper_authors._id: 67591885536995a5bc59a8a2
paper_authors.name: Xiao Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:54:19.786Z
paper_authors._id: 63aef2cafcca84593e6682db
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1672409763337-noauth.jpeg
paper_authors.fullname: Xiao Fu
paper_authors.type: user
paper_authors.user: lemonaddie
paper_authors._id: 67591885536995a5bc59a8a3
paper_authors.name: Zuozhu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T08:25:03.968Z
paper_authors._id: 6458b8d0990172cd1d703715
paper_authors.avatarUrl: /avatars/55f0695e3cb9933c3903fde5a8f740d5.svg
paper_authors.fullname: Zuozhu Liu
paper_authors.type: user
paper_authors.user: Zuozhu
paper_authors._id: 67591885536995a5bc59a8a4
paper_authors.name: Haoji Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T08:25:09.626Z
paper_authors._id: 66c46129d67297a9b93e03c5
paper_authors.avatarUrl: /avatars/cffd8b07fa3655e240efc8e81f99d97d.svg
paper_authors.fullname: Haoji Hu
paper_authors.type: user
paper_authors.user: garland1979
paper_authors._id: 67591885536995a5bc59a8a5
paper_authors.name: Pengfei Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-17T14:43:09.243Z
paper_authors._id: 662f93942510ef5735d7ad00
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
paper_authors.fullname: magicwpf
paper_authors.type: user
paper_authors.user: magicwpf
paper_authors._id: 67591885536995a5bc59a8a6
paper_authors.name: Di Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T08:24:11.595Z
paper_authors._id: 64bce15bafd1e46c5504ad38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64bce15bafd1e46c5504ad38/vkEjiu-mIagKlrXzDH75o.png
paper_authors.fullname: Di Zhang
paper_authors.type: user
paper_authors.user: di-zhang-fdu
paper_title: SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse
    Viewpoints
paper_summary: Recent advancements in video diffusion models have shown exceptional
  abilities in simulating real-world dynamics and maintaining 3D consistency.
  This progress inspires us to investigate the potential of these models to
  ensure dynamic consistency across various viewpoints, a highly desirable
  feature for applications such as virtual filming. Unlike existing methods
  focused on multi-view generation of single objects for 4D reconstruction, our
  interest lies in generating open-world videos from arbitrary viewpoints,
  incorporating 6 DoF camera poses. To achieve this, we propose a plug-and-play
  module that enhances a pre-trained text-to-video model for multi-camera video
  generation, ensuring consistent content across different viewpoints.
  Specifically, we introduce a multi-view synchronization module to maintain
  appearance and geometry consistency across these viewpoints. Given the scarcity
  of high-quality training data, we design a hybrid training scheme that
  leverages multi-camera images and monocular videos to supplement Unreal
  Engine-rendered multi-camera videos. Furthermore, our method enables intriguing
  extensions, such as re-rendering a video from novel viewpoints. We also release
  a multi-view synchronized video dataset, named SynCamVideo-Dataset. Project
  page: https://jianhongbai.github.io/SynCamMaster/.
paper_ai_summary: A plug-and-play module enhances text-to-video models for multi-camera video generation, incorporating 6 DoF camera poses and ensuring dynamic consistency across viewpoints using a hybrid training scheme and multi-view synchronization.
paper_ai_keywords: video diffusion models 3D consistency virtual filming multi-view generation 6 DoF camera poses plug-and-play module multi-view synchronization module appearance consistency geometry consistency multi-camera images monocular videos Unreal Engine-rendered multi-camera videos SynCamVideo-Dataset