SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse
  Viewpoints
Recent advancements in video diffusion models have shown exceptional
abilities in simulating real-world dynamics and maintaining 3D consistency.
This progress inspires us to investigate the potential of these models to
ensure dynamic consistency across various viewpoints, a highly desirable
feature for applications such as virtual filming. Unlike existing methods
focused on multi-view generation of single objects for 4D reconstruction, our
interest lies in generating open-world videos from arbitrary viewpoints,
incorporating 6 DoF camera poses. To achieve this, we propose a plug-and-play
module that enhances a pre-trained text-to-video model for multi-camera video
generation, ensuring consistent content across different viewpoints.
Specifically, we introduce a multi-view synchronization module to maintain
appearance and geometry consistency across these viewpoints. Given the scarcity
of high-quality training data, we design a hybrid training scheme that
leverages multi-camera images and monocular videos to supplement Unreal
Engine-rendered multi-camera videos. Furthermore, our method enables intriguing
extensions, such as re-rendering a video from novel viewpoints. We also release
a multi-view synchronized video dataset, named SynCamVideo-Dataset. Project
page: https://jianhongbai.github.io/SynCamMaster/.
67591885536995a5bc59a89e
Jianhong Bai
claimed_verified
2024-12-11T09:19:50.897Z
6530bf50f145530101ec03a2
/avatars/c61c00c314cf202b64968e51e855694d.svg
Jianhong Bai
user
jianhongbai
67591885536995a5bc59a89f
Menghan Xia
admin_assigned
2024-12-12T08:24:51.075Z
63401c89f81b9d101361f712
https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
Richard
user
menghanxia
67591885536995a5bc59a8a0
Xintao Wang
admin_assigned
2024-12-12T08:24:43.182Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
67591885536995a5bc59a8a1
Ziyang Yuan
admin_assigned
2024-12-12T08:24:57.808Z
66c15837b186e4f6a0dac80c
/avatars/a3b75d6945f1608e64a2fcff887a5024.svg
Ziyang Yuan
user
ziyangy
67591885536995a5bc59a8a2
Xiao Fu
claimed_verified
2024-12-11T09:54:19.786Z
63aef2cafcca84593e6682db
https://cdn-avatars.huggingface.co/v1/production/uploads/1672409763337-noauth.jpeg
Xiao Fu
user
lemonaddie
67591885536995a5bc59a8a3
Zuozhu Liu
admin_assigned
2024-12-12T08:25:03.968Z
6458b8d0990172cd1d703715
/avatars/55f0695e3cb9933c3903fde5a8f740d5.svg
Zuozhu Liu
user
Zuozhu
67591885536995a5bc59a8a4
Haoji Hu
admin_assigned
2024-12-12T08:25:09.626Z
66c46129d67297a9b93e03c5
/avatars/cffd8b07fa3655e240efc8e81f99d97d.svg
Haoji Hu
user
garland1979
67591885536995a5bc59a8a5
Pengfei Wan
claimed_verified
2025-03-17T14:43:09.243Z
662f93942510ef5735d7ad00
https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
magicwpf
user
magicwpf
67591885536995a5bc59a8a6
Di Zhang
admin_assigned
2024-12-12T08:24:11.595Z
64bce15bafd1e46c5504ad38
https://cdn-avatars.huggingface.co/v1/production/uploads/64bce15bafd1e46c5504ad38/vkEjiu-mIagKlrXzDH75o.png
Di Zhang
user
di-zhang-fdu
SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse
  Viewpoints
Recent advancements in video diffusion models have shown exceptional
abilities in simulating real-world dynamics and maintaining 3D consistency.
This progress inspires us to investigate the potential of these models to
ensure dynamic consistency across various viewpoints, a highly desirable
feature for applications such as virtual filming. Unlike existing methods
focused on multi-view generation of single objects for 4D reconstruction, our
interest lies in generating open-world videos from arbitrary viewpoints,
incorporating 6 DoF camera poses. To achieve this, we propose a plug-and-play
module that enhances a pre-trained text-to-video model for multi-camera video
generation, ensuring consistent content across different viewpoints.
Specifically, we introduce a multi-view synchronization module to maintain
appearance and geometry consistency across these viewpoints. Given the scarcity
of high-quality training data, we design a hybrid training scheme that
leverages multi-camera images and monocular videos to supplement Unreal
Engine-rendered multi-camera videos. Furthermore, our method enables intriguing
extensions, such as re-rendering a video from novel viewpoints. We also release
a multi-view synchronized video dataset, named SynCamVideo-Dataset. Project
page: https://jianhongbai.github.io/SynCamMaster/.
A plug-and-play module enhances text-to-video models for multi-camera video generation, incorporating 6 DoF camera poses and ensuring dynamic consistency across viewpoints using a hybrid training scheme and multi-view synchronization.
video diffusion models 3D consistency virtual filming multi-view generation 6 DoF camera poses plug-and-play module multi-view synchronization module appearance consistency geometry consistency multi-camera images monocular videos Unreal Engine-rendered multi-camera videos SynCamVideo-Dataset