ControlVideo: Adding Conditional Control for One Shot Text-to-Video
  Editing
In this paper, we present ControlVideo, a novel method for text-driven video
editing. Leveraging the capabilities of text-to-image diffusion models and
ControlNet, ControlVideo aims to enhance the fidelity and temporal consistency
of videos that align with a given text while preserving the structure of the
source video. This is achieved by incorporating additional conditions such as
edge maps, fine-tuning the key-frame and temporal attention on the source
video-text pair with carefully designed strategies. An in-depth exploration of
ControlVideo's design is conducted to inform future research on one-shot tuning
video diffusion models. Quantitatively, ControlVideo outperforms a range of
competitive baselines in terms of faithfulness and consistency while still
aligning with the textual prompt. Additionally, it delivers videos with high
visual realism and fidelity w.r.t. the source content, demonstrating
flexibility in utilizing controls containing varying degrees of source video
information, and the potential for multiple control combinations. The project
page is available at
https://ml.cs.tsinghua.edu.cn/controlvideo/{https://ml.cs.tsinghua.edu.cn/controlvideo/}.
647416250da364bd0d009ce5
Min Zhao
647416250da364bd0d009ce6
Rongzhen Wang
647416250da364bd0d009ce7
Fan Bao
claimed_verified
2023-05-30T14:15:04.451Z
633131798ef21f47308ce49b
/avatars/1a3b5b032cbbfc1499ec96aba5260902.svg
fan bao
user
baofff
647416250da364bd0d009ce8
Chongxuan Li
647416250da364bd0d009ce9
Jun Zhu
ControlVideo: Adding Conditional Control for One Shot Text-to-Video
  Editing
In this paper, we present ControlVideo, a novel method for text-driven video
editing. Leveraging the capabilities of text-to-image diffusion models and
ControlNet, ControlVideo aims to enhance the fidelity and temporal consistency
of videos that align with a given text while preserving the structure of the
source video. This is achieved by incorporating additional conditions such as
edge maps, fine-tuning the key-frame and temporal attention on the source
video-text pair with carefully designed strategies. An in-depth exploration of
ControlVideo's design is conducted to inform future research on one-shot tuning
video diffusion models. Quantitatively, ControlVideo outperforms a range of
competitive baselines in terms of faithfulness and consistency while still
aligning with the textual prompt. Additionally, it delivers videos with high
visual realism and fidelity w.r.t. the source content, demonstrating
flexibility in utilizing controls containing varying degrees of source video
information, and the potential for multiple control combinations. The project
page is available at
https://ml.cs.tsinghua.edu.cn/controlvideo/{https://ml.cs.tsinghua.edu.cn/controlvideo/}.
ControlVideo enhances text-driven video editing by integrating text-to-image diffusion models and ControlNet, improving video fidelity and temporal consistency while maintaining source structure.
text-to-image diffusion models ControlNet video editing fidelity temporal consistency key-frame temporal attention one-shot tuning visual realism