title: ControlVideo: Adding Conditional Control for One Shot Text-to-Video
    Editing
summary: In this paper, we present ControlVideo, a novel method for text-driven video
  editing. Leveraging the capabilities of text-to-image diffusion models and
  ControlNet, ControlVideo aims to enhance the fidelity and temporal consistency
  of videos that align with a given text while preserving the structure of the
  source video. This is achieved by incorporating additional conditions such as
  edge maps, fine-tuning the key-frame and temporal attention on the source
  video-text pair with carefully designed strategies. An in-depth exploration of
  ControlVideo's design is conducted to inform future research on one-shot tuning
  video diffusion models. Quantitatively, ControlVideo outperforms a range of
  competitive baselines in terms of faithfulness and consistency while still
  aligning with the textual prompt. Additionally, it delivers videos with high
  visual realism and fidelity w.r.t. the source content, demonstrating
  flexibility in utilizing controls containing varying degrees of source video
  information, and the potential for multiple control combinations. The project
  page is available at
  https://ml.cs.tsinghua.edu.cn/controlvideo/{https://ml.cs.tsinghua.edu.cn/controlvideo/}.
paper_authors._id: 647416250da364bd0d009ce5
paper_authors.name: Min Zhao
paper_authors._id: 647416250da364bd0d009ce6
paper_authors.name: Rongzhen Wang
paper_authors._id: 647416250da364bd0d009ce7
paper_authors.name: Fan Bao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T14:15:04.451Z
paper_authors._id: 633131798ef21f47308ce49b
paper_authors.avatarUrl: /avatars/1a3b5b032cbbfc1499ec96aba5260902.svg
paper_authors.fullname: fan bao
paper_authors.type: user
paper_authors.user: baofff
paper_authors._id: 647416250da364bd0d009ce8
paper_authors.name: Chongxuan Li
paper_authors._id: 647416250da364bd0d009ce9
paper_authors.name: Jun Zhu
paper_title: ControlVideo: Adding Conditional Control for One Shot Text-to-Video
    Editing
paper_summary: In this paper, we present ControlVideo, a novel method for text-driven video
  editing. Leveraging the capabilities of text-to-image diffusion models and
  ControlNet, ControlVideo aims to enhance the fidelity and temporal consistency
  of videos that align with a given text while preserving the structure of the
  source video. This is achieved by incorporating additional conditions such as
  edge maps, fine-tuning the key-frame and temporal attention on the source
  video-text pair with carefully designed strategies. An in-depth exploration of
  ControlVideo's design is conducted to inform future research on one-shot tuning
  video diffusion models. Quantitatively, ControlVideo outperforms a range of
  competitive baselines in terms of faithfulness and consistency while still
  aligning with the textual prompt. Additionally, it delivers videos with high
  visual realism and fidelity w.r.t. the source content, demonstrating
  flexibility in utilizing controls containing varying degrees of source video
  information, and the potential for multiple control combinations. The project
  page is available at
  https://ml.cs.tsinghua.edu.cn/controlvideo/{https://ml.cs.tsinghua.edu.cn/controlvideo/}.
paper_ai_summary: ControlVideo enhances text-driven video editing by integrating text-to-image diffusion models and ControlNet, improving video fidelity and temporal consistency while maintaining source structure.
paper_ai_keywords: text-to-image diffusion models ControlNet video editing fidelity temporal consistency key-frame temporal attention one-shot tuning visual realism