title: Control-A-Video: Controllable Text-to-Video Generation with Diffusion
    Models
summary: This paper presents a controllable text-to-video (T2V) diffusion model, named
  Video-ControlNet, that generates videos conditioned on a sequence of control
  signals, such as edge or depth maps. Video-ControlNet is built on a pre-trained
  conditional text-to-image (T2I) diffusion model by incorporating a
  spatial-temporal self-attention mechanism and trainable temporal layers for
  efficient cross-frame modeling. A first-frame conditioning strategy is proposed
  to facilitate the model to generate videos transferred from the image domain as
  well as arbitrary-length videos in an auto-regressive manner. Moreover,
  Video-ControlNet employs a novel residual-based noise initialization strategy
  to introduce motion prior from an input video, producing more coherent videos.
  With the proposed architecture and strategies, Video-ControlNet can achieve
  resource-efficient convergence and generate superior quality and consistent
  videos with fine-grained control. Extensive experiments demonstrate its success
  in various video generative tasks such as video editing and video style
  transfer, outperforming previous methods in terms of consistency and quality.
  Project Page: https://controlavideo.github.io/
paper_authors._id: 646d6cf8fe25e5f8d9760975
paper_authors.name: Weifeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T07:03:49.036Z
paper_authors._id: 62a7fc03ac97233f161c2433
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a7fc03ac97233f161c2433/19wL1aCwyR-ik2WDXvns9.jpeg
paper_authors.fullname: weifeng
paper_authors.type: user
paper_authors.user: weifeng-chen
paper_authors._id: 646d6cf8fe25e5f8d9760976
paper_authors.name: Jie Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-22T07:37:46.613Z
paper_authors._id: 6381c5d63680a7cf34e08ca9
paper_authors.avatarUrl: /avatars/731467e2d80d0ae163c4a00a9e3ff9e5.svg
paper_authors.fullname: wujie10558@gmail.com
paper_authors.type: user
paper_authors.user: wujie10
paper_authors._id: 646d6cf8fe25e5f8d9760977
paper_authors.name: Pan Xie
paper_authors._id: 646d6cf8fe25e5f8d9760978
paper_authors.name: Hefeng Wu
paper_authors._id: 646d6cf8fe25e5f8d9760979
paper_authors.name: Jiashi Li
paper_authors._id: 646d6cf8fe25e5f8d976097a
paper_authors.name: Xin Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-09T18:39:02.320Z
paper_authors._id: 63089a78ff78e2aead8d10e7
paper_authors.avatarUrl: /avatars/f326fd08abee7e31599a78923be30003.svg
paper_authors.fullname: XinXia
paper_authors.type: user
paper_authors.user: XiaXin-Aloys
paper_authors._id: 646d6cf8fe25e5f8d976097b
paper_authors.name: Xuefeng Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T08:17:48.228Z
paper_authors._id: 646b7f71df2609a541c1ab9f
paper_authors.avatarUrl: /avatars/48b82e5fd9b06f41ff825507c36816cd.svg
paper_authors.fullname: Xuefeng Xiao
paper_authors.type: user
paper_authors.user: xiaoxuefeng
paper_authors._id: 646d6cf8fe25e5f8d976097c
paper_authors.name: Liang Lin
paper_title: Control-A-Video: Controllable Text-to-Video Generation with Diffusion
    Models
paper_summary: This paper presents a controllable text-to-video (T2V) diffusion model, named
  Video-ControlNet, that generates videos conditioned on a sequence of control
  signals, such as edge or depth maps. Video-ControlNet is built on a pre-trained
  conditional text-to-image (T2I) diffusion model by incorporating a
  spatial-temporal self-attention mechanism and trainable temporal layers for
  efficient cross-frame modeling. A first-frame conditioning strategy is proposed
  to facilitate the model to generate videos transferred from the image domain as
  well as arbitrary-length videos in an auto-regressive manner. Moreover,
  Video-ControlNet employs a novel residual-based noise initialization strategy
  to introduce motion prior from an input video, producing more coherent videos.
  With the proposed architecture and strategies, Video-ControlNet can achieve
  resource-efficient convergence and generate superior quality and consistent
  videos with fine-grained control. Extensive experiments demonstrate its success
  in various video generative tasks such as video editing and video style
  transfer, outperforming previous methods in terms of consistency and quality.
  Project Page: https://controlavideo.github.io/
paper_ai_summary: Video-ControlNet is a controllable T2V diffusion model that uses spatial-temporal self-attention and residual-based noise initialization to generate high-quality, consistent videos conditioned on control signals.
paper_ai_keywords: text-to-video Video-ControlNet control signals pre-trained conditional text-to-image spatial-temporal self-attention temporal layers first-frame conditioning residual-based noise initialization video editing video style transfer