title: VideoBooth: Diffusion-based Video Generation with Image Prompts
summary: Text-driven video generation witnesses rapid progress. However, merely using
  text prompts is not enough to depict the desired subject appearance that
  accurately aligns with users' intents, especially for customized content
  creation. In this paper, we study the task of video generation with image
  prompts, which provide more accurate and direct content control beyond the text
  prompts. Specifically, we propose a feed-forward framework VideoBooth, with two
  dedicated designs: 1) We propose to embed image prompts in a coarse-to-fine
  manner. Coarse visual embeddings from image encoder provide high-level
  encodings of image prompts, while fine visual embeddings from the proposed
  attention injection module provide multi-scale and detailed encoding of image
  prompts. These two complementary embeddings can faithfully capture the desired
  appearance. 2) In the attention injection module at fine level, multi-scale
  image prompts are fed into different cross-frame attention layers as additional
  keys and values. This extra spatial information refines the details in the
  first frame and then it is propagated to the remaining frames, which maintains
  temporal consistency. Extensive experiments demonstrate that VideoBooth
  achieves state-of-the-art performance in generating customized high-quality
  videos with subjects specified in image prompts. Notably, VideoBooth is a
  generalizable framework where a single model works for a wide range of image
  prompts with feed-forward pass.
paper_authors._id: 656d39417c934a7b3c307513
paper_authors.name: Yuming Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T09:36:08.537Z
paper_authors._id: 629c95b7a5d6f5fe10e6ed45
paper_authors.avatarUrl: /avatars/c9a5f0e7fae3a079d26db266e9ff90a3.svg
paper_authors.fullname: Yuming Jiang
paper_authors.type: user
paper_authors.user: yumingj
paper_authors._id: 656d39417c934a7b3c307514
paper_authors.name: Tianxing Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T09:36:20.250Z
paper_authors._id: 63185a8973e9e4aa51dbc0ce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676359176854-63185a8973e9e4aa51dbc0ce.jpeg
paper_authors.fullname: Tianxing Wu
paper_authors.type: user
paper_authors.user: TianxingWu
paper_authors._id: 656d39417c934a7b3c307515
paper_authors.name: Shuai Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T09:37:15.117Z
paper_authors._id: 62a54d0410334c1d024e2f59
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664764278226-62a54d0410334c1d024e2f59.jpeg
paper_authors.fullname: Shuai Yang
paper_authors.type: user
paper_authors.user: PKUWilliamYang
paper_authors._id: 656d39417c934a7b3c307516
paper_authors.name: Chenyang Si
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T09:37:24.369Z
paper_authors._id: 635f8ed47c05eb9f59963d3a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635f8ed47c05eb9f59963d3a/uQf4p9N9pSaFy87Wg9v4k.jpeg
paper_authors.fullname: ChenyangSi
paper_authors.type: user
paper_authors.user: ChenyangSi
paper_authors._id: 656d39417c934a7b3c307517
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T09:37:33.588Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 656d39417c934a7b3c307518
paper_authors.name: Yu Qiao
paper_authors._id: 656d39417c934a7b3c307519
paper_authors.name: Chen Change Loy
paper_authors._id: 656d39417c934a7b3c30751a
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T09:38:44.028Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: VideoBooth: Diffusion-based Video Generation with Image Prompts
paper_summary: Text-driven video generation witnesses rapid progress. However, merely using
  text prompts is not enough to depict the desired subject appearance that
  accurately aligns with users' intents, especially for customized content
  creation. In this paper, we study the task of video generation with image
  prompts, which provide more accurate and direct content control beyond the text
  prompts. Specifically, we propose a feed-forward framework VideoBooth, with two
  dedicated designs: 1) We propose to embed image prompts in a coarse-to-fine
  manner. Coarse visual embeddings from image encoder provide high-level
  encodings of image prompts, while fine visual embeddings from the proposed
  attention injection module provide multi-scale and detailed encoding of image
  prompts. These two complementary embeddings can faithfully capture the desired
  appearance. 2) In the attention injection module at fine level, multi-scale
  image prompts are fed into different cross-frame attention layers as additional
  keys and values. This extra spatial information refines the details in the
  first frame and then it is propagated to the remaining frames, which maintains
  temporal consistency. Extensive experiments demonstrate that VideoBooth
  achieves state-of-the-art performance in generating customized high-quality
  videos with subjects specified in image prompts. Notably, VideoBooth is a
  generalizable framework where a single model works for a wide range of image
  prompts with feed-forward pass.
paper_ai_summary: VideoBooth generates high-quality customized videos using image prompts through a coarse-to-fine embedding and attention injection module, achieving state-of-the-art results in video content creation.
paper_ai_keywords: feed-forward framework VideoBooth coarse-to-fine embedding image encoder fine visual embeddings attention injection module cross-frame attention layers multi-scale image prompts temporal consistency