VideoBooth: Diffusion-based Video Generation with Image Prompts
Text-driven video generation witnesses rapid progress. However, merely using
text prompts is not enough to depict the desired subject appearance that
accurately aligns with users' intents, especially for customized content
creation. In this paper, we study the task of video generation with image
prompts, which provide more accurate and direct content control beyond the text
prompts. Specifically, we propose a feed-forward framework VideoBooth, with two
dedicated designs: 1) We propose to embed image prompts in a coarse-to-fine
manner. Coarse visual embeddings from image encoder provide high-level
encodings of image prompts, while fine visual embeddings from the proposed
attention injection module provide multi-scale and detailed encoding of image
prompts. These two complementary embeddings can faithfully capture the desired
appearance. 2) In the attention injection module at fine level, multi-scale
image prompts are fed into different cross-frame attention layers as additional
keys and values. This extra spatial information refines the details in the
first frame and then it is propagated to the remaining frames, which maintains
temporal consistency. Extensive experiments demonstrate that VideoBooth
achieves state-of-the-art performance in generating customized high-quality
videos with subjects specified in image prompts. Notably, VideoBooth is a
generalizable framework where a single model works for a wide range of image
prompts with feed-forward pass.
656d39417c934a7b3c307513
Yuming Jiang
admin_assigned
2023-12-04T09:36:08.537Z
629c95b7a5d6f5fe10e6ed45
/avatars/c9a5f0e7fae3a079d26db266e9ff90a3.svg
Yuming Jiang
user
yumingj
656d39417c934a7b3c307514
Tianxing Wu
admin_assigned
2023-12-04T09:36:20.250Z
63185a8973e9e4aa51dbc0ce
https://cdn-avatars.huggingface.co/v1/production/uploads/1676359176854-63185a8973e9e4aa51dbc0ce.jpeg
Tianxing Wu
user
TianxingWu
656d39417c934a7b3c307515
Shuai Yang
admin_assigned
2023-12-04T09:37:15.117Z
62a54d0410334c1d024e2f59
https://cdn-avatars.huggingface.co/v1/production/uploads/1664764278226-62a54d0410334c1d024e2f59.jpeg
Shuai Yang
user
PKUWilliamYang
656d39417c934a7b3c307516
Chenyang Si
admin_assigned
2023-12-04T09:37:24.369Z
635f8ed47c05eb9f59963d3a
https://cdn-avatars.huggingface.co/v1/production/uploads/635f8ed47c05eb9f59963d3a/uQf4p9N9pSaFy87Wg9v4k.jpeg
ChenyangSi
user
ChenyangSi
656d39417c934a7b3c307517
Dahua Lin
admin_assigned
2023-12-04T09:37:33.588Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
656d39417c934a7b3c307518
Yu Qiao
656d39417c934a7b3c307519
Chen Change Loy
656d39417c934a7b3c30751a
Ziwei Liu
admin_assigned
2023-12-04T09:38:44.028Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
VideoBooth: Diffusion-based Video Generation with Image Prompts
Text-driven video generation witnesses rapid progress. However, merely using
text prompts is not enough to depict the desired subject appearance that
accurately aligns with users' intents, especially for customized content
creation. In this paper, we study the task of video generation with image
prompts, which provide more accurate and direct content control beyond the text
prompts. Specifically, we propose a feed-forward framework VideoBooth, with two
dedicated designs: 1) We propose to embed image prompts in a coarse-to-fine
manner. Coarse visual embeddings from image encoder provide high-level
encodings of image prompts, while fine visual embeddings from the proposed
attention injection module provide multi-scale and detailed encoding of image
prompts. These two complementary embeddings can faithfully capture the desired
appearance. 2) In the attention injection module at fine level, multi-scale
image prompts are fed into different cross-frame attention layers as additional
keys and values. This extra spatial information refines the details in the
first frame and then it is propagated to the remaining frames, which maintains
temporal consistency. Extensive experiments demonstrate that VideoBooth
achieves state-of-the-art performance in generating customized high-quality
videos with subjects specified in image prompts. Notably, VideoBooth is a
generalizable framework where a single model works for a wide range of image
prompts with feed-forward pass.
VideoBooth generates high-quality customized videos using image prompts through a coarse-to-fine embedding and attention injection module, achieving state-of-the-art results in video content creation.
feed-forward framework VideoBooth coarse-to-fine embedding image encoder fine visual embeddings attention injection module cross-frame attention layers multi-scale image prompts temporal consistency