CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects
Customized text-to-video generation aims to generate high-quality videos
guided by text prompts and subject references. Current approaches designed for
single subjects suffer from tackling multiple subjects, which is a more
challenging and practical scenario. In this work, we aim to promote
multi-subject guided text-to-video customization. We propose CustomVideo, a
novel framework that can generate identity-preserving videos with the guidance
of multiple subjects. To be specific, firstly, we encourage the co-occurrence
of multiple subjects via composing them in a single image. Further, upon a
basic text-to-video diffusion model, we design a simple yet effective attention
control strategy to disentangle different subjects in the latent space of
diffusion model. Moreover, to help the model focus on the specific object area,
we segment the object from given reference images and provide a corresponding
object mask for attention learning. Also, we collect a multi-subject
text-to-video generation dataset as a comprehensive benchmark, with 69
individual subjects and 57 meaningful pairs. Extensive qualitative,
quantitative, and user study results demonstrate the superiority of our method,
compared with the previous state-of-the-art approaches.
65a9e778d63812c33ed6218c
Zhao Wang
claimed_verified
2024-01-22T10:31:07.758Z
6432798c07bad11484a70b85
/avatars/98a546fc1ea5dae96c70aab2bb9e3e49.svg
Zhao Wang
user
kyfafyd
65a9e778d63812c33ed6218d
Aoxue Li
65a9e778d63812c33ed6218e
Enze Xie
admin_assigned
2024-01-19T13:38:34.713Z
616961bdd3f656f79ad18ec1
https://cdn-avatars.huggingface.co/v1/production/uploads/616961bdd3f656f79ad18ec1/NYpxas378lDevRuUDepJJ.jpeg
xieenze
user
xieenze
65a9e778d63812c33ed6218f
Lingting Zhu
admin_assigned
2024-01-19T13:38:42.682Z
6423f5e6774cc34079730f31
/avatars/7ebeb1f623c86b1a676c95bb67572f8b.svg
Lingting Zhu
user
ltzhu
65a9e778d63812c33ed62190
Yong Guo
65a9e778d63812c33ed62191
Qi Dou
65a9e778d63812c33ed62192
Zhenguo Li
admin_assigned
2024-01-19T13:39:18.360Z
639944bef7c4648254d124f0
/avatars/c5cbb87cd51cc1341844f67d42c55151.svg
lzg
user
lizhenguo
CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects
Customized text-to-video generation aims to generate high-quality videos
guided by text prompts and subject references. Current approaches designed for
single subjects suffer from tackling multiple subjects, which is a more
challenging and practical scenario. In this work, we aim to promote
multi-subject guided text-to-video customization. We propose CustomVideo, a
novel framework that can generate identity-preserving videos with the guidance
of multiple subjects. To be specific, firstly, we encourage the co-occurrence
of multiple subjects via composing them in a single image. Further, upon a
basic text-to-video diffusion model, we design a simple yet effective attention
control strategy to disentangle different subjects in the latent space of
diffusion model. Moreover, to help the model focus on the specific object area,
we segment the object from given reference images and provide a corresponding
object mask for attention learning. Also, we collect a multi-subject
text-to-video generation dataset as a comprehensive benchmark, with 69
individual subjects and 57 meaningful pairs. Extensive qualitative,
quantitative, and user study results demonstrate the superiority of our method,
compared with the previous state-of-the-art approaches.
A novel framework CustomVideo enhances multi-subject text-to-video generation by using an attention control strategy in latent space and object segmentation masks to produce identity-preserving videos.
customvideo text-to-video diffusion model attention control strategy latent space object segmentation object mask multi-subject text-to-video generation dataset