title: CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects
summary: Customized text-to-video generation aims to generate high-quality videos
  guided by text prompts and subject references. Current approaches designed for
  single subjects suffer from tackling multiple subjects, which is a more
  challenging and practical scenario. In this work, we aim to promote
  multi-subject guided text-to-video customization. We propose CustomVideo, a
  novel framework that can generate identity-preserving videos with the guidance
  of multiple subjects. To be specific, firstly, we encourage the co-occurrence
  of multiple subjects via composing them in a single image. Further, upon a
  basic text-to-video diffusion model, we design a simple yet effective attention
  control strategy to disentangle different subjects in the latent space of
  diffusion model. Moreover, to help the model focus on the specific object area,
  we segment the object from given reference images and provide a corresponding
  object mask for attention learning. Also, we collect a multi-subject
  text-to-video generation dataset as a comprehensive benchmark, with 69
  individual subjects and 57 meaningful pairs. Extensive qualitative,
  quantitative, and user study results demonstrate the superiority of our method,
  compared with the previous state-of-the-art approaches.
paper_authors._id: 65a9e778d63812c33ed6218c
paper_authors.name: Zhao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-22T10:31:07.758Z
paper_authors._id: 6432798c07bad11484a70b85
paper_authors.avatarUrl: /avatars/98a546fc1ea5dae96c70aab2bb9e3e49.svg
paper_authors.fullname: Zhao Wang
paper_authors.type: user
paper_authors.user: kyfafyd
paper_authors._id: 65a9e778d63812c33ed6218d
paper_authors.name: Aoxue Li
paper_authors._id: 65a9e778d63812c33ed6218e
paper_authors.name: Enze Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T13:38:34.713Z
paper_authors._id: 616961bdd3f656f79ad18ec1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/616961bdd3f656f79ad18ec1/NYpxas378lDevRuUDepJJ.jpeg
paper_authors.fullname: xieenze
paper_authors.type: user
paper_authors.user: xieenze
paper_authors._id: 65a9e778d63812c33ed6218f
paper_authors.name: Lingting Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T13:38:42.682Z
paper_authors._id: 6423f5e6774cc34079730f31
paper_authors.avatarUrl: /avatars/7ebeb1f623c86b1a676c95bb67572f8b.svg
paper_authors.fullname: Lingting Zhu
paper_authors.type: user
paper_authors.user: ltzhu
paper_authors._id: 65a9e778d63812c33ed62190
paper_authors.name: Yong Guo
paper_authors._id: 65a9e778d63812c33ed62191
paper_authors.name: Qi Dou
paper_authors._id: 65a9e778d63812c33ed62192
paper_authors.name: Zhenguo Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T13:39:18.360Z
paper_authors._id: 639944bef7c4648254d124f0
paper_authors.avatarUrl: /avatars/c5cbb87cd51cc1341844f67d42c55151.svg
paper_authors.fullname: lzg
paper_authors.type: user
paper_authors.user: lizhenguo
paper_title: CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects
paper_summary: Customized text-to-video generation aims to generate high-quality videos
  guided by text prompts and subject references. Current approaches designed for
  single subjects suffer from tackling multiple subjects, which is a more
  challenging and practical scenario. In this work, we aim to promote
  multi-subject guided text-to-video customization. We propose CustomVideo, a
  novel framework that can generate identity-preserving videos with the guidance
  of multiple subjects. To be specific, firstly, we encourage the co-occurrence
  of multiple subjects via composing them in a single image. Further, upon a
  basic text-to-video diffusion model, we design a simple yet effective attention
  control strategy to disentangle different subjects in the latent space of
  diffusion model. Moreover, to help the model focus on the specific object area,
  we segment the object from given reference images and provide a corresponding
  object mask for attention learning. Also, we collect a multi-subject
  text-to-video generation dataset as a comprehensive benchmark, with 69
  individual subjects and 57 meaningful pairs. Extensive qualitative,
  quantitative, and user study results demonstrate the superiority of our method,
  compared with the previous state-of-the-art approaches.
paper_ai_summary: A novel framework CustomVideo enhances multi-subject text-to-video generation by using an attention control strategy in latent space and object segmentation masks to produce identity-preserving videos.
paper_ai_keywords: customvideo text-to-video diffusion model attention control strategy latent space object segmentation object mask multi-subject text-to-video generation dataset