title: Ingredients: Blending Custom Photos with Video Diffusion Transformers
summary: This paper presents a powerful framework to customize video creations by
  incorporating multiple specific identity (ID) photos, with video diffusion
  Transformers, referred to as Ingredients. Generally, our method
  consists of three primary modules: (i) a facial extractor that
  captures versatile and precise facial features for each human ID from both
  global and local perspectives; (ii) a multi-scale projector that maps
  face embeddings into the contextual space of image query in video diffusion
  transformers; (iii) an ID router that dynamically combines and
  allocates multiple ID embedding to the corresponding space-time regions.
  Leveraging a meticulously curated text-video dataset and a multi-stage training
  protocol, Ingredients demonstrates superior performance in turning
  custom photos into dynamic and personalized video content. Qualitative
  evaluations highlight the advantages of proposed method, positioning it as a
  significant advancement toward more effective generative video control tools in
  Transformer-based architecture, compared to existing methods. The data, code,
  and model weights are publicly available at:
  https://github.com/feizc/Ingredients.
paper_authors._id: 677c162919230f94bdcc45e2
paper_authors.name: Zhengcong Fei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T10:01:30.625Z
paper_authors._id: 617ba1820e4237bd1731b867
paper_authors.avatarUrl: /avatars/f9de06363e64bddd7dc977e96e85df8a.svg
paper_authors.fullname: zhengcong fei
paper_authors.type: user
paper_authors.user: onion
paper_authors._id: 677c162919230f94bdcc45e3
paper_authors.name: Debang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T10:01:40.288Z
paper_authors._id: 65dc3a850af7e21ba40e939f
paper_authors.avatarUrl: /avatars/e129c64617675edd05d4317d39604318.svg
paper_authors.fullname: Li
paper_authors.type: user
paper_authors.user: Debang
paper_authors._id: 677c162919230f94bdcc45e4
paper_authors.name: Di Qiu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T13:15:36.627Z
paper_authors._id: 65bef422fdb8d33cefeaccc3
paper_authors.avatarUrl: /avatars/d40b0d7dda21fa1a68c291d11bc357ec.svg
paper_authors.fullname: Qiu Di
paper_authors.type: user
paper_authors.user: diqiu7
paper_authors._id: 677c162919230f94bdcc45e5
paper_authors.name: Changqian Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T10:01:50.216Z
paper_authors._id: 6419e0b3ed725fef6444f53a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6419e0b3ed725fef6444f53a/3ZzIafBJa50-gAu1PASuG.png
paper_authors.fullname: Yu Changqian
paper_authors.type: user
paper_authors.user: Changqian
paper_authors._id: 677c162919230f94bdcc45e6
paper_authors.name: Mingyuan Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T10:02:01.415Z
paper_authors._id: 634672bfb7b4e71c7f45360f
paper_authors.avatarUrl: /avatars/4b646fc3e271be90b9ec619d42ce3e99.svg
paper_authors.fullname: Fan Mingyuan
paper_authors.type: user
paper_authors.user: MichaelFan
paper_title: Ingredients: Blending Custom Photos with Video Diffusion Transformers
paper_summary: This paper presents a powerful framework to customize video creations by
  incorporating multiple specific identity (ID) photos, with video diffusion
  Transformers, referred to as Ingredients. Generally, our method
  consists of three primary modules: (i) a facial extractor that
  captures versatile and precise facial features for each human ID from both
  global and local perspectives; (ii) a multi-scale projector that maps
  face embeddings into the contextual space of image query in video diffusion
  transformers; (iii) an ID router that dynamically combines and
  allocates multiple ID embedding to the corresponding space-time regions.
  Leveraging a meticulously curated text-video dataset and a multi-stage training
  protocol, Ingredients demonstrates superior performance in turning
  custom photos into dynamic and personalized video content. Qualitative
  evaluations highlight the advantages of proposed method, positioning it as a
  significant advancement toward more effective generative video control tools in
  Transformer-based architecture, compared to existing methods. The data, code,
  and model weights are publicly available at:
  https://github.com/feizc/Ingredients.
paper_ai_summary: A framework called Ingredients uses video diffusion Transformers to create customized videos by integrating multiple identity photos through facial extraction, multi-scale projection, and ID routing.
paper_ai_keywords: video diffusion Transformers facial extractor multi-scale projector ID router contextual space image query space-time regions multi-stage training protocol generative video control tools