title: FastComposer: Tuning-Free Multi-Subject Image Generation with Localized
    Attention
summary: Diffusion models excel at text-to-image generation, especially in
  subject-driven generation for personalized images. However, existing methods
  are inefficient due to the subject-specific fine-tuning, which is
  computationally intensive and hampers efficient deployment. Moreover, existing
  methods struggle with multi-subject generation as they often blend features
  among subjects. We present FastComposer which enables efficient, personalized,
  multi-subject text-to-image generation without fine-tuning. FastComposer uses
  subject embeddings extracted by an image encoder to augment the generic text
  conditioning in diffusion models, enabling personalized image generation based
  on subject images and textual instructions with only forward passes. To address
  the identity blending problem in the multi-subject generation, FastComposer
  proposes cross-attention localization supervision during training, enforcing
  the attention of reference subjects localized to the correct regions in the
  target images. Naively conditioning on subject embeddings results in subject
  overfitting. FastComposer proposes delayed subject conditioning in the
  denoising step to maintain both identity and editability in subject-driven
  image generation. FastComposer generates images of multiple unseen individuals
  with different styles, actions, and contexts. It achieves
  300times-2500times speedup compared to fine-tuning-based methods and
  requires zero extra storage for new subjects. FastComposer paves the way for
  efficient, personalized, and high-quality multi-subject image creation. Code,
  model, and dataset are available at
  https://github.com/mit-han-lab/fastcomposer.
paper_authors._id: 64657c4ee27766e8921b6ef7
paper_authors.name: Guangxuan Xiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-15T09:17:40.339Z
paper_authors._id: 6362fefe19cf373a5fc5b39e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6362fefe19cf373a5fc5b39e/v4uJ5bzjpZJOxqGUHYPM2.jpeg
paper_authors.fullname: Guangxuan Xiao
paper_authors.type: user
paper_authors.user: Guangxuan-Xiao
paper_authors._id: 64657c4ee27766e8921b6ef8
paper_authors.name: Tianwei Yin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T19:03:39.036Z
paper_authors._id: 63363b864067f020756275b7
paper_authors.avatarUrl: /avatars/bf18f9f76183f81589a80b3d0215f1a5.svg
paper_authors.fullname: Tianwei Yin
paper_authors.type: user
paper_authors.user: tianweiy
paper_authors._id: 64657c4ee27766e8921b6ef9
paper_authors.name: William T. Freeman
paper_authors._id: 64657c4ee27766e8921b6efa
paper_authors.name: Frédo Durand
paper_authors._id: 64657c4ee27766e8921b6efb
paper_authors.name: Song Han
paper_title: FastComposer: Tuning-Free Multi-Subject Image Generation with Localized
    Attention
paper_summary: Diffusion models excel at text-to-image generation, especially in
  subject-driven generation for personalized images. However, existing methods
  are inefficient due to the subject-specific fine-tuning, which is
  computationally intensive and hampers efficient deployment. Moreover, existing
  methods struggle with multi-subject generation as they often blend features
  among subjects. We present FastComposer which enables efficient, personalized,
  multi-subject text-to-image generation without fine-tuning. FastComposer uses
  subject embeddings extracted by an image encoder to augment the generic text
  conditioning in diffusion models, enabling personalized image generation based
  on subject images and textual instructions with only forward passes. To address
  the identity blending problem in the multi-subject generation, FastComposer
  proposes cross-attention localization supervision during training, enforcing
  the attention of reference subjects localized to the correct regions in the
  target images. Naively conditioning on subject embeddings results in subject
  overfitting. FastComposer proposes delayed subject conditioning in the
  denoising step to maintain both identity and editability in subject-driven
  image generation. FastComposer generates images of multiple unseen individuals
  with different styles, actions, and contexts. It achieves
  300times-2500times speedup compared to fine-tuning-based methods and
  requires zero extra storage for new subjects. FastComposer paves the way for
  efficient, personalized, and high-quality multi-subject image creation. Code,
  model, and dataset are available at
  https://github.com/mit-han-lab/fastcomposer.
paper_ai_summary: FastComposer achieves efficient, personalized, and high-quality multi-subject text-to-image generation without fine-tuning by using subject embeddings and cross-attention localization supervision.
paper_ai_keywords: diffusion models text-to-image generation subject-specific fine-tuning subject embeddings image encoder generic text conditioning cross-attention localization supervision delayed subject conditioning denoising step identity blending subject overfitting image generation speedup