title: MoA: Mixture-of-Attention for Subject-Context Disentanglement in
    Personalized Image Generation
summary: We introduce a new architecture for personalization of text-to-image
  diffusion models, coined Mixture-of-Attention (MoA). Inspired by the
  Mixture-of-Experts mechanism utilized in large language models (LLMs), MoA
  distributes the generation workload between two attention pathways: a
  personalized branch and a non-personalized prior branch. MoA is designed to
  retain the original model's prior by fixing its attention layers in the prior
  branch, while minimally intervening in the generation process with the
  personalized branch that learns to embed subjects in the layout and context
  generated by the prior branch. A novel routing mechanism manages the
  distribution of pixels in each layer across these branches to optimize the
  blend of personalized and generic content creation. Once trained, MoA
  facilitates the creation of high-quality, personalized images featuring
  multiple subjects with compositions and interactions as diverse as those
  generated by the original model. Crucially, MoA enhances the distinction
  between the model's pre-existing capability and the newly augmented
  personalized intervention, thereby offering a more disentangled subject-context
  control that was previously unattainable. Project page:
  https://snap-research.github.io/mixture-of-attention
paper_authors._id: 6621d740fa7c948e09cf9173
paper_authors.name: Kuan-Chieh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:53:21.649Z
paper_authors._id: 648ca58a39d2584ee47efef6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648ca58a39d2584ee47efef6/R7B72bnwc59mdK45rmzYS.png
paper_authors.fullname: Kuan-Chieh Wang
paper_authors.type: user
paper_authors.user: wangkua1
paper_authors._id: 6621d740fa7c948e09cf9174
paper_authors.name: Wang
paper_authors._id: 6621d740fa7c948e09cf9175
paper_authors.name: Daniil Ostashev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:52:46.710Z
paper_authors._id: 64cd594f6f107411da526af6
paper_authors.avatarUrl: /avatars/acddd3174e0495c1d111c889b26d3fcb.svg
paper_authors.fullname: Daniil Ostashev
paper_authors.type: user
paper_authors.user: dostashev
paper_authors._id: 6621d740fa7c948e09cf9176
paper_authors.name: Yuwei Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:52:55.740Z
paper_authors._id: 6406d1dcc3ab325efa9db198
paper_authors.avatarUrl: /avatars/781f3cd455596f7cffa0906cc487f79b.svg
paper_authors.fullname: Yuwei Fang
paper_authors.type: user
paper_authors.user: studyfang
paper_authors._id: 6621d740fa7c948e09cf9177
paper_authors.name: Sergey Tulyakov
paper_authors._id: 6621d740fa7c948e09cf9178
paper_authors.name: Kfir Aberman
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-04-19T02:30:36.132Z
paper_authors._id: 64db29097266618e853dd6ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64db29097266618e853dd6ec/r0MaPQCfAxeKv3ycdKYLK.jpeg
paper_authors.fullname: Kfir Aberman
paper_authors.type: user
paper_authors.user: kaberman
paper_title: MoA: Mixture-of-Attention for Subject-Context Disentanglement in
    Personalized Image Generation
paper_summary: We introduce a new architecture for personalization of text-to-image
  diffusion models, coined Mixture-of-Attention (MoA). Inspired by the
  Mixture-of-Experts mechanism utilized in large language models (LLMs), MoA
  distributes the generation workload between two attention pathways: a
  personalized branch and a non-personalized prior branch. MoA is designed to
  retain the original model's prior by fixing its attention layers in the prior
  branch, while minimally intervening in the generation process with the
  personalized branch that learns to embed subjects in the layout and context
  generated by the prior branch. A novel routing mechanism manages the
  distribution of pixels in each layer across these branches to optimize the
  blend of personalized and generic content creation. Once trained, MoA
  facilitates the creation of high-quality, personalized images featuring
  multiple subjects with compositions and interactions as diverse as those
  generated by the original model. Crucially, MoA enhances the distinction
  between the model's pre-existing capability and the newly augmented
  personalized intervention, thereby offering a more disentangled subject-context
  control that was previously unattainable. Project page:
  https://snap-research.github.io/mixture-of-attention
paper_ai_keywords: Mixture-of-Attention (MoA) Mixture-of-Experts mechanism diffusion models attention pathways personalized branch non-personalized prior branch attention layers routing mechanism Mixture-of-Attention (MoA) Mixture-of-Experts mechanismchanism diffusion models attention pathways personalized branch non-personalized prior branch attention layers... routing mechanism high-quality personalized images question generation disentangled subject-context control