MoA: Mixture-of-Attention for Subject-Context Disentanglement in
  Personalized Image Generation
We introduce a new architecture for personalization of text-to-image
diffusion models, coined Mixture-of-Attention (MoA). Inspired by the
Mixture-of-Experts mechanism utilized in large language models (LLMs), MoA
distributes the generation workload between two attention pathways: a
personalized branch and a non-personalized prior branch. MoA is designed to
retain the original model's prior by fixing its attention layers in the prior
branch, while minimally intervening in the generation process with the
personalized branch that learns to embed subjects in the layout and context
generated by the prior branch. A novel routing mechanism manages the
distribution of pixels in each layer across these branches to optimize the
blend of personalized and generic content creation. Once trained, MoA
facilitates the creation of high-quality, personalized images featuring
multiple subjects with compositions and interactions as diverse as those
generated by the original model. Crucially, MoA enhances the distinction
between the model's pre-existing capability and the newly augmented
personalized intervention, thereby offering a more disentangled subject-context
control that was previously unattainable. Project page:
https://snap-research.github.io/mixture-of-attention
6621d740fa7c948e09cf9173
Kuan-Chieh
admin_assigned
2024-04-19T08:53:21.649Z
648ca58a39d2584ee47efef6
https://cdn-avatars.huggingface.co/v1/production/uploads/648ca58a39d2584ee47efef6/R7B72bnwc59mdK45rmzYS.png
Kuan-Chieh Wang
user
wangkua1
6621d740fa7c948e09cf9174
Wang
6621d740fa7c948e09cf9175
Daniil Ostashev
admin_assigned
2024-04-19T08:52:46.710Z
64cd594f6f107411da526af6
/avatars/acddd3174e0495c1d111c889b26d3fcb.svg
Daniil Ostashev
user
dostashev
6621d740fa7c948e09cf9176
Yuwei Fang
admin_assigned
2024-04-19T08:52:55.740Z
6406d1dcc3ab325efa9db198
/avatars/781f3cd455596f7cffa0906cc487f79b.svg
Yuwei Fang
user
studyfang
6621d740fa7c948e09cf9177
Sergey Tulyakov
6621d740fa7c948e09cf9178
Kfir Aberman
extracted_pending
2024-04-19T02:30:36.132Z
64db29097266618e853dd6ec
https://cdn-avatars.huggingface.co/v1/production/uploads/64db29097266618e853dd6ec/r0MaPQCfAxeKv3ycdKYLK.jpeg
Kfir Aberman
user
kaberman
MoA: Mixture-of-Attention for Subject-Context Disentanglement in
  Personalized Image Generation
We introduce a new architecture for personalization of text-to-image
diffusion models, coined Mixture-of-Attention (MoA). Inspired by the
Mixture-of-Experts mechanism utilized in large language models (LLMs), MoA
distributes the generation workload between two attention pathways: a
personalized branch and a non-personalized prior branch. MoA is designed to
retain the original model's prior by fixing its attention layers in the prior
branch, while minimally intervening in the generation process with the
personalized branch that learns to embed subjects in the layout and context
generated by the prior branch. A novel routing mechanism manages the
distribution of pixels in each layer across these branches to optimize the
blend of personalized and generic content creation. Once trained, MoA
facilitates the creation of high-quality, personalized images featuring
multiple subjects with compositions and interactions as diverse as those
generated by the original model. Crucially, MoA enhances the distinction
between the model's pre-existing capability and the newly augmented
personalized intervention, thereby offering a more disentangled subject-context
control that was previously unattainable. Project page:
https://snap-research.github.io/mixture-of-attention
Mixture-of-Attention (MoA) architecture personalizes text-to-image diffusion models by blending a fixed attention prior branch with a learnable personalized branch to enhance subject-context control.
Mixture-of-Attention (MoA) Mixture-of-Experts diffusion models attention pathways personalized branch non-personalized prior branch routing mechanism high-quality images subject-context control