title: Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept
    Customization of Diffusion Models
summary: Public large-scale text-to-image diffusion models, such as Stable Diffusion,
  have gained significant attention from the community. These models can be
  easily customized for new concepts using low-rank adaptations (LoRAs). However,
  the utilization of multiple concept LoRAs to jointly support multiple
  customized concepts presents a challenge. We refer to this scenario as
  decentralized multi-concept customization, which involves single-client concept
  tuning and center-node concept fusion. In this paper, we propose a new
  framework called Mix-of-Show that addresses the challenges of decentralized
  multi-concept customization, including concept conflicts resulting from
  existing single-client LoRA tuning and identity loss during model fusion.
  Mix-of-Show adopts an embedding-decomposed LoRA (ED-LoRA) for single-client
  tuning and gradient fusion for the center node to preserve the in-domain
  essence of single concepts and support theoretically limitless concept fusion.
  Additionally, we introduce regionally controllable sampling, which extends
  spatially controllable sampling (e.g., ControlNet and T2I-Adaptor) to address
  attribute binding and missing object problems in multi-concept sampling.
  Extensive experiments demonstrate that Mix-of-Show is capable of composing
  multiple customized concepts with high fidelity, including characters, objects,
  and scenes.
paper_authors._id: 64756f8122b4dec4524c5614
paper_authors.name: Yuchao Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T09:01:32.114Z
paper_authors._id: 63021630a35b21bd8a53305a
paper_authors.avatarUrl: /avatars/7a7e8b39749eda61e57d8a1908726558.svg
paper_authors.fullname: Gu Yuchao
paper_authors.type: user
paper_authors.user: guyuchao
paper_authors._id: 64756f8122b4dec4524c5615
paper_authors.name: Xintao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-18T08:21:29.663Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 64756f8122b4dec4524c5616
paper_authors.name: Jay Zhangjie Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-16T07:25:28.335Z
paper_authors._id: 633aaf695df91da9cea92960
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633aaf695df91da9cea92960/h1oOWUudbdlsd63Q5w0hM.png
paper_authors.fullname: Jay Wu
paper_authors.type: user
paper_authors.user: jayw
paper_authors._id: 64756f8122b4dec4524c5617
paper_authors.name: Yujun Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-10T09:25:23.067Z
paper_authors._id: 645ce309a03f3ebb0bdf2b89
paper_authors.avatarUrl: /avatars/652db4d2badd1191c6612f0950a88a88.svg
paper_authors.fullname: Yujun Shi
paper_authors.type: user
paper_authors.user: PeiyangShi
paper_authors._id: 64756f8122b4dec4524c5618
paper_authors.name: Yunpeng Chen
paper_authors._id: 64756f8122b4dec4524c5619
paper_authors.name: Zihan Fan
paper_authors._id: 64756f8122b4dec4524c561a
paper_authors.name: Wuyou Xiao
paper_authors._id: 64756f8122b4dec4524c561b
paper_authors.name: Rui Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T07:49:27.060Z
paper_authors._id: 652b83b73b5997ed71a310f2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
paper_authors.fullname: Rui Zhao
paper_authors.type: user
paper_authors.user: ruizhaocv
paper_authors._id: 64756f8122b4dec4524c561c
paper_authors.name: Shuning Chang
paper_authors._id: 64756f8122b4dec4524c561d
paper_authors.name: Weijia Wu
paper_authors._id: 64756f8122b4dec4524c561e
paper_authors.name: Yixiao Ge
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-01T16:30:43.920Z
paper_authors._id: 640e9762b03f4cd29f58d982
paper_authors.avatarUrl: /avatars/81da37d628163fe3e094b247c7c3a3b5.svg
paper_authors.fullname: Yixiao Ge
paper_authors.type: user
paper_authors.user: yxgeee
paper_authors._id: 64756f8122b4dec4524c561f
paper_authors.name: Ying Shan
paper_authors._id: 64756f8122b4dec4524c5620
paper_authors.name: Mike Zheng Shou
paper_title: Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept
    Customization of Diffusion Models
paper_summary: Public large-scale text-to-image diffusion models, such as Stable Diffusion,
  have gained significant attention from the community. These models can be
  easily customized for new concepts using low-rank adaptations (LoRAs). However,
  the utilization of multiple concept LoRAs to jointly support multiple
  customized concepts presents a challenge. We refer to this scenario as
  decentralized multi-concept customization, which involves single-client concept
  tuning and center-node concept fusion. In this paper, we propose a new
  framework called Mix-of-Show that addresses the challenges of decentralized
  multi-concept customization, including concept conflicts resulting from
  existing single-client LoRA tuning and identity loss during model fusion.
  Mix-of-Show adopts an embedding-decomposed LoRA (ED-LoRA) for single-client
  tuning and gradient fusion for the center node to preserve the in-domain
  essence of single concepts and support theoretically limitless concept fusion.
  Additionally, we introduce regionally controllable sampling, which extends
  spatially controllable sampling (e.g., ControlNet and T2I-Adaptor) to address
  attribute binding and missing object problems in multi-concept sampling.
  Extensive experiments demonstrate that Mix-of-Show is capable of composing
  multiple customized concepts with high fidelity, including characters, objects,
  and scenes.
paper_ai_summary: Mix-of-Show addresses challenges in decentralized multi-concept customization for text-to-image diffusion models using embedding-decomposed LoRA, gradient fusion, and regionally controllable sampling.
paper_ai_keywords: text-to-image diffusion models Stable Diffusion low-rank adaptations LoRAs decentralized multi-concept customization single-client concept tuning center-node concept fusion concept conflicts identity loss embedding-decomposed LoRA gradient fusion regionally controllable sampling ControlNet T2I-Adaptor attribute binding missing object problems