title: Orthogonal Adaptation for Modular Customization of Diffusion Models
summary: Customization techniques for text-to-image models have paved the way for a
  wide range of previously unattainable applications, enabling the generation of
  specific concepts across diverse contexts and styles. While existing methods
  facilitate high-fidelity customization for individual concepts or a limited,
  pre-defined set of them, they fall short of achieving scalability, where a
  single model can seamlessly render countless concepts. In this paper, we
  address a new problem called Modular Customization, with the goal of
  efficiently merging customized models that were fine-tuned independently for
  individual concepts. This allows the merged model to jointly synthesize
  concepts in one image without compromising fidelity or incurring any additional
  computational costs.
    To address this problem, we introduce Orthogonal Adaptation, a method
  designed to encourage the customized models, which do not have access to each
  other during fine-tuning, to have orthogonal residual weights. This ensures
  that during inference time, the customized models can be summed with minimal
  interference.
    Our proposed method is both simple and versatile, applicable to nearly all
  optimizable weights in the model architecture. Through an extensive set of
  quantitative and qualitative evaluations, our method consistently outperforms
  relevant baselines in terms of efficiency and identity preservation,
  demonstrating a significant leap toward scalable customization of diffusion
  models.
paper_authors._id: 656fe71fcb4bff8b10752575
paper_authors.name: Ryan Po
paper_authors._id: 656fe71fcb4bff8b10752576
paper_authors.name: Guandao Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T12:26:44.545Z
paper_authors._id: 64547ca006728ff79a38e7a5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64547ca006728ff79a38e7a5/xfTVQXMSfRv-IbljFpswQ.jpeg
paper_authors.fullname: Guandao Yang
paper_authors.type: user
paper_authors.user: guandao
paper_authors._id: 656fe71fcb4bff8b10752577
paper_authors.name: Kfir Aberman
paper_authors._id: 656fe71fcb4bff8b10752578
paper_authors.name: Gordon Wetzstein
paper_title: Orthogonal Adaptation for Modular Customization of Diffusion Models
paper_summary: Customization techniques for text-to-image models have paved the way for a
  wide range of previously unattainable applications, enabling the generation of
  specific concepts across diverse contexts and styles. While existing methods
  facilitate high-fidelity customization for individual concepts or a limited,
  pre-defined set of them, they fall short of achieving scalability, where a
  single model can seamlessly render countless concepts. In this paper, we
  address a new problem called Modular Customization, with the goal of
  efficiently merging customized models that were fine-tuned independently for
  individual concepts. This allows the merged model to jointly synthesize
  concepts in one image without compromising fidelity or incurring any additional
  computational costs.
    To address this problem, we introduce Orthogonal Adaptation, a method
  designed to encourage the customized models, which do not have access to each
  other during fine-tuning, to have orthogonal residual weights. This ensures
  that during inference time, the customized models can be summed with minimal
  interference.
    Our proposed method is both simple and versatile, applicable to nearly all
  optimizable weights in the model architecture. Through an extensive set of
  quantitative and qualitative evaluations, our method consistently outperforms
  relevant baselines in terms of efficiency and identity preservation,
  demonstrating a significant leap toward scalable customization of diffusion
  models.
paper_ai_keywords: text-to-image models high-fidelity customization Modular Customization Orthogonal Adaptation residual weights diffusion models