title: Mixture-of-Subspaces in Low-Rank Adaptation
summary: In this paper, we introduce a subspace-inspired Low-Rank Adaptation (LoRA)
  method, which is computationally efficient, easy to implement, and readily
  applicable to large language, multimodal, and diffusion models. Initially, we
  equivalently decompose the weights of LoRA into two subspaces, and find that
  simply mixing them can enhance performance. To study such a phenomenon, we
  revisit it through a fine-grained subspace lens, showing that such modification
  is equivalent to employing a fixed mixer to fuse the subspaces. To be more
  flexible, we jointly learn the mixer with the original LoRA weights, and term
  the method Mixture-of-Subspaces LoRA (MoSLoRA). MoSLoRA consistently
  outperforms LoRA on tasks in different modalities, including commonsense
  reasoning, visual instruction tuning, and subject-driven text-to-image
  generation, demonstrating its effectiveness and robustness. Codes are available
  at https://github.com/wutaiqiang/MoSLoRA{github}.
paper_authors._id: 66729d4109651d9c009a6c1e
paper_authors.name: Taiqiang Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:19:15.593Z
paper_authors._id: 6621cea88850e38ffbb1854f
paper_authors.avatarUrl: /avatars/6d73d947046faa32260ee325069976d9.svg
paper_authors.fullname: Taki WU
paper_authors.type: user
paper_authors.user: taki555
paper_authors._id: 66729d4109651d9c009a6c1f
paper_authors.name: Jiahao Wang
paper_authors._id: 66729d4109651d9c009a6c20
paper_authors.name: Zhe Zhao
paper_authors._id: 66729d4109651d9c009a6c21
paper_authors.name: Ngai Wong
paper_title: Mixture-of-Subspaces in Low-Rank Adaptation
paper_summary: In this paper, we introduce a subspace-inspired Low-Rank Adaptation (LoRA)
  method, which is computationally efficient, easy to implement, and readily
  applicable to large language, multimodal, and diffusion models. Initially, we
  equivalently decompose the weights of LoRA into two subspaces, and find that
  simply mixing them can enhance performance. To study such a phenomenon, we
  revisit it through a fine-grained subspace lens, showing that such modification
  is equivalent to employing a fixed mixer to fuse the subspaces. To be more
  flexible, we jointly learn the mixer with the original LoRA weights, and term
  the method Mixture-of-Subspaces LoRA (MoSLoRA). MoSLoRA consistently
  outperforms LoRA on tasks in different modalities, including commonsense
  reasoning, visual instruction tuning, and subject-driven text-to-image
  generation, demonstrating its effectiveness and robustness. Codes are available
  at https://github.com/wutaiqiang/MoSLoRA{github}.
paper_ai_summary: A subspace-inspired method called Mixture-of-Subspaces LoRA (MoSLoRA) enhances performance across various tasks by flexible learning of a mixer with LoRA weights.
paper_ai_keywords: Low-Rank Adaptation (LoRA) subspace Mixture-of-Subspaces LoRA (MoSLoRA) commonsense reasoning visual instruction tuning text-to-image generation