title: MultiLoRA: Democratizing LoRA for Better Multi-Task Learning
summary: LoRA achieves remarkable resource efficiency and comparable performance when
  adapting LLMs for specific tasks. Since ChatGPT demonstrated superior
  performance on various tasks, there has been a growing desire to adapt one
  model for all tasks. However, the explicit low-rank of LoRA limits the
  adaptation performance in complex multi-task scenarios. LoRA is dominated by a
  small number of top singular vectors while fine-tuning decomposes into a set of
  less important unitary transforms. In this paper, we propose MultiLoRA for
  better multi-task adaptation by reducing the dominance of top singular vectors
  observed in LoRA. MultiLoRA scales LoRA modules horizontally and change
  parameter initialization of adaptation matrices to reduce parameter dependency,
  thus yields more balanced unitary subspaces. We unprecedentedly construct
  specialized training data by mixing datasets of instruction follow, natural
  language understanding, world knowledge, to cover semantically and
  syntactically different samples. With only 2.5% of additional parameters,
  MultiLoRA outperforms single LoRA counterparts and fine-tuning on multiple
  benchmarks and model scales. Further investigation into weight update matrices
  of MultiLoRA exhibits reduced dependency on top singular vectors and more
  democratic unitary transform contributions.
paper_authors._id: 655c194eb9d6d283fbedd5a5
paper_authors.name: Yiming Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T11:24:28.333Z
paper_authors._id: 62dce680aae4aa3ed0980de4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658644064540-noauth.jpeg
paper_authors.fullname: yiming Wang
paper_authors.type: user
paper_authors.user: Alsace08
paper_authors._id: 655c194eb9d6d283fbedd5a6
paper_authors.name: Yu Lin
paper_authors._id: 655c194eb9d6d283fbedd5a7
paper_authors.name: Xiaodong Zeng
paper_authors._id: 655c194eb9d6d283fbedd5a8
paper_authors.name: Guannan Zhang
paper_title: MultiLoRA: Democratizing LoRA for Better Multi-Task Learning
paper_summary: LoRA achieves remarkable resource efficiency and comparable performance when
  adapting LLMs for specific tasks. Since ChatGPT demonstrated superior
  performance on various tasks, there has been a growing desire to adapt one
  model for all tasks. However, the explicit low-rank of LoRA limits the
  adaptation performance in complex multi-task scenarios. LoRA is dominated by a
  small number of top singular vectors while fine-tuning decomposes into a set of
  less important unitary transforms. In this paper, we propose MultiLoRA for
  better multi-task adaptation by reducing the dominance of top singular vectors
  observed in LoRA. MultiLoRA scales LoRA modules horizontally and change
  parameter initialization of adaptation matrices to reduce parameter dependency,
  thus yields more balanced unitary subspaces. We unprecedentedly construct
  specialized training data by mixing datasets of instruction follow, natural
  language understanding, world knowledge, to cover semantically and
  syntactically different samples. With only 2.5% of additional parameters,
  MultiLoRA outperforms single LoRA counterparts and fine-tuning on multiple
  benchmarks and model scales. Further investigation into weight update matrices
  of MultiLoRA exhibits reduced dependency on top singular vectors and more
  democratic unitary transform contributions.
paper_ai_keywords: LoRA LLMs (Large Language Models) ChatGPT low-rank factorization fine-tuning MultiLoRA singular vectors unitary transforms parameter initialization specialized training data instruction follow natural language understanding world knowledge weight update matrices parameter dependency