title: LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA
    Composition
summary: Low-rank adaptations (LoRA) are often employed to fine-tune large language
  models (LLMs) for new tasks. This paper investigates LoRA composability for
  cross-task generalization and introduces LoraHub, a strategic framework devised
  for the purposive assembly of LoRA modules trained on diverse given tasks, with
  the objective of achieving adaptable performance on unseen tasks. With just a
  few examples from a novel task, LoraHub enables the fluid combination of
  multiple LoRA modules, eradicating the need for human expertise. Notably, the
  composition requires neither additional model parameters nor gradients. Our
  empirical results, derived from the Big-Bench Hard (BBH) benchmark, suggest
  that LoraHub can effectively mimic the performance of in-context learning in
  few-shot scenarios, excluding the necessity of in-context examples alongside
  each inference input. A significant contribution of our research is the
  fostering of a community for LoRA, where users can share their trained LoRA
  modules, thereby facilitating their application to new tasks. We anticipate
  this resource will widen access to and spur advancements in general
  intelligence as well as LLMs in production. Code will be available at
  https://github.com/sail-sg/lorahub.
paper_authors._id: 64c086e65e03515457f6905a
paper_authors.name: Chengsong Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-26T09:03:42.431Z
paper_authors._id: 62ea79dd01ed9b0e8f61ccd3
paper_authors.avatarUrl: /avatars/70af83e0e267be39fcd5f23b85e2dafa.svg
paper_authors.fullname: Chengsong Huang
paper_authors.type: user
paper_authors.user: ChengsongHuang
paper_authors._id: 64c086e65e03515457f6905b
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-26T08:43:11.010Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 64c086e65e03515457f6905c
paper_authors.name: Bill Yuchen Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-26T19:12:37.394Z
paper_authors._id: 607f666a4ad99100d63ce35c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
paper_authors.fullname: Bill Yuchen Lin
paper_authors.type: user
paper_authors.user: yuchenlin
paper_authors._id: 64c086e65e03515457f6905d
paper_authors.name: Tianyu Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-26T08:51:46.760Z
paper_authors._id: 63d91b6d255ef6add20e1b38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
paper_authors.fullname: Tianyu Pang
paper_authors.type: user
paper_authors.user: P2333
paper_authors._id: 64c086e65e03515457f6905e
paper_authors.name: Chao Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-26T08:53:11.748Z
paper_authors._id: 632407c892e07e3ca20aca28
paper_authors.avatarUrl: /avatars/23b51b37b12b51a0947f687d1de4d3b5.svg
paper_authors.fullname: Chao Du
paper_authors.type: user
paper_authors.user: duchao
paper_authors._id: 64c086e65e03515457f6905f
paper_authors.name: Min Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-22T12:01:09.645Z
paper_authors._id: 6565d5fd656151d58b1d9afb
paper_authors.avatarUrl: /avatars/0c416ea74347c6b5bd27f49aec309c48.svg
paper_authors.fullname: Min Lin
paper_authors.type: user
paper_authors.user: mavenlin
paper_title: LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA
    Composition
paper_summary: Low-rank adaptations (LoRA) are often employed to fine-tune large language
  models (LLMs) for new tasks. This paper investigates LoRA composability for
  cross-task generalization and introduces LoraHub, a strategic framework devised
  for the purposive assembly of LoRA modules trained on diverse given tasks, with
  the objective of achieving adaptable performance on unseen tasks. With just a
  few examples from a novel task, LoraHub enables the fluid combination of
  multiple LoRA modules, eradicating the need for human expertise. Notably, the
  composition requires neither additional model parameters nor gradients. Our
  empirical results, derived from the Big-Bench Hard (BBH) benchmark, suggest
  that LoraHub can effectively mimic the performance of in-context learning in
  few-shot scenarios, excluding the necessity of in-context examples alongside
  each inference input. A significant contribution of our research is the
  fostering of a community for LoRA, where users can share their trained LoRA
  modules, thereby facilitating their application to new tasks. We anticipate
  this resource will widen access to and spur advancements in general
  intelligence as well as LLMs in production. Code will be available at
  https://github.com/sail-sg/lorahub.
paper_ai_summary: LoraHub, a framework for combining LoRA modules, enables few-shot cross-task generalization and facilitates community sharing of LRA modules for large language models.
paper_ai_keywords: Low-rank adaptations LoRA large language models LLMs cross-task generalization LoraHub Big-Bench Hard BBH benchmark few-shot scenarios in-context learning general intelligence