LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA
  Composition
Low-rank adaptations (LoRA) are often employed to fine-tune large language
models (LLMs) for new tasks. This paper investigates LoRA composability for
cross-task generalization and introduces LoraHub, a strategic framework devised
for the purposive assembly of LoRA modules trained on diverse given tasks, with
the objective of achieving adaptable performance on unseen tasks. With just a
few examples from a novel task, LoraHub enables the fluid combination of
multiple LoRA modules, eradicating the need for human expertise. Notably, the
composition requires neither additional model parameters nor gradients. Our
empirical results, derived from the Big-Bench Hard (BBH) benchmark, suggest
that LoraHub can effectively mimic the performance of in-context learning in
few-shot scenarios, excluding the necessity of in-context examples alongside
each inference input. A significant contribution of our research is the
fostering of a community for LoRA, where users can share their trained LoRA
modules, thereby facilitating their application to new tasks. We anticipate
this resource will widen access to and spur advancements in general
intelligence as well as LLMs in production. Code will be available at
https://github.com/sail-sg/lorahub.
64c086e65e03515457f6905a
Chengsong Huang
claimed_verified
2023-07-26T09:03:42.431Z
62ea79dd01ed9b0e8f61ccd3
/avatars/70af83e0e267be39fcd5f23b85e2dafa.svg
Chengsong Huang
user
ChengsongHuang
64c086e65e03515457f6905b
Qian Liu
claimed_verified
2023-07-26T08:43:11.010Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
64c086e65e03515457f6905c
Bill Yuchen Lin
claimed_verified
2023-07-26T19:12:37.394Z
607f666a4ad99100d63ce35c
https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
Bill Yuchen Lin
user
yuchenlin
64c086e65e03515457f6905d
Tianyu Pang
admin_assigned
2023-07-26T08:51:46.760Z
63d91b6d255ef6add20e1b38
https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
Tianyu Pang
user
P2333
64c086e65e03515457f6905e
Chao Du
admin_assigned
2023-07-26T08:53:11.748Z
632407c892e07e3ca20aca28
/avatars/23b51b37b12b51a0947f687d1de4d3b5.svg
Chao Du
user
duchao
64c086e65e03515457f6905f
Min Lin
claimed_verified
2024-01-22T12:01:09.645Z
6565d5fd656151d58b1d9afb
/avatars/0c416ea74347c6b5bd27f49aec309c48.svg
Min Lin
user
mavenlin
LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA
  Composition
Low-rank adaptations (LoRA) are often employed to fine-tune large language
models (LLMs) for new tasks. This paper investigates LoRA composability for
cross-task generalization and introduces LoraHub, a strategic framework devised
for the purposive assembly of LoRA modules trained on diverse given tasks, with
the objective of achieving adaptable performance on unseen tasks. With just a
few examples from a novel task, LoraHub enables the fluid combination of
multiple LoRA modules, eradicating the need for human expertise. Notably, the
composition requires neither additional model parameters nor gradients. Our
empirical results, derived from the Big-Bench Hard (BBH) benchmark, suggest
that LoraHub can effectively mimic the performance of in-context learning in
few-shot scenarios, excluding the necessity of in-context examples alongside
each inference input. A significant contribution of our research is the
fostering of a community for LoRA, where users can share their trained LoRA
modules, thereby facilitating their application to new tasks. We anticipate
this resource will widen access to and spur advancements in general
intelligence as well as LLMs in production. Code will be available at
https://github.com/sail-sg/lorahub.
LoraHub, a framework for combining LoRA modules, enables few-shot cross-task generalization and facilitates community sharing of LRA modules for large language models.
Low-rank adaptations LoRA large language models LLMs cross-task generalization LoraHub Big-Bench Hard BBH benchmark few-shot scenarios in-context learning general intelligence