LoRA-based Multi-task Fine-tuning. LoRAHub [55]
leverages a composition of multiple trained LoRA modules
for cross-task transfer to fine-tune the model on new tasks.
Specifically, LoRAHub trains task-specific LoRA modules in
a variety of tasks to obtain a synthesized module, ˆm =
up), which
(w1W 1
is then amalgamated with the LLMs to adapt the new task.
Thus, the objective of LoRAHub is to find the best weight
set {w1, w2,··· , wN}, which is achieved by the gradient-free
combinatorial optimization approach Shiwa [91]. MOELoRA
[56] combines LoRA with mixture-of-experts (MoE) for multi-
task fine-tuning, in which each expert is a LoRA module
for learning task-specific knowledge. Additionally, MOELoRA
devises a task-motivated gate function to produce distinct
fine-tuned parameters for various tasks. L-LoRA (Linearized
LoRA) [57] is a linearized PEFT method to improve the
multi-task fusion capability of fine-tuned task-specific models
with low computation costs. L-LoRA constructs a linear
Parameter-EfficientFine-TuningMethods