Tied-Lora: Enhacing parameter efficiency of LoRA with weight tying
We propose Tied-LoRA, a simple paradigm utilizes weight tying and selective
training to further increase parameter efficiency of the Low-rank adaptation
(LoRA) method. Our investigations include all feasible combinations parameter
training/freezing in conjunction with weight tying to identify the optimal
balance between performance and the number of trainable parameters. Through
experiments covering a variety of tasks and two base language models, we
provide analysis revealing trade-offs between efficiency and performance. Our
experiments uncovered a particular Tied-LoRA configuration that stands out by
demonstrating comparable performance across several tasks while employing only
13~\% percent of parameters utilized by the standard LoRA method.
6556df5bc4865c852d4df805
Adithya Renduchintala
6556df5bc4865c852d4df806
Tugrul Konuk
6556df5bc4865c852d4df807
Oleksii Kuchaiev
Tied-Lora: Enhacing parameter efficiency of LoRA with weight tying
We propose Tied-LoRA, a simple paradigm utilizes weight tying and selective
training to further increase parameter efficiency of the Low-rank adaptation
(LoRA) method. Our investigations include all feasible combinations parameter
training/freezing in conjunction with weight tying to identify the optimal
balance between performance and the number of trainable parameters. Through
experiments covering a variety of tasks and two base language models, we
provide analysis revealing trade-offs between efficiency and performance. Our
experiments uncovered a particular Tied-LoRA configuration that stands out by
demonstrating comparable performance across several tasks while employing only
13~\% percent of parameters utilized by the standard LoRA method.
Tied-LoRA increases parameter efficiency in LoRA by using weight tying and selective training, demonstrating better efficiency with reduced parameters.
Tied-LoRA weight tying selective training Low-rank adaptation LoRA parameter efficiency base language models parameter training freezing performance efficiency