title: Tied-Lora: Enhacing parameter efficiency of LoRA with weight tying
summary: We propose Tied-LoRA, a simple paradigm utilizes weight tying and selective
  training to further increase parameter efficiency of the Low-rank adaptation
  (LoRA) method. Our investigations include all feasible combinations parameter
  training/freezing in conjunction with weight tying to identify the optimal
  balance between performance and the number of trainable parameters. Through
  experiments covering a variety of tasks and two base language models, we
  provide analysis revealing trade-offs between efficiency and performance. Our
  experiments uncovered a particular Tied-LoRA configuration that stands out by
  demonstrating comparable performance across several tasks while employing only
  13~\% percent of parameters utilized by the standard LoRA method.
paper_authors._id: 6556df5bc4865c852d4df805
paper_authors.name: Adithya Renduchintala
paper_authors._id: 6556df5bc4865c852d4df806
paper_authors.name: Tugrul Konuk
paper_authors._id: 6556df5bc4865c852d4df807
paper_authors.name: Oleksii Kuchaiev
paper_title: Tied-Lora: Enhacing parameter efficiency of LoRA with weight tying
paper_summary: We propose Tied-LoRA, a simple paradigm utilizes weight tying and selective
  training to further increase parameter efficiency of the Low-rank adaptation
  (LoRA) method. Our investigations include all feasible combinations parameter
  training/freezing in conjunction with weight tying to identify the optimal
  balance between performance and the number of trainable parameters. Through
  experiments covering a variety of tasks and two base language models, we
  provide analysis revealing trade-offs between efficiency and performance. Our
  experiments uncovered a particular Tied-LoRA configuration that stands out by
  demonstrating comparable performance across several tasks while employing only
  13~\% percent of parameters utilized by the standard LoRA method.
paper_ai_summary: Tied-LoRA increases parameter efficiency in LoRA by using weight tying and selective training, demonstrating better efficiency with reduced parameters.
paper_ai_keywords: Tied-LoRA weight tying selective training Low-rank adaptation LoRA parameter efficiency base language models parameter training freezing performance efficiency