title: Decoupling Angles and Strength in Low-rank Adaptation
summary: Parameter-Efficient FineTuning (PEFT) methods have recently gained
  significant popularity thanks to the widespread availability of large-scale
  pretrained models. These methods allow for quick adaptation to downstream tasks
  with minimal computational cost. However, popular finetuning methods such as
  LoRA exhibit limited robustness when it comes to hyperparameter choices or
  extended training regimes, preventing optimal out-of-the-box performance. In
  contrast, bounded approaches, such as ETHER, provide greater robustness but are
  limited to extremely low-rank adaptations and fixed-strength transformations,
  reducing their adaptation expressive power. In this work, we propose Decoupled
  Low-rank Adaptation (DeLoRA), a novel finetuning method that normalizes and
  scales learnable low-rank matrices. By bounding the distance of the
  transformation, DeLoRA effectively decouples the angular learning from the
  adaptation strength, enhancing robustness without compromising performance.
  Through evaluations on subject-driven image generation, natural language
  understanding, and instruction tuning, we show that DeLoRA matches or surpasses
  performance of competing PEFT methods, while exhibiting stronger robustness.
  Code is available at https://github.com/ExplainableML/DeLoRA.
paper_authors._id: 67ebabe3c545cab686735182
paper_authors.name: Massimo Bini
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-01T15:59:29.143Z
paper_authors._id: 63f62ee3b29015adc33aafa0
paper_authors.avatarUrl: /avatars/b7d7fe1b65333fb698402ed065fc5d13.svg
paper_authors.fullname: Massimo Bini
paper_authors.type: user
paper_authors.user: mwbini
paper_authors._id: 67ebabe3c545cab686735183
paper_authors.name: Leander Girrbach
paper_authors._id: 67ebabe3c545cab686735184
paper_authors.name: Zeynep Akata
paper_title: Decoupling Angles and Strength in Low-rank Adaptation
paper_summary: Parameter-Efficient FineTuning (PEFT) methods have recently gained
  significant popularity thanks to the widespread availability of large-scale
  pretrained models. These methods allow for quick adaptation to downstream tasks
  with minimal computational cost. However, popular finetuning methods such as
  LoRA exhibit limited robustness when it comes to hyperparameter choices or
  extended training regimes, preventing optimal out-of-the-box performance. In
  contrast, bounded approaches, such as ETHER, provide greater robustness but are
  limited to extremely low-rank adaptations and fixed-strength transformations,
  reducing their adaptation expressive power. In this work, we propose Decoupled
  Low-rank Adaptation (DeLoRA), a novel finetuning method that normalizes and
  scales learnable low-rank matrices. By bounding the distance of the
  transformation, DeLoRA effectively decouples the angular learning from the
  adaptation strength, enhancing robustness without compromising performance.
  Through evaluations on subject-driven image generation, natural language
  understanding, and instruction tuning, we show that DeLoRA matches or surpasses
  performance of competing PEFT methods, while exhibiting stronger robustness.
  Code is available at https://github.com/ExplainableML/DeLoRA.
paper_ai_summary: DeLoRA, a novel parameter-efficient fine-tuning method, enhances robustness and performance in various tasks compared to existing approaches by decoupling angular learning from adaptation strength.
paper_ai_keywords: Parameter-Efficient FineTuning (PEFT) LoRA bounded approaches ETHER Decoupled Low-rank Adaptation (DeLoRA) low-rank matrices robustness subject-driven image generation natural language understanding instruction tuning