SingLoRA: Low Rank Adaptation Using a Single Matrix
Low-Rank Adaptation (LoRA) has significantly advanced parameter-efficient
fine-tuning of large pretrained models. LoRA augments the pre-trained weights
of a model by adding the product of two smaller matrices that together form a
low-rank matrix update. Recent research has shown that scale disparities
between these two matrices often cause unstable training dynamics, leading to
suboptimal performance. In this paper, we propose SingLoRA, which reformulates
low-rank adaptation by learning the weights update as a decomposition of a
single low-rank matrix multiplied by its transpose. This simple design
inherently removes inter-matrix scale conflicts, ensuring stable optimization,
and roughly halves the parameter count. We analyze SingLoRA within the
infinite-width neural network framework, showing that it guarantees stable
feature learning by construction. Extensive experiments on multiple tasks
validate these benefits. In common sense reasoning, fine-tuning LLama 7B on
MNLI with SingLoRA achieves 91.3% accuracy - surpassing LoRA (89.1%) and LoRA+
(90.2%) - while using only 60% of their parameter budget. In image generation,
fine-tuning Stable Diffusion with SingLoRA significantly improves image
fidelity on DreamBooth, achieving a DINO similarity score of 0.151, compared to
scores of 0.148 and 0.143 for DoRA and LoRA, respectively.
686e0a5dcb5725779c60b4e6
David Bensaïd
686e0a5dcb5725779c60b4e7
Noam Rotstein
claimed_verified
2025-07-09T08:49:36.311Z
62b3e85bcbd2a402fc7804b1
/avatars/63125ce8a1e20b8c6e836f223d24284f.svg
noam rotstein
user
noamrot
686e0a5dcb5725779c60b4e8
Roy Velich
claimed_verified
2025-07-09T08:49:33.313Z
63ecaf7460ff4b318ad03ebb
/avatars/7a5bf1854f1eae9cc5fd8392a3f9fba3.svg
Roy Velich
user
royve
686e0a5dcb5725779c60b4e9
Daniel Bensaïd
686e0a5dcb5725779c60b4ea
Ron Kimmel
SingLoRA: Low Rank Adaptation Using a Single Matrix
Low-Rank Adaptation (LoRA) has significantly advanced parameter-efficient
fine-tuning of large pretrained models. LoRA augments the pre-trained weights
of a model by adding the product of two smaller matrices that together form a
low-rank matrix update. Recent research has shown that scale disparities
between these two matrices often cause unstable training dynamics, leading to
suboptimal performance. In this paper, we propose SingLoRA, which reformulates
low-rank adaptation by learning the weights update as a decomposition of a
single low-rank matrix multiplied by its transpose. This simple design
inherently removes inter-matrix scale conflicts, ensuring stable optimization,
and roughly halves the parameter count. We analyze SingLoRA within the
infinite-width neural network framework, showing that it guarantees stable
feature learning by construction. Extensive experiments on multiple tasks
validate these benefits. In common sense reasoning, fine-tuning LLama 7B on
MNLI with SingLoRA achieves 91.3% accuracy - surpassing LoRA (89.1%) and LoRA+
(90.2%) - while using only 60% of their parameter budget. In image generation,
fine-tuning Stable Diffusion with SingLoRA significantly improves image
fidelity on DreamBooth, achieving a DINO similarity score of 0.151, compared to
scores of 0.148 and 0.143 for DoRA and LoRA, respectively.
SingLoRA, a reformulated low-rank adaptation method, enhances parameter-efficient fine-tuning by learning a single low-rank matrix and its transpose, ensuring stable optimization and reducing parameter count.
Low-Rank Adaptation LoRA SingLoRA low-rank matrix parameter-efficient fine-tuning infinite-width neural network framework feature learning common sense reasoning fine-tuning LLama 7B MNLI image generation Stable Diffusion DreamBooth DINO similarity score DoRA