GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient
  Fine-Tuning
Low-Rank Adaptation (LoRA) is a popular method for parameter-efficient
fine-tuning (PEFT) of generative models, valued for its simplicity and
effectiveness. Despite recent enhancements, LoRA still suffers from a
fundamental limitation: overfitting when the bottleneck is widened. It performs
best at ranks 32-64, yet its accuracy stagnates or declines at higher ranks,
still falling short of full fine-tuning (FFT) performance. We identify the root
cause as LoRA's structural bottleneck, which introduces gradient entanglement
to the unrelated input channels and distorts gradient propagation. To address
this, we introduce a novel structure, Granular Low-Rank Adaptation (GraLoRA)
that partitions weight matrices into sub-blocks, each with its own low-rank
adapter. With negligible computational or storage cost, GraLoRA overcomes
LoRA's limitations, effectively increases the representational capacity, and
more closely approximates FFT behavior. Experiments on code generation and
commonsense reasoning benchmarks show that GraLoRA consistently outperforms
LoRA and other baselines, achieving up to +8.5% absolute gain in Pass@1 on
HumanEval+. These improvements hold across model sizes and rank settings,
making GraLoRA a scalable and robust solution for PEFT. Code, data, and scripts
are available at https://github.com/SqueezeBits/GraLoRA.git
683674c419543f12e85c4f47
Yeonjoon Jung
claimed_verified
2025-05-28T08:57:44.292Z
66a8ba3b29470b614485db2e
/avatars/64d855b18df75b35eaed35b4b9282b78.svg
Yeonjoon Jung
user
yeonjoon-jung
683674c419543f12e85c4f48
Daehyun Ahn
admin_assigned
2025-05-28T12:22:24.944Z
64f58fd1d72303b303098198
/avatars/e45839a71445c9bffb4001f59241d931.svg
Daehyun Ahn
user
daehyunahn
683674c419543f12e85c4f49
Hyungjun Kim
admin_assigned
2025-05-28T12:22:39.658Z
671f5c7bd79a70b18f7db600
/avatars/80f6eaf612893f66c90c4e977f45483c.svg
Hyungjun Kim
user
HyungjunKim
683674c419543f12e85c4f4a
Taesu Kim
admin_assigned
2025-05-28T12:22:59.898Z
64dc388069d21c567b0794b0
/avatars/795ab1b7f970df3b589f6c1ca75df9ad.svg
Taesu Kim
user
tae-su-kim
683674c419543f12e85c4f4b
Eunhyeok Park
admin_assigned
2025-05-28T12:23:05.667Z
670a70e95f4ed2ff1f6d02ed
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1V2NN6G6w-JBWRsVYW2Ed.png
Eunhyeok Park
user
hyek90
GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient
  Fine-Tuning
Low-Rank Adaptation (LoRA) is a popular method for parameter-efficient
fine-tuning (PEFT) of generative models, valued for its simplicity and
effectiveness. Despite recent enhancements, LoRA still suffers from a
fundamental limitation: overfitting when the bottleneck is widened. It performs
best at ranks 32-64, yet its accuracy stagnates or declines at higher ranks,
still falling short of full fine-tuning (FFT) performance. We identify the root
cause as LoRA's structural bottleneck, which introduces gradient entanglement
to the unrelated input channels and distorts gradient propagation. To address
this, we introduce a novel structure, Granular Low-Rank Adaptation (GraLoRA)
that partitions weight matrices into sub-blocks, each with its own low-rank
adapter. With negligible computational or storage cost, GraLoRA overcomes
LoRA's limitations, effectively increases the representational capacity, and
more closely approximates FFT behavior. Experiments on code generation and
commonsense reasoning benchmarks show that GraLoRA consistently outperforms
LoRA and other baselines, achieving up to +8.5% absolute gain in Pass@1 on
HumanEval+. These improvements hold across model sizes and rank settings,
making GraLoRA a scalable and robust solution for PEFT. Code, data, and scripts
are available at https://github.com/SqueezeBits/GraLoRA.git
Granular Low-Rank Adaptation (GraLoRA) improves upon Low-Rank Adaptation (LoRA) by partitioning weight matrices to mitigate overfitting and enhance performance in parameter-efficient fine-tuning.
Low-Rank Adaptation LoRA parameter-efficient fine-tuning PEFT full fine-tuning FFT gradient entanglement Granular Low-Rank Adaptation GraLoRA weight matrices sub-blocks low-rank adapter Pass@1 HumanEval+