title: GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient
    Fine-Tuning
summary: Low-Rank Adaptation (LoRA) is a popular method for parameter-efficient
  fine-tuning (PEFT) of generative models, valued for its simplicity and
  effectiveness. Despite recent enhancements, LoRA still suffers from a
  fundamental limitation: overfitting when the bottleneck is widened. It performs
  best at ranks 32-64, yet its accuracy stagnates or declines at higher ranks,
  still falling short of full fine-tuning (FFT) performance. We identify the root
  cause as LoRA's structural bottleneck, which introduces gradient entanglement
  to the unrelated input channels and distorts gradient propagation. To address
  this, we introduce a novel structure, Granular Low-Rank Adaptation (GraLoRA)
  that partitions weight matrices into sub-blocks, each with its own low-rank
  adapter. With negligible computational or storage cost, GraLoRA overcomes
  LoRA's limitations, effectively increases the representational capacity, and
  more closely approximates FFT behavior. Experiments on code generation and
  commonsense reasoning benchmarks show that GraLoRA consistently outperforms
  LoRA and other baselines, achieving up to +8.5% absolute gain in Pass@1 on
  HumanEval+. These improvements hold across model sizes and rank settings,
  making GraLoRA a scalable and robust solution for PEFT. Code, data, and scripts
  are available at https://github.com/SqueezeBits/GraLoRA.git
paper_authors._id: 683674c419543f12e85c4f47
paper_authors.name: Yeonjoon Jung
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:57:44.292Z
paper_authors._id: 66a8ba3b29470b614485db2e
paper_authors.avatarUrl: /avatars/64d855b18df75b35eaed35b4b9282b78.svg
paper_authors.fullname: Yeonjoon Jung
paper_authors.type: user
paper_authors.user: yeonjoon-jung
paper_authors._id: 683674c419543f12e85c4f48
paper_authors.name: Daehyun Ahn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:22:24.944Z
paper_authors._id: 64f58fd1d72303b303098198
paper_authors.avatarUrl: /avatars/e45839a71445c9bffb4001f59241d931.svg
paper_authors.fullname: Daehyun Ahn
paper_authors.type: user
paper_authors.user: daehyunahn
paper_authors._id: 683674c419543f12e85c4f49
paper_authors.name: Hyungjun Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:22:39.658Z
paper_authors._id: 671f5c7bd79a70b18f7db600
paper_authors.avatarUrl: /avatars/80f6eaf612893f66c90c4e977f45483c.svg
paper_authors.fullname: Hyungjun Kim
paper_authors.type: user
paper_authors.user: HyungjunKim
paper_authors._id: 683674c419543f12e85c4f4a
paper_authors.name: Taesu Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:22:59.898Z
paper_authors._id: 64dc388069d21c567b0794b0
paper_authors.avatarUrl: /avatars/795ab1b7f970df3b589f6c1ca75df9ad.svg
paper_authors.fullname: Taesu Kim
paper_authors.type: user
paper_authors.user: tae-su-kim
paper_authors._id: 683674c419543f12e85c4f4b
paper_authors.name: Eunhyeok Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:23:05.667Z
paper_authors._id: 670a70e95f4ed2ff1f6d02ed
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1V2NN6G6w-JBWRsVYW2Ed.png
paper_authors.fullname: Eunhyeok Park
paper_authors.type: user
paper_authors.user: hyek90
paper_title: GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient
    Fine-Tuning
paper_summary: Low-Rank Adaptation (LoRA) is a popular method for parameter-efficient
  fine-tuning (PEFT) of generative models, valued for its simplicity and
  effectiveness. Despite recent enhancements, LoRA still suffers from a
  fundamental limitation: overfitting when the bottleneck is widened. It performs
  best at ranks 32-64, yet its accuracy stagnates or declines at higher ranks,
  still falling short of full fine-tuning (FFT) performance. We identify the root
  cause as LoRA's structural bottleneck, which introduces gradient entanglement
  to the unrelated input channels and distorts gradient propagation. To address
  this, we introduce a novel structure, Granular Low-Rank Adaptation (GraLoRA)
  that partitions weight matrices into sub-blocks, each with its own low-rank
  adapter. With negligible computational or storage cost, GraLoRA overcomes
  LoRA's limitations, effectively increases the representational capacity, and
  more closely approximates FFT behavior. Experiments on code generation and
  commonsense reasoning benchmarks show that GraLoRA consistently outperforms
  LoRA and other baselines, achieving up to +8.5% absolute gain in Pass@1 on
  HumanEval+. These improvements hold across model sizes and rank settings,
  making GraLoRA a scalable and robust solution for PEFT. Code, data, and scripts
  are available at https://github.com/SqueezeBits/GraLoRA.git
paper_ai_summary: Granular Low-Rank Adaptation (GraLoRA) improves upon Low-Rank Adaptation (LoRA) by partitioning weight matrices to mitigate overfitting and enhance performance in parameter-efficient fine-tuning.
paper_ai_keywords: Low-Rank Adaptation LoRA parameter-efficient fine-tuning PEFT full fine-tuning FFT gradient entanglement Granular Low-Rank Adaptation GraLoRA weight matrices sub-blocks low-rank adapter Pass@1 HumanEval+