title: Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and
    Mixture-of-Experts Optimization Alignment
summary: While Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning for
  Large Language Models (LLMs), its performance often falls short of Full
  Fine-Tuning (Full FT). Current methods optimize LoRA by initializing with
  static singular value decomposition (SVD) subsets, leading to suboptimal
  leveraging of pre-trained knowledge. Another path for improving LoRA is
  incorporating a Mixture-of-Experts (MoE) architecture. However, weight
  misalignment and complex gradient dynamics make it challenging to adopt SVD
  prior to the LoRA MoE architecture. To mitigate these issues, we propose
  Great LoRA Mixture-of-Expert
  (GOAT), a framework that (1) adaptively integrates relevant priors using an
  SVD-structured MoE, and (2) aligns optimization with full fine-tuned MoE by
  deriving a theoretical scaling factor. We demonstrate that proper scaling,
  without modifying the architecture or training algorithms, boosts LoRA MoE's
  efficiency and performance. Experiments across 25 datasets, including natural
  language understanding, commonsense reasoning, image classification, and
  natural language generation, demonstrate GOAT's state-of-the-art performance,
  closing the gap with Full FT.
paper_authors._id: 67bd396ea06bae99f3866911
paper_authors.name: Chenghao Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-25T09:39:38.942Z
paper_authors._id: 641aa5e391e3376a057bbd4c
paper_authors.avatarUrl: /avatars/5818797f27444fde078b503774ee081c.svg
paper_authors.fullname: Chenghao Fan
paper_authors.type: user
paper_authors.user: Facico
paper_authors._id: 67bd396ea06bae99f3866912
paper_authors.name: Zhenyi Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:22:55.129Z
paper_authors._id: 666b0a01e6cfd60425d00fd9
paper_authors.avatarUrl: /avatars/f6e5447e95785563e850ffcbe7dd6e3d.svg
paper_authors.fullname: LUZHENYI
paper_authors.type: user
paper_authors.user: LUzhenyi123111
paper_authors._id: 67bd396ea06bae99f3866913
paper_authors.name: Sichen Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:37:56.904Z
paper_authors._id: 641dbda28dc52733fa4419cf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641dbda28dc52733fa4419cf/vdIsa6UlMIaqHinGrYDb-.png
paper_authors.fullname: Sichen Liu
paper_authors.type: user
paper_authors.user: Seas0
paper_authors._id: 67bd396ea06bae99f3866914
paper_authors.name: Xiaoye Qu
paper_authors._id: 67bd396ea06bae99f3866915
paper_authors.name: Wei Wei
paper_authors._id: 67bd396ea06bae99f3866916
paper_authors.name: Chengfeng Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:23:29.392Z
paper_authors._id: 675fc6bd34f2e646c06fbb07
paper_authors.avatarUrl: /avatars/60f77c237cd652f80b7f1ecfd358afe1.svg
paper_authors.fullname: gu chengfeng
paper_authors.type: user
paper_authors.user: gucf
paper_authors._id: 67bd396ea06bae99f3866917
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:53:02.137Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_title: Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and
    Mixture-of-Experts Optimization Alignment
paper_summary: While Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning for
  Large Language Models (LLMs), its performance often falls short of Full
  Fine-Tuning (Full FT). Current methods optimize LoRA by initializing with
  static singular value decomposition (SVD) subsets, leading to suboptimal
  leveraging of pre-trained knowledge. Another path for improving LoRA is
  incorporating a Mixture-of-Experts (MoE) architecture. However, weight
  misalignment and complex gradient dynamics make it challenging to adopt SVD
  prior to the LoRA MoE architecture. To mitigate these issues, we propose
  Great LoRA Mixture-of-Expert
  (GOAT), a framework that (1) adaptively integrates relevant priors using an
  SVD-structured MoE, and (2) aligns optimization with full fine-tuned MoE by
  deriving a theoretical scaling factor. We demonstrate that proper scaling,
  without modifying the architecture or training algorithms, boosts LoRA MoE's
  efficiency and performance. Experiments across 25 datasets, including natural
  language understanding, commonsense reasoning, image classification, and
  natural language generation, demonstrate GOAT's state-of-the-art performance,
  closing the gap with Full FT.
paper_ai_summary: GOAT, a framework that integrates relevant priors using an SVD-structured Mixture-of-Experts and derives a theoretical scaling factor, enhances LoRA MoE's efficiency and performance, approaching Full Fine-Tuning results.
paper_ai_keywords: Low-Rank Adaptation LoRA Parameter-efficient fine-tuning Large Language Models Full Fine-Tuning Singular value decomposition SVD Mixture-of-Experts MoE Weight misalignment Gradient dynamics SVD-structured MoE Theoretical scaling factor