title: LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for
    Parameter-Efficient Fine-Tuning
summary: The rapid growth of model scale has necessitated substantial computational
  resources for fine-tuning. Existing approach such as Low-Rank Adaptation (LoRA)
  has sought to address the problem of handling the large updated parameters in
  full fine-tuning. However, LoRA utilize random initialization and optimization
  of low-rank matrices to approximate updated weights, which can result in
  suboptimal convergence and an accuracy gap compared to full fine-tuning. To
  address these issues, we propose LoLDU, a Parameter-Efficient Fine-Tuning
  (PEFT) approach that significantly reduces trainable parameters by 2600 times
  compared to regular PEFT methods while maintaining comparable performance.
  LoLDU leverages Lower-Diag-Upper Decomposition (LDU) to initialize low-rank
  matrices for faster convergence and orthogonality. We focus on optimizing the
  diagonal matrix for scaling transformations. To the best of our knowledge,
  LoLDU has the fewest parameters among all PEFT approaches. We conducted
  extensive experiments across 4 instruction-following datasets, 6 natural
  language understanding (NLU) datasets, 8 image classification datasets, and
  image generation datasets with multiple model types (LLaMA2, RoBERTa, ViT, and
  Stable Diffusion), providing a comprehensive and detailed analysis. Our
  open-source code can be accessed at
  https://github.com/SKDDJ/LoLDU{https://github.com/SKDDJ/LoLDU}.
paper_authors._id: 6711dc54b265bbd413b8ec7b
paper_authors.name: Yiming Shi
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-18T04:03:46.223Z
paper_authors._id: 642ebcefd09a9c63c2124bd2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642ebcefd09a9c63c2124bd2/HFSkoaxIM6nTq1CrY6jyF.jpeg
paper_authors.fullname: Yiming Shi
paper_authors.type: user
paper_authors.user: Shiym
paper_authors._id: 6711dc54b265bbd413b8ec7c
paper_authors.name: Jiwei Wei
paper_authors._id: 6711dc54b265bbd413b8ec7d
paper_authors.name: Yujia Wu
paper_authors._id: 6711dc54b265bbd413b8ec7e
paper_authors.name: Ran Ran
paper_authors._id: 6711dc54b265bbd413b8ec7f
paper_authors.name: Chengwei Sun
paper_authors._id: 6711dc54b265bbd413b8ec80
paper_authors.name: Shiyuan He
paper_authors._id: 6711dc54b265bbd413b8ec81
paper_authors.name: Yang Yang
paper_title: LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for
    Parameter-Efficient Fine-Tuning
paper_summary: The rapid growth of model scale has necessitated substantial computational
  resources for fine-tuning. Existing approach such as Low-Rank Adaptation (LoRA)
  has sought to address the problem of handling the large updated parameters in
  full fine-tuning. However, LoRA utilize random initialization and optimization
  of low-rank matrices to approximate updated weights, which can result in
  suboptimal convergence and an accuracy gap compared to full fine-tuning. To
  address these issues, we propose LoLDU, a Parameter-Efficient Fine-Tuning
  (PEFT) approach that significantly reduces trainable parameters by 2600 times
  compared to regular PEFT methods while maintaining comparable performance.
  LoLDU leverages Lower-Diag-Upper Decomposition (LDU) to initialize low-rank
  matrices for faster convergence and orthogonality. We focus on optimizing the
  diagonal matrix for scaling transformations. To the best of our knowledge,
  LoLDU has the fewest parameters among all PEFT approaches. We conducted
  extensive experiments across 4 instruction-following datasets, 6 natural
  language understanding (NLU) datasets, 8 image classification datasets, and
  image generation datasets with multiple model types (LLaMA2, RoBERTa, ViT, and
  Stable Diffusion), providing a comprehensive and detailed analysis. Our
  open-source code can be accessed at
  https://github.com/SKDDJ/LoLDU{https://github.com/SKDDJ/LoLDU}.
paper_ai_summary: LoLDU, a Parameter-Efficient Fine-Tuning approach using Lower-Diag-Upper Decomposition, significantly reduces trainable parameters while maintaining performance across various datasets and models.
paper_ai_keywords: Low-Rank Adaptation (LoRA) Parameter-Efficient Fine-Tuning (PEFT) Lower-Diag-Upper Decomposition (LDU) instruction-following datasets natural language understanding (NLU) datasets image classification datasets image generation datasets LLaMA2 RoBERTa ViT Stable Diffusion