title: KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models
summary: The increasing sizes of large language models (LLMs) result in significant
  computational overhead and memory usage when adapting these models to specific
  tasks or domains. Various parameter-efficient fine-tuning (PEFT) methods have
  been devised to mitigate these challenges by training a small set of parameters
  for the task-specific updates of the model weights. Among PEFT methods, LoRA
  stands out for its simplicity and efficiency, inspiring the development of a
  series of variants. However, LoRA and its successors disregard the knowledge
  that is noisy or irrelevant to the targeted task, detrimentally impacting model
  performance and leading to suboptimality. To address this limitation, we
  introduce Knowledge-aware Singular-value Adaptation (KaSA), a PEFT method that
  leverages singular value decomposition (SVD) with knowledge-aware singular
  values to dynamically activate knowledge based on its relevance to the task at
  hand. We conduct extensive experiments across a range of LLMs on tasks spanning
  natural language understanding (NLU), generation (NLG), instruction following,
  and commonsense reasoning. The experimental results demonstrate that KaSA
  consistently outperforms FFT and 14 popular PEFT baselines across 16 benchmarks
  and 4 synthetic datasets, underscoring our method's efficacy and adaptability.
  The source code of our method is available at
  https://github.com/juyongjiang/KaSA.
paper_authors._id: 675ad3a39d2eead3eb196816
paper_authors.name: Fan Wang
paper_authors._id: 675ad3a39d2eead3eb196817
paper_authors.name: Juyong Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:37:01.554Z
paper_authors._id: 64b63712b3b69063d9aff341
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b63712b3b69063d9aff341/nP6GyBKBtnWWoA0TvQF3K.jpeg
paper_authors.fullname: John
paper_authors.type: user
paper_authors.user: juyongjiang
paper_authors._id: 675ad3a39d2eead3eb196818
paper_authors.name: Chansung Park
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-12T13:40:54.230Z
paper_authors._id: 60d3b57ad7b174177faabd6e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1659971187637-60d3b57ad7b174177faabd6e.jpeg
paper_authors.fullname: chansung park
paper_authors.type: user
paper_authors.user: chansung
paper_authors._id: 675ad3a39d2eead3eb196819
paper_authors.name: Sunghun Kim
paper_authors._id: 675ad3a39d2eead3eb19681a
paper_authors.name: Jing Tang
paper_title: KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models
paper_summary: The increasing sizes of large language models (LLMs) result in significant
  computational overhead and memory usage when adapting these models to specific
  tasks or domains. Various parameter-efficient fine-tuning (PEFT) methods have
  been devised to mitigate these challenges by training a small set of parameters
  for the task-specific updates of the model weights. Among PEFT methods, LoRA
  stands out for its simplicity and efficiency, inspiring the development of a
  series of variants. However, LoRA and its successors disregard the knowledge
  that is noisy or irrelevant to the targeted task, detrimentally impacting model
  performance and leading to suboptimality. To address this limitation, we
  introduce Knowledge-aware Singular-value Adaptation (KaSA), a PEFT method that
  leverages singular value decomposition (SVD) with knowledge-aware singular
  values to dynamically activate knowledge based on its relevance to the task at
  hand. We conduct extensive experiments across a range of LLMs on tasks spanning
  natural language understanding (NLU), generation (NLG), instruction following,
  and commonsense reasoning. The experimental results demonstrate that KaSA
  consistently outperforms FFT and 14 popular PEFT baselines across 16 benchmarks
  and 4 synthetic datasets, underscoring our method's efficacy and adaptability.
  The source code of our method is available at
  https://github.com/juyongjiang/KaSA.
paper_ai_keywords: large language models (LLMs) parameter-efficient fine-tuning (PEFT) LoRA Knowledge-aware Singular-value Adaptation (KaSA) singular value decomposition (SVD) singular values natural-language understanding (NLU) natural-language generation (NLG) commonsense reasoning benchmarks synthetic datasets