title: GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs
summary: Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and
  democratized the usage of Large Language Models (LLMs). Recent studies have
  shown that a small subset of weights significantly impacts performance. Based
  on this observation, we introduce a novel PEFT method, called Gaussian noise
  Injected Fine Tuning of Salient Weights (GIFT-SW). Our method updates only
  salient columns, while injecting Gaussian noise into non-salient ones. To
  identify these columns, we developeda generalized sensitivity metric that
  extends and unifies metrics from previous studies. Experiments with LLaMA
  models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT
  methods under the same computational budget. Moreover, GIFT-SW offers practical
  advantages to recover performance of models subjected to mixed-precision
  quantization with keeping salient weights in full precision.
paper_authors._id: 66d44c5c8a438492b073a251
paper_authors.name: Maxim Zhelnin
paper_authors._id: 66d44c5c8a438492b073a252
paper_authors.name: Viktor Moskvoretskii
paper_authors._id: 66d44c5c8a438492b073a253
paper_authors.name: Egor Shvetsov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-02T07:17:05.907Z
paper_authors._id: 65afde6ba0b4bf3b0e95b4e8
paper_authors.avatarUrl: /avatars/e9b97040b0a619bf6609465d1678705c.svg
paper_authors.fullname: Egor Shvetsov
paper_authors.type: user
paper_authors.user: dalime
paper_authors._id: 66d44c5c8a438492b073a254
paper_authors.name: Egor Venediktov
paper_authors._id: 66d44c5c8a438492b073a255
paper_authors.name: Mariya Krylova
paper_authors._id: 66d44c5c8a438492b073a256
paper_authors.name: Aleksandr Zuev
paper_authors._id: 66d44c5c8a438492b073a257
paper_authors.name: Evgeny Burnaev
paper_title: GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs
paper_summary: Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and
  democratized the usage of Large Language Models (LLMs). Recent studies have
  shown that a small subset of weights significantly impacts performance. Based
  on this observation, we introduce a novel PEFT method, called Gaussian noise
  Injected Fine Tuning of Salient Weights (GIFT-SW). Our method updates only
  salient columns, while injecting Gaussian noise into non-salient ones. To
  identify these columns, we developeda generalized sensitivity metric that
  extends and unifies metrics from previous studies. Experiments with LLaMA
  models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT
  methods under the same computational budget. Moreover, GIFT-SW offers practical
  advantages to recover performance of models subjected to mixed-precision
  quantization with keeping salient weights in full precision.
paper_ai_summary: GIFT-SW, a new PEFT method, fine-tunes only salient weights and injects Gaussian noise into non-salient ones, outperforming full fine-tuning and other PEFT methods while maintaining model performance post-quantization.
paper_ai_keywords: Parameter Efficient Fine-Tuning PEFT Large Language Models LLMs Gaussian noise salient weights generalized sensitivity metric full fine-tuning mixed-precision quantization