GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs
<|reference_start|>GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs: Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and democratized the usage of Large Language Models (LLMs). Recent studies have shown that a small subset of weights significantly impacts performance. Based on this observation, we introduce a novel PEFT method, called Gaussian noise Injected Fine Tuning of Salient Weights (GIFT-SW). Our method updates only salient columns, while injecting Gaussian noise into non-salient ones. To identify these columns, we developeda generalized sensitivity metric that extends and unifies metrics from previous studies. Experiments with LLaMA models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT methods under the same computational budget. Moreover, GIFT-SW offers practical advantages to recover performance of models subjected to mixed-precision quantization with keeping salient weights in full precision.<|reference_end|>
@article{zhelnin2024gift-sw:,
    title={GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs},
    author={Maxim Zhelnin, Viktor Moskvoretskii, Egor Shvetsov, Egor Venediktov,
  Mariya Krylova, Aleksandr Zuev, Evgeny Burnaev},
    journal={arXiv preprint arXiv:2408.15300},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.15300},
    primaryClass={cs.LG cs.AI}
}