SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Values
<|reference_start|>SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Values: Large pre-trained models (LPMs) have demonstrated exceptional performance in diverse natural language processing and computer vision tasks. However, fully fine-tuning these models poses substantial memory challenges, particularly in resource-constrained environments. Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, mitigate this issue by adjusting only a small subset of parameters. Nevertheless, these methods typically employ random initialization for low-rank matrices, which can lead to inefficiencies in gradient descent and diminished generalizability due to suboptimal starting points. To address these limitations, we propose SVFit, a novel PEFT approach that leverages singular value decomposition (SVD) to initialize low-rank matrices using critical singular values as trainable parameters. Specifically, SVFit performs SVD on the pre-trained weight matrix to obtain the best rank-r approximation matrix, emphasizing the most critical singular values that capture over 99% of the matrix's information. These top-r singular values are then used as trainable parameters to scale the fundamental subspaces of the matrix, facilitating rapid domain adaptation. Extensive experiments across various pre-trained models in natural language understanding, text-to-image generation, and image classification tasks reveal that SVFit outperforms LoRA while requiring 16 times fewer trainable parameters.<|reference_end|>
@article{sun2024svfit:,
    title={SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using
  Singular Values},
    author={Chengwei Sun, Jiwei Wei, Yujia Wu, Yiming Shi, Shiyuan He, Zeyu Ma,
  Ning Xie, Yang Yang},
    journal={arXiv preprint arXiv:2409.05926},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.05926},
    primaryClass={cs.LG cs.CL}
}