SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors
<|reference_start|>SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors: Popular parameter-efficient fine-tuning (PEFT) methods, such as LoRA and its variants, freeze pre-trained model weights \(W\) and inject learnable matrices \(\Delta W\). These \(\Delta W\) matrices are structured for efficient parameterization, often using techniques like low-rank approximations or scaling vectors. However, these methods typically show a performance gap compared to full fine-tuning. Although recent PEFT methods have narrowed this gap, they do so at the cost of additional learnable parameters. We propose SVFT, a simple approach that fundamentally differs from existing methods: the structure imposed on \(\Delta W\) depends on the specific weight matrix \(W\). Specifically, SVFT updates \(W\) as a sparse combination of outer products of its singular vectors, training only the coefficients (scales) of these sparse combinations. This approach allows fine-grained control over expressivity through the number of coefficients. Extensive experiments on language and vision benchmarks show that SVFT recovers up to 96% of full fine-tuning performance while training only 0.006 to 0.25% of parameters, outperforming existing methods that only recover up to 85% performance using 0.03 to 0.8% of the trainable parameter budget.<|reference_end|>
@article{lingam2024svft:,
    title={SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors},
    author={Vijay Lingam, Atula Tejaswi, Aditya Vavre, Aneesh Shetty, Gautham
  Krishna Gudur, Joydeep Ghosh, Alex Dimakis, Eunsol Choi, Aleksandar
  Bojchevski, Sujay Sanghavi},
    journal={arXiv preprint arXiv:2405.19597},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.19597},
    primaryClass={cs.LG cs.AI cs.CL}
}