VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks
<|reference_start|>VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks: As the adoption of large language models increases and the need for per-user or per-task model customization grows, the parameter-efficient fine-tuning (PEFT) methods, such as low-rank adaptation (LoRA) and its variants, incur substantial storage and transmission costs. To further reduce stored parameters, we introduce a "divide-and-share" paradigm that breaks the barriers of low-rank decomposition across matrix dimensions, modules and layers by sharing parameters globally via a vector bank. As an instantiation of the paradigm to LoRA, our proposed VB-LoRA composites all the low-rank matrices of LoRA from a shared vector bank with a differentiable top-$k$ admixture module. VB-LoRA achieves extreme parameter efficiency while maintaining comparable or better performance compared to state-of-the-art PEFT methods. Extensive experiments demonstrate the effectiveness of VB-LoRA on natural language understanding, natural language generation, and instruction tuning tasks. When fine-tuning the Llama2-13B model, VB-LoRA only uses 0.4% of LoRA's stored parameters, yet achieves superior results. Our source code is available at https://github.com/leo-yangli/VB-LoRA.<|reference_end|>
@article{li2024vb-lora:,
    title={VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks},
    author={Yang Li, Shaobo Han, Shihao Ji},
    journal={arXiv preprint arXiv:2405.15179},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.15179},
    primaryClass={cs.CL}
}