PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning
<|reference_start|>PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning: Low-rank adaptation (LoRA) and its variants have recently gained much interest due to their ability to avoid excessive inference costs. However, LoRA still encounters the following challenges: (1) Limitation of low-rank assumption; and (2) Its initialization method may be suboptimal. To this end, we propose PMSS(Pre-trained Matrices Skeleton Selection), which enables high-rank updates with low costs while leveraging semantic and linguistic information inherent in pre-trained weight. It achieves this by selecting skeletons from the pre-trained weight matrix and only learning a small matrix instead. Experiments demonstrate that PMSS outperforms LoRA and other fine-tuning methods across tasks with much less trainable parameters. We demonstrate its effectiveness, especially in handling complex tasks such as DROP benchmark(+3.4%/+5.9% on LLaMA2-7B/13B) and math reasoning(+12.89%/+5.61%/+3.11% on LLaMA2-7B, Mistral-7B and Gemma-7B of GSM8K). The code and model will be released soon.<|reference_end|>
@article{wang2024pmss:,
    title={PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning},
    author={Qibin Wang, Xiaolin Hu, Weikai Xu, Wei Liu, Jian Luan, Bin Wang},
    journal={arXiv preprint arXiv:2409.16722},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.16722},
    primaryClass={cs.CL cs.LG}
}