Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape
<|reference_start|>Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape: Fine-tuning large-scale pre-trained models is prohibitively expensive in terms of computational and memory costs. Low-Rank Adaptation (LoRA), a popular Parameter-Efficient Fine-Tuning (PEFT) method, provides an efficient way to fine-tune models by optimizing only a low-rank matrix. Despite recent progress made in improving LoRA's performance, the connection between the LoRA optimization space and the original full parameter space is often overlooked. A solution that appears flat in the LoRA space may exist sharp directions in the full parameter space, potentially harming generalization performance. In this paper, we propose Flat-LoRA, an efficient approach that seeks a low-rank adaptation located in a flat region of the full parameter space.Instead of relying on the well-established sharpness-aware minimization approach, which can incur significant computational and memory burdens, we utilize random weight perturbation with a Bayesian expectation loss objective to maintain training efficiency and design a refined perturbation generation strategy for improved performance. Experiments on natural language processing and image classification tasks with various architectures demonstrate the effectiveness of our approach.<|reference_end|>
@article{li2024flat-lora:,
    title={Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape},
    author={Tao Li, Zhengbao He, Yujun Li, Yasheng Wang, Lifeng Shang, Xiaolin
  Huang},
    journal={arXiv preprint arXiv:2409.14396},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.14396},
    primaryClass={cs.LG}
}