MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuning
<|reference_start|>MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuning: In response to the challenges posed by the extensive parameter updates required for full fine-tuning of large-scale pre-trained models, parameter-efficient fine-tuning (PEFT) methods, exemplified by Low-Rank Adaptation (LoRA), have emerged. LoRA simplifies the fine-tuning process but may still struggle with a certain level of redundancy in low-rank matrices and limited effectiveness from merely increasing their rank. To address these issues, a natural idea is to enhance the independence and diversity of the learning process for the low-rank matrices. Therefore, we propose Masked LoRA Experts (MLAE), an innovative approach that applies the concept of masking to visual PEFT. Our method incorporates a cellular decomposition strategy that transforms a low-rank matrix into independent rank-1 submatrices, or "experts", thus enhancing independence. Additionally, we introduce a binary mask matrix that selectively activates these experts during training to promote more diverse and anisotropic learning, based on expert-level dropout strategies. Our investigations reveal that this selective activation not only enhances performance but also fosters a more diverse acquisition of knowledge with a marked decrease in parameter similarity among MLAE, significantly boosting the quality of the model. Remarkably, MLAE achieves new state-of-the-art (SOTA) performance with an average accuracy score of 78.8% on the VTAB-1k benchmark and 90.9% on the FGVC benchmark, surpassing the previous SOTA result by an average of 0.8% on both benchmarks with approximately half parameters. Our code is available at https://github.com/jie040109/MLAE.<|reference_end|>
@article{wang2024mlae:,
    title={MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuning},
    author={Junjie Wang, Guangjing Yang, Wentao Chen, Huahui Yi, Xiaohu Wu,
  Zhouchen Lin, Qicheng Lao},
    journal={arXiv preprint arXiv:2405.18897},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.18897},
    primaryClass={cs.CV}
}