MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
Low-rank adaptation is a popular parameter-efficient fine-tuning method for
large language models. In this paper, we analyze the impact of low-rank
updating, as implemented in LoRA. Our findings suggest that the low-rank
updating mechanism may limit the ability of LLMs to effectively learn and
memorize new knowledge. Inspired by this observation, we propose a new method
called MoRA, which employs a square matrix to achieve high-rank updating while
maintaining the same number of trainable parameters. To achieve it, we
introduce the corresponding non-parameter operators to reduce the input
dimension and increase the output dimension for the square matrix. Furthermore,
these operators ensure that the weight can be merged back into LLMs, which
makes our method can be deployed like LoRA. We perform a comprehensive
evaluation of our method across five tasks: instruction tuning, mathematical
reasoning, continual pretraining, memory and pretraining. Our method
outperforms LoRA on memory-intensive tasks and achieves comparable performance
on other tasks.
664c11e7a8f6e1c87611edab
Ting Jiang
664c11e7a8f6e1c87611edac
Shaohan Huang
admin_assigned
2024-05-21T07:44:28.645Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
664c11e7a8f6e1c87611edad
Shengyue Luo
admin_assigned
2024-05-21T07:44:40.415Z
6405c4c091ee7d7f2873ed93
/avatars/a2ece07508da5a2a794e9745036f2767.svg
shengyue
user
luoshengyue
664c11e7a8f6e1c87611edae
Zihan Zhang
664c11e7a8f6e1c87611edaf
Haizhen Huang
admin_assigned
2024-05-21T07:45:42.158Z
6302e8e57e137e3d6b3f5a06
/avatars/aa82fe4fb1212dc120ecccd406568c5d.svg
Huang
user
Haizhen
664c11e7a8f6e1c87611edb0
Furu Wei
admin_assigned
2024-05-21T07:45:49.085Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
664c11e7a8f6e1c87611edb1
Weiwei Deng
admin_assigned
2024-05-21T07:45:54.921Z
642da49dc1adada5b345c72b
/avatars/5fa8d345a7248246454444999756105b.svg
deng
user
weiweideng
664c11e7a8f6e1c87611edb2
Feng Sun
admin_assigned
2024-05-21T07:46:01.176Z
64b88a7449bde5d9481e4755
/avatars/45a963517ec2c54c81ee80449d655014.svg
feng sun
user
forserty
664c11e7a8f6e1c87611edb3
Qi Zhang
664c11e7a8f6e1c87611edb4
Deqing Wang
664c11e7a8f6e1c87611edb5
Fuzhen Zhuang
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
Low-rank adaptation is a popular parameter-efficient fine-tuning method for
large language models. In this paper, we analyze the impact of low-rank
updating, as implemented in LoRA. Our findings suggest that the low-rank
updating mechanism may limit the ability of LLMs to effectively learn and
memorize new knowledge. Inspired by this observation, we propose a new method
called MoRA, which employs a square matrix to achieve high-rank updating while
maintaining the same number of trainable parameters. To achieve it, we
introduce the corresponding non-parameter operators to reduce the input
dimension and increase the output dimension for the square matrix. Furthermore,
these operators ensure that the weight can be merged back into LLMs, which
makes our method can be deployed like LoRA. We perform a comprehensive
evaluation of our method across five tasks: instruction tuning, mathematical
reasoning, continual pretraining, memory and pretraining. Our method
outperforms LoRA on memory-intensive tasks and achieves comparable performance
on other tasks.
MoRA, a high-rank updating method using square matrices, enhances the ability of large language models to learn and memorize new knowledge, especially in memory-intensive tasks, compared to LoRA.
low-rank adaptation parameter-efficient fine-tuning large language models low-rank updating LoRA square matrix high-rank updating non-parameter operators instruction tuning mathematical reasoning continual pretraining memory pretraining