title: MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
summary: Low-rank adaptation is a popular parameter-efficient fine-tuning method for
  large language models. In this paper, we analyze the impact of low-rank
  updating, as implemented in LoRA. Our findings suggest that the low-rank
  updating mechanism may limit the ability of LLMs to effectively learn and
  memorize new knowledge. Inspired by this observation, we propose a new method
  called MoRA, which employs a square matrix to achieve high-rank updating while
  maintaining the same number of trainable parameters. To achieve it, we
  introduce the corresponding non-parameter operators to reduce the input
  dimension and increase the output dimension for the square matrix. Furthermore,
  these operators ensure that the weight can be merged back into LLMs, which
  makes our method can be deployed like LoRA. We perform a comprehensive
  evaluation of our method across five tasks: instruction tuning, mathematical
  reasoning, continual pretraining, memory and pretraining. Our method
  outperforms LoRA on memory-intensive tasks and achieves comparable performance
  on other tasks.
paper_authors._id: 664c11e7a8f6e1c87611edab
paper_authors.name: Ting Jiang
paper_authors._id: 664c11e7a8f6e1c87611edac
paper_authors.name: Shaohan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T07:44:28.645Z
paper_authors._id: 632bd2f72d6a805eeb4bc601
paper_authors.avatarUrl: /avatars/6e1533e8a599f3068290aa69ac82cab7.svg
paper_authors.fullname: HUANG SHAOHAN
paper_authors.type: user
paper_authors.user: buaahsh
paper_authors._id: 664c11e7a8f6e1c87611edad
paper_authors.name: Shengyue Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T07:44:40.415Z
paper_authors._id: 6405c4c091ee7d7f2873ed93
paper_authors.avatarUrl: /avatars/a2ece07508da5a2a794e9745036f2767.svg
paper_authors.fullname: shengyue
paper_authors.type: user
paper_authors.user: luoshengyue
paper_authors._id: 664c11e7a8f6e1c87611edae
paper_authors.name: Zihan Zhang
paper_authors._id: 664c11e7a8f6e1c87611edaf
paper_authors.name: Haizhen Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T07:45:42.158Z
paper_authors._id: 6302e8e57e137e3d6b3f5a06
paper_authors.avatarUrl: /avatars/aa82fe4fb1212dc120ecccd406568c5d.svg
paper_authors.fullname: Huang
paper_authors.type: user
paper_authors.user: Haizhen
paper_authors._id: 664c11e7a8f6e1c87611edb0
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T07:45:49.085Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_authors._id: 664c11e7a8f6e1c87611edb1
paper_authors.name: Weiwei Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T07:45:54.921Z
paper_authors._id: 642da49dc1adada5b345c72b
paper_authors.avatarUrl: /avatars/5fa8d345a7248246454444999756105b.svg
paper_authors.fullname: deng
paper_authors.type: user
paper_authors.user: weiweideng
paper_authors._id: 664c11e7a8f6e1c87611edb2
paper_authors.name: Feng Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T07:46:01.176Z
paper_authors._id: 64b88a7449bde5d9481e4755
paper_authors.avatarUrl: /avatars/45a963517ec2c54c81ee80449d655014.svg
paper_authors.fullname: feng sun
paper_authors.type: user
paper_authors.user: forserty
paper_authors._id: 664c11e7a8f6e1c87611edb3
paper_authors.name: Qi Zhang
paper_authors._id: 664c11e7a8f6e1c87611edb4
paper_authors.name: Deqing Wang
paper_authors._id: 664c11e7a8f6e1c87611edb5
paper_authors.name: Fuzhen Zhuang
paper_title: MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
paper_summary: Low-rank adaptation is a popular parameter-efficient fine-tuning method for
  large language models. In this paper, we analyze the impact of low-rank
  updating, as implemented in LoRA. Our findings suggest that the low-rank
  updating mechanism may limit the ability of LLMs to effectively learn and
  memorize new knowledge. Inspired by this observation, we propose a new method
  called MoRA, which employs a square matrix to achieve high-rank updating while
  maintaining the same number of trainable parameters. To achieve it, we
  introduce the corresponding non-parameter operators to reduce the input
  dimension and increase the output dimension for the square matrix. Furthermore,
  these operators ensure that the weight can be merged back into LLMs, which
  makes our method can be deployed like LoRA. We perform a comprehensive
  evaluation of our method across five tasks: instruction tuning, mathematical
  reasoning, continual pretraining, memory and pretraining. Our method
  outperforms LoRA on memory-intensive tasks and achieves comparable performance
  on other tasks.
paper_ai_summary: MoRA, a high-rank updating method using square matrices, enhances the ability of large language models to learn and memorize new knowledge, especially in memory-intensive tasks, compared to LoRA.
paper_ai_keywords: low-rank adaptation parameter-efficient fine-tuning large language models low-rank updating LoRA square matrix high-rank updating non-parameter operators instruction tuning mathematical reasoning continual pretraining memory pretraining