title: VeRA: Vector-based Random Matrix Adaptation
summary: Low-rank adapation (LoRA) is a popular method that reduces the number of
  trainable parameters when finetuning large language models, but still faces
  acute storage challenges when scaling to even larger models or deploying
  numerous per-user or per-task adapted models. In this work, we present
  Vector-based Random Matrix Adaptation (VeRA), which reduces the number of
  trainable parameters by 10x compared to LoRA, yet maintains the same
  performance. It achieves this by using a single pair of low-rank matrices
  shared across all layers and learning small scaling vectors instead. We
  demonstrate its effectiveness on the GLUE and E2E benchmarks, and show its
  application in instruction-following with just 1.4M parameters using the Llama2
  7B model.
paper_authors._id: 652f5762660516114945795c
paper_authors.name: Dawid Jan Kopiczko
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-10-18T10:50:02.300Z
paper_authors._id: 637de0fa1342ba1762422495
paper_authors.avatarUrl: /avatars/b1cce386a8b33007fc381fcbfc5cdc9a.svg
paper_authors.fullname: Dawid
paper_authors.type: user
paper_authors.user: dakopi
paper_authors._id: 652f5762660516114945795d
paper_authors.name: Tijmen Blankevoort
paper_authors._id: 652f5762660516114945795e
paper_authors.name: Yuki Markus Asano
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-18T09:07:54.798Z
paper_authors._id: 637d21239a5217b88b7549c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637d21239a5217b88b7549c3/LrIGPiva5VGVZG87rTAJz.jpeg
paper_authors.fullname: Yuki Asano
paper_authors.type: user
paper_authors.user: yukimasano
paper_title: VeRA: Vector-based Random Matrix Adaptation
paper_summary: Low-rank adapation (LoRA) is a popular method that reduces the number of
  trainable parameters when finetuning large language models, but still faces
  acute storage challenges when scaling to even larger models or deploying
  numerous per-user or per-task adapted models. In this work, we present
  Vector-based Random Matrix Adaptation (VeRA), which reduces the number of
  trainable parameters by 10x compared to LoRA, yet maintains the same
  performance. It achieves this by using a single pair of low-rank matrices
  shared across all layers and learning small scaling vectors instead. We
  demonstrate its effectiveness on the GLUE and E2E benchmarks, and show its
  application in instruction-following with just 1.4M parameters using the Llama2
  7B model.
paper_ai_keywords: Low-rank adapation (LoRA) Vector-based Random Matrix Adaptation (VeRA) low-rank matrices scaling vectors GLUE benchmark E2E benchmark instruction-following