title: VeRA: Vector-based Random Matrix Adaptation
summary: Low-rank adapation (LoRA) is a popular method that reduces the number of
  trainable parameters when finetuning large language models, but still faces
  acute storage challenges when scaling to even larger models or deploying
  numerous per-user or per-task adapted models. In this work, we present
  Vector-based Random Matrix Adaptation (VeRA), which reduces the number of
  trainable parameters by 10x compared to LoRA, yet maintains the same
  performance. It achieves this by using a single pair of low-rank matrices
  shared across all layers and learning small scaling vectors instead. We
  demonstrate its effectiveness on the GLUE and E2E benchmarks, and show its
  application in instruction-following with just 1.4M parameters using the Llama2
  7B model.
paper_authors._id: 652f5762660516114945795c
paper_authors.name: Dawid Jan Kopiczko
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-10-18T10:50:02.300Z
paper_authors._id: 637de0fa1342ba1762422495
paper_authors.avatarUrl: /avatars/b1cce386a8b33007fc381fcbfc5cdc9a.svg
paper_authors.fullname: Dawid
paper_authors.type: user
paper_authors.user: dakopi
paper_authors._id: 652f5762660516114945795d
paper_authors.name: Tijmen Blankevoort
paper_authors._id: 652f5762660516114945795e
paper_authors.name: Yuki Markus Asano
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-18T09:07:54.798Z
paper_authors._id: 637d21239a5217b88b7549c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637d21239a5217b88b7549c3/LrIGPiva5VGVZG87rTAJz.jpeg
paper_authors.fullname: Yuki Asano
paper_authors.type: user
paper_authors.user: yukimasano
paper_title: VeRA: Vector-based Random Matrix Adaptation
paper_summary: Low-rank adapation (LoRA) is a popular method that reduces the number of
  trainable parameters when finetuning large language models, but still faces
  acute storage challenges when scaling to even larger models or deploying
  numerous per-user or per-task adapted models. In this work, we present
  Vector-based Random Matrix Adaptation (VeRA), which reduces the number of
  trainable parameters by 10x compared to LoRA, yet maintains the same
  performance. It achieves this by using a single pair of low-rank matrices
  shared across all layers and learning small scaling vectors instead. We
  demonstrate its effectiveness on the GLUE and E2E benchmarks, and show its
  application in instruction-following with just 1.4M parameters using the Llama2
  7B model.
paper_ai_summary: Vector-based Random Matrix Adaptation (VeRA) reduces the number of trainable parameters by 10x compared to LoRA while maintaining performance, and is demonstrated on benchmarks like GLUE and E2E, showing its utility in instruction-following.
paper_ai_keywords: Low-rank adaptation LoRA Vector-based Random Matrix Adaptation VeRA GLUE E2E instruction-following Llama2