title: VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections
summary: Large language models (LLMs) have recently emerged as powerful tools for
  tackling many language-processing tasks. Despite their success, training and
  fine-tuning these models is still far too computationally and memory intensive.
  In this paper, we identify and characterise the important components needed for
  effective model convergence using gradient descent. In doing so we find that
  the intermediate activations used to implement backpropagation can be
  excessively compressed without incurring any degradation in performance. This
  result leads us to a cheap and memory-efficient algorithm for both fine-tuning
  and pre-training LLMs. The proposed algorithm simply divides the tokens up into
  smaller sub-tokens before projecting them onto a fixed 1-dimensional subspace
  during the forward pass. These features are then coarsely reconstructed during
  the backward pass to implement the update rules. We confirm the effectiveness
  of our algorithm as being complimentary to many state-of-the-art PEFT methods
  on the VTAB-1k fine-tuning benchmark. Furthermore, we outperform QLoRA for
  fine-tuning LLaMA and show competitive performance against other
  memory-efficient pre-training methods on the large-scale C4 dataset.
paper_authors._id: 66569a0b7d360370970b97be
paper_authors.name: Roy Miles
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-29T08:26:18.417Z
paper_authors._id: 6478cea13b7f8b1f624997e5
paper_authors.avatarUrl: /avatars/34641d634da565cd50a7310de3f4513f.svg
paper_authors.fullname: Roy Miles
paper_authors.type: user
paper_authors.user: iyop45
paper_authors._id: 66569a0b7d360370970b97bf
paper_authors.name: Pradyumna Reddy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-29T08:26:24.007Z
paper_authors._id: 6511b8bbb6ffd31931bf5f3e
paper_authors.avatarUrl: /avatars/a5cdd5d0750a8104f754c3347abd2103.svg
paper_authors.fullname: Pradyumna Reddy
paper_authors.type: user
paper_authors.user: preddy5
paper_authors._id: 66569a0b7d360370970b97c0
paper_authors.name: Ismail Elezi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-29T08:26:30.253Z
paper_authors._id: 63ea3565e7f148f46013f950
paper_authors.avatarUrl: /avatars/afbaacf52e50a92344217a7752e2fb52.svg
paper_authors.fullname: Ismail Elezi
paper_authors.type: user
paper_authors.user: Ili1991
paper_authors._id: 66569a0b7d360370970b97c1
paper_authors.name: Jiankang Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-29T08:26:36.213Z
paper_authors._id: 62cc7a38376917c0223dd24b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1657566065867-noauth.png
paper_authors.fullname: JiankangDeng
paper_authors.type: user
paper_authors.user: JiankangDeng
paper_title: VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections
paper_summary: Large language models (LLMs) have recently emerged as powerful tools for
  tackling many language-processing tasks. Despite their success, training and
  fine-tuning these models is still far too computationally and memory intensive.
  In this paper, we identify and characterise the important components needed for
  effective model convergence using gradient descent. In doing so we find that
  the intermediate activations used to implement backpropagation can be
  excessively compressed without incurring any degradation in performance. This
  result leads us to a cheap and memory-efficient algorithm for both fine-tuning
  and pre-training LLMs. The proposed algorithm simply divides the tokens up into
  smaller sub-tokens before projecting them onto a fixed 1-dimensional subspace
  during the forward pass. These features are then coarsely reconstructed during
  the backward pass to implement the update rules. We confirm the effectiveness
  of our algorithm as being complimentary to many state-of-the-art PEFT methods
  on the VTAB-1k fine-tuning benchmark. Furthermore, we outperform QLoRA for
  fine-tuning LLaMA and show competitive performance against other
  memory-efficient pre-training methods on the large-scale C4 dataset.
paper_ai_summary: The proposed algorithm reduces memory usage during the training and fine-tuning of large language models by compressing intermediate activations without affecting performance.
paper_ai_keywords: gradient descent backpropagation parameter-efficient fine-tuning (PEFT) forward pass backward pass sub-tokens 1-dimensional subspace VTAB-1k fine-tuning benchmark QLoRA C4 dataset