VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections
Large language models (LLMs) have recently emerged as powerful tools for
tackling many language-processing tasks. Despite their success, training and
fine-tuning these models is still far too computationally and memory intensive.
In this paper, we identify and characterise the important components needed for
effective model convergence using gradient descent. In doing so we find that
the intermediate activations used to implement backpropagation can be
excessively compressed without incurring any degradation in performance. This
result leads us to a cheap and memory-efficient algorithm for both fine-tuning
and pre-training LLMs. The proposed algorithm simply divides the tokens up into
smaller sub-tokens before projecting them onto a fixed 1-dimensional subspace
during the forward pass. These features are then coarsely reconstructed during
the backward pass to implement the update rules. We confirm the effectiveness
of our algorithm as being complimentary to many state-of-the-art PEFT methods
on the VTAB-1k fine-tuning benchmark. Furthermore, we outperform QLoRA for
fine-tuning LLaMA and show competitive performance against other
memory-efficient pre-training methods on the large-scale C4 dataset.
66569a0b7d360370970b97be
Roy Miles
admin_assigned
2024-05-29T08:26:18.417Z
6478cea13b7f8b1f624997e5
/avatars/34641d634da565cd50a7310de3f4513f.svg
Roy Miles
user
iyop45
66569a0b7d360370970b97bf
Pradyumna Reddy
admin_assigned
2024-05-29T08:26:24.007Z
6511b8bbb6ffd31931bf5f3e
/avatars/a5cdd5d0750a8104f754c3347abd2103.svg
Pradyumna Reddy
user
preddy5
66569a0b7d360370970b97c0
Ismail Elezi
admin_assigned
2024-05-29T08:26:30.253Z
63ea3565e7f148f46013f950
/avatars/afbaacf52e50a92344217a7752e2fb52.svg
Ismail Elezi
user
Ili1991
66569a0b7d360370970b97c1
Jiankang Deng
admin_assigned
2024-05-29T08:26:36.213Z
62cc7a38376917c0223dd24b
https://cdn-avatars.huggingface.co/v1/production/uploads/1657566065867-noauth.png
JiankangDeng
user
JiankangDeng
VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections
Large language models (LLMs) have recently emerged as powerful tools for
tackling many language-processing tasks. Despite their success, training and
fine-tuning these models is still far too computationally and memory intensive.
In this paper, we identify and characterise the important components needed for
effective model convergence using gradient descent. In doing so we find that
the intermediate activations used to implement backpropagation can be
excessively compressed without incurring any degradation in performance. This
result leads us to a cheap and memory-efficient algorithm for both fine-tuning
and pre-training LLMs. The proposed algorithm simply divides the tokens up into
smaller sub-tokens before projecting them onto a fixed 1-dimensional subspace
during the forward pass. These features are then coarsely reconstructed during
the backward pass to implement the update rules. We confirm the effectiveness
of our algorithm as being complimentary to many state-of-the-art PEFT methods
on the VTAB-1k fine-tuning benchmark. Furthermore, we outperform QLoRA for
fine-tuning LLaMA and show competitive performance against other
memory-efficient pre-training methods on the large-scale C4 dataset.
The proposed algorithm reduces memory usage during the training and fine-tuning of large language models by compressing intermediate activations without affecting performance.
gradient descent backpropagation parameter-efficient fine-tuning (PEFT) forward pass backward pass sub-tokens 1-dimensional subspace VTAB-1k fine-tuning benchmark QLoRA C4 dataset