GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
Training Large Language Models (LLMs) presents significant memory challenges,
predominantly due to the growing size of weights and optimizer states. Common
memory-reduction approaches, such as low-rank adaptation (LoRA), add a
trainable low-rank matrix to the frozen pre-trained weight in each layer,
reducing trainable parameters and optimizer states. However, such approaches
typically underperform training with full-rank weights in both pre-training and
fine-tuning stages since they limit the parameter search to a low-rank subspace
and alter the training dynamics, and further, may require full-rank warm start.
In this work, we propose Gradient Low-Rank Projection (GaLore), a training
strategy that allows full-parameter learning but is more memory-efficient than
common low-rank adaptation methods such as LoRA. Our approach reduces memory
usage by up to 65.5% in optimizer states while maintaining both efficiency and
performance for pre-training on LLaMA 1B and 7B architectures with C4 dataset
with up to 19.7B tokens, and on fine-tuning RoBERTa on GLUE tasks. Our 8-bit
GaLore further reduces optimizer memory by up to 82.5% and total training
memory by 63.3%, compared to a BF16 baseline. Notably, we demonstrate, for the
first time, the feasibility of pre-training a 7B model on consumer GPUs with
24GB memory (e.g., NVIDIA RTX 4090) without model parallel, checkpointing, or
offloading strategies.
65e944d31a58734e13cfd732
Jiawei Zhao
claimed_verified
2024-03-07T13:50:24.165Z
64dd8355573d067c9e858262
https://cdn-avatars.huggingface.co/v1/production/uploads/64dd8355573d067c9e858262/Bu2kFs6-lcYc93A_SE8WU.jpeg
Jiawei Zhao
user
jiaweizhao
65e944d31a58734e13cfd733
Zhenyu Zhang
claimed_verified
2024-03-07T09:14:13.350Z
649c888f67fd6c6aa97e5f85
/avatars/9967b729916d1128773102797fed1673.svg
Zhenyu Zhang
user
Kyriection
65e944d31a58734e13cfd734
Beidi Chen
admin_assigned
2024-03-07T09:37:44.839Z
64b732f832403871593e082c
/avatars/dd21932b0c167131ee7545a622c46c3c.svg
Beidi Chen
user
beidic
65e944d31a58734e13cfd735
Zhangyang Wang
65e944d31a58734e13cfd736
Anima Anandkumar
admin_assigned
2024-03-07T09:38:17.113Z
6532920b3e385cfc6002938d
/avatars/cb9cc6d2733031582c83f56dc6cd1dd5.svg
Anima Anandkumar
user
animakumar
65e944d31a58734e13cfd737
Yuandong Tian
admin_assigned
2024-03-07T09:38:34.212Z
6344cf73ee1504dbcd5bdfe7
/avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
Yuandong Tian
user
tydsh
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
Training Large Language Models (LLMs) presents significant memory challenges,
predominantly due to the growing size of weights and optimizer states. Common
memory-reduction approaches, such as low-rank adaptation (LoRA), add a
trainable low-rank matrix to the frozen pre-trained weight in each layer,
reducing trainable parameters and optimizer states. However, such approaches
typically underperform training with full-rank weights in both pre-training and
fine-tuning stages since they limit the parameter search to a low-rank subspace
and alter the training dynamics, and further, may require full-rank warm start.
In this work, we propose Gradient Low-Rank Projection (GaLore), a training
strategy that allows full-parameter learning but is more memory-efficient than
common low-rank adaptation methods such as LoRA. Our approach reduces memory
usage by up to 65.5% in optimizer states while maintaining both efficiency and
performance for pre-training on LLaMA 1B and 7B architectures with C4 dataset
with up to 19.7B tokens, and on fine-tuning RoBERTa on GLUE tasks. Our 8-bit
GaLore further reduces optimizer memory by up to 82.5% and total training
memory by 63.3%, compared to a BF16 baseline. Notably, we demonstrate, for the
first time, the feasibility of pre-training a 7B model on consumer GPUs with
24GB memory (e.g., NVIDIA RTX 4090) without model parallel, checkpointing, or
offloading strategies.
Gradient Low-Rank Projection (GaLore) improves memory efficiency for training large language models without sacrificing performance, enabling pre-training of 7B models on consumer GPUs.
low-rank adaptation (LoRA) Gradient Low-Rank Projection (GaLore) memory efficiency pre-training fine-tuning LLaMA C4 dataset RoBERTa GLUE tasks 8-bit GaLore BF16 baseline consumer GPUs model parallel checkpointing offloading strategies