Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive
  Low-Rank Gradients
Training Large Language Models (LLMs) is memory-intensive due to the large
number of parameters and associated optimization states. GaLore, a recent
method, reduces memory usage by projecting weight gradients into a low-rank
subspace without compromising performance. However, GaLore relies on
time-consuming Singular Value Decomposition (SVD) operations to identify the
subspace, and the frequent subspace updates lead to significant training time
overhead. Moreover, GaLore offers minimal improvements in accuracy and
efficiency compared to LoRA in more accessible fine-tuning scenarios. To
address these limitations, we introduce Q-Galore, a novel approach that
substantially reduces memory usage by combining quantization and low-rank
projection, surpassing the benefits of GaLore. Our method is based on two key
observations: (i) the gradient subspace exhibits diverse properties, with some
layers converging early in training while others are subject to frequent
changes; (ii) the projection matrices are highly resilient to low-bit
quantization. Leveraging these insights, Q-GaLore adaptively updates the
gradient subspace based on its convergence statistics, achieving comparable
performance while significantly reducing the number of SVD operations. We
maintain the projection matrices in INT4 format and weights in INT8 format,
incorporating stochastic rounding to capture accumulated gradient information.
This approach enables a high-precision training trajectory using only
low-precision weights. We demonstrate that Q-GaLore achieves highly competitive
performance with exceptional memory efficiency. At pre-training, Q-GaLore
facilitates training a LLaMA-7B model from scratch on a single NVIDIA RTX 4060
Ti with only 16 GB memory. At fine-tuning, it reduces memory consumption by up
to 50% compared to LoRA and GaLore, while consistently outperforming QLoRA at
the same memory cost.
6690b32d21b5557342304704
Zhenyu Zhang
claimed_verified
2024-07-12T07:39:30.970Z
649c888f67fd6c6aa97e5f85
/avatars/9967b729916d1128773102797fed1673.svg
Zhenyu Zhang
user
Kyriection
6690b32d21b5557342304705
Ajay Jaiswal
admin_assigned
2024-07-12T12:34:39.957Z
64ad7afdafb6aa5534221c96
/avatars/c1c83d8f5ca683ce60fd4d50c028471c.svg
Ajay Jaiswal
user
Ajay1994
6690b32d21b5557342304706
Lu Yin
6690b32d21b5557342304707
Shiwei Liu
claimed_verified
2025-03-03T11:15:34.304Z
65b04d2291e63920a7898c9e
https://cdn-avatars.huggingface.co/v1/production/uploads/65b04d2291e63920a7898c9e/iUHs235G4bqK-KnH_94ti.jpeg
Liu
user
Shiweiliuiiiiiii
6690b32d21b5557342304708
Jiawei Zhao
admin_assigned
2024-07-12T12:34:53.760Z
64dd8355573d067c9e858262
https://cdn-avatars.huggingface.co/v1/production/uploads/64dd8355573d067c9e858262/Bu2kFs6-lcYc93A_SE8WU.jpeg
Jiawei Zhao
user
jiaweizhao
6690b32d21b5557342304709
Yuandong Tian
admin_assigned
2024-07-12T12:35:06.852Z
6344cf73ee1504dbcd5bdfe7
/avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
Yuandong Tian
user
tydsh
6690b32d21b555734230470a
Zhangyang Wang
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive
  Low-Rank Gradients
Training Large Language Models (LLMs) is memory-intensive due to the large
number of parameters and associated optimization states. GaLore, a recent
method, reduces memory usage by projecting weight gradients into a low-rank
subspace without compromising performance. However, GaLore relies on
time-consuming Singular Value Decomposition (SVD) operations to identify the
subspace, and the frequent subspace updates lead to significant training time
overhead. Moreover, GaLore offers minimal improvements in accuracy and
efficiency compared to LoRA in more accessible fine-tuning scenarios. To
address these limitations, we introduce Q-Galore, a novel approach that
substantially reduces memory usage by combining quantization and low-rank
projection, surpassing the benefits of GaLore. Our method is based on two key
observations: (i) the gradient subspace exhibits diverse properties, with some
layers converging early in training while others are subject to frequent
changes; (ii) the projection matrices are highly resilient to low-bit
quantization. Leveraging these insights, Q-GaLore adaptively updates the
gradient subspace based on its convergence statistics, achieving comparable
performance while significantly reducing the number of SVD operations. We
maintain the projection matrices in INT4 format and weights in INT8 format,
incorporating stochastic rounding to capture accumulated gradient information.
This approach enables a high-precision training trajectory using only
low-precision weights. We demonstrate that Q-GaLore achieves highly competitive
performance with exceptional memory efficiency. At pre-training, Q-GaLore
facilitates training a LLaMA-7B model from scratch on a single NVIDIA RTX 4060
Ti with only 16 GB memory. At fine-tuning, it reduces memory consumption by up
to 50% compared to LoRA and GaLore, while consistently outperforming QLoRA at
the same memory cost.
Q-GaLore reduces memory usage for training large language models by combining quantization and low-rank projection, surpassing GaLore and LoRA in terms of efficiency and performance.
Training Large Language Models LLMs GaLore Singular Value Decomposition SVD gradient subspace low-rank projection quantization gradient convergence INT4 INT8 stochastic rounding Q-GaLore QLoRA pre-training fine-tuning LLaMA-7B