Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse
  Gradients
Large language model (LLM) training and finetuning are often bottlenecked by
limited GPU memory. While existing projection-based optimization methods
address this by projecting gradients into a lower-dimensional subspace to
reduce optimizer state memory, they typically rely on dense projection
matrices, which can introduce computational and memory overheads. In this work,
we propose Grass (GRAdient Stuctured Sparsification), a novel approach that
leverages sparse projections to transform gradients into structured sparse
updates. This design not only significantly reduces memory usage for optimizer
states but also minimizes gradient memory footprint, computation, and
communication costs, leading to substantial throughput improvements. Extensive
experiments on pretraining and finetuning tasks demonstrate that Grass achieves
competitive performance to full-rank training and existing projection-based
methods. Notably, Grass enables half-precision pretraining of a 13B parameter
LLaMA model on a single 40GB A100 GPU--a feat infeasible for previous
methods--and yields up to a 2times throughput improvement on an 8-GPU
system. Code can be found at https://github.com/aashiqmuhamed/GRASS .
667c13f77e940351583d9323
Aashiq Muhamed
claimed_verified
2024-06-26T13:20:17.469Z
64755a83e0b188d3cb2579d8
/avatars/2c50590905f4bd398a4c9991e1b4b5bb.svg
Aashiq Muhamed
user
aashiqmuhamed
667c13f77e940351583d9324
Oscar Li
667c13f77e940351583d9325
David Woodruff
admin_assigned
2024-06-26T13:26:07.534Z
6330a6fa16eecf0872949a6f
/avatars/519678ea1bafe627aefe79ac13d427de.svg
David Woodruff
user
dmw1101
667c13f77e940351583d9326
Mona Diab
667c13f77e940351583d9327
Virginia Smith
Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse
  Gradients
Large language model (LLM) training and finetuning are often bottlenecked by
limited GPU memory. While existing projection-based optimization methods
address this by projecting gradients into a lower-dimensional subspace to
reduce optimizer state memory, they typically rely on dense projection
matrices, which can introduce computational and memory overheads. In this work,
we propose Grass (GRAdient Stuctured Sparsification), a novel approach that
leverages sparse projections to transform gradients into structured sparse
updates. This design not only significantly reduces memory usage for optimizer
states but also minimizes gradient memory footprint, computation, and
communication costs, leading to substantial throughput improvements. Extensive
experiments on pretraining and finetuning tasks demonstrate that Grass achieves
competitive performance to full-rank training and existing projection-based
methods. Notably, Grass enables half-precision pretraining of a 13B parameter
LLaMA model on a single 40GB A100 GPU--a feat infeasible for previous
methods--and yields up to a 2times throughput improvement on an 8-GPU
system. Code can be found at https://github.com/aashiqmuhamed/GRASS .
Grass uses sparse projections to reduce memory and compute costs during the training and fine-tuning of large language models, enabling efficient pretraining on GPUs.
Grass GRAdient Stuctured Sparsification sparse projections gradient updates memory usage computational overheads throughput improvements large language models pretraining finetuning parameter-efficient fine-tuning