title: Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse
    Gradients
summary: Large language model (LLM) training and finetuning are often bottlenecked by
  limited GPU memory. While existing projection-based optimization methods
  address this by projecting gradients into a lower-dimensional subspace to
  reduce optimizer state memory, they typically rely on dense projection
  matrices, which can introduce computational and memory overheads. In this work,
  we propose Grass (GRAdient Stuctured Sparsification), a novel approach that
  leverages sparse projections to transform gradients into structured sparse
  updates. This design not only significantly reduces memory usage for optimizer
  states but also minimizes gradient memory footprint, computation, and
  communication costs, leading to substantial throughput improvements. Extensive
  experiments on pretraining and finetuning tasks demonstrate that Grass achieves
  competitive performance to full-rank training and existing projection-based
  methods. Notably, Grass enables half-precision pretraining of a 13B parameter
  LLaMA model on a single 40GB A100 GPU--a feat infeasible for previous
  methods--and yields up to a 2times throughput improvement on an 8-GPU
  system. Code can be found at https://github.com/aashiqmuhamed/GRASS .
paper_authors._id: 667c13f77e940351583d9323
paper_authors.name: Aashiq Muhamed
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T13:20:17.469Z
paper_authors._id: 64755a83e0b188d3cb2579d8
paper_authors.avatarUrl: /avatars/2c50590905f4bd398a4c9991e1b4b5bb.svg
paper_authors.fullname: Aashiq Muhamed
paper_authors.type: user
paper_authors.user: aashiqmuhamed
paper_authors._id: 667c13f77e940351583d9324
paper_authors.name: Oscar Li
paper_authors._id: 667c13f77e940351583d9325
paper_authors.name: David Woodruff
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:26:07.534Z
paper_authors._id: 6330a6fa16eecf0872949a6f
paper_authors.avatarUrl: /avatars/519678ea1bafe627aefe79ac13d427de.svg
paper_authors.fullname: David Woodruff
paper_authors.type: user
paper_authors.user: dmw1101
paper_authors._id: 667c13f77e940351583d9326
paper_authors.name: Mona Diab
paper_authors._id: 667c13f77e940351583d9327
paper_authors.name: Virginia Smith
paper_title: Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse
    Gradients
paper_summary: Large language model (LLM) training and finetuning are often bottlenecked by
  limited GPU memory. While existing projection-based optimization methods
  address this by projecting gradients into a lower-dimensional subspace to
  reduce optimizer state memory, they typically rely on dense projection
  matrices, which can introduce computational and memory overheads. In this work,
  we propose Grass (GRAdient Stuctured Sparsification), a novel approach that
  leverages sparse projections to transform gradients into structured sparse
  updates. This design not only significantly reduces memory usage for optimizer
  states but also minimizes gradient memory footprint, computation, and
  communication costs, leading to substantial throughput improvements. Extensive
  experiments on pretraining and finetuning tasks demonstrate that Grass achieves
  competitive performance to full-rank training and existing projection-based
  methods. Notably, Grass enables half-precision pretraining of a 13B parameter
  LLaMA model on a single 40GB A100 GPU--a feat infeasible for previous
  methods--and yields up to a 2times throughput improvement on an 8-GPU
  system. Code can be found at https://github.com/aashiqmuhamed/GRASS .
paper_ai_keywords: gradient projection lower-dimensional subspace dense projection matrices sparse projections structured sparse updates optimizer state memory gradient memory footprint computational costs communication costs throughput improvements pretraining finetuning half-precision pretraining