title: GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
summary: Training Large Language Models (LLMs) presents significant memory challenges,
  predominantly due to the growing size of weights and optimizer states. Common
  memory-reduction approaches, such as low-rank adaptation (LoRA), add a
  trainable low-rank matrix to the frozen pre-trained weight in each layer,
  reducing trainable parameters and optimizer states. However, such approaches
  typically underperform training with full-rank weights in both pre-training and
  fine-tuning stages since they limit the parameter search to a low-rank subspace
  and alter the training dynamics, and further, may require full-rank warm start.
  In this work, we propose Gradient Low-Rank Projection (GaLore), a training
  strategy that allows full-parameter learning but is more memory-efficient than
  common low-rank adaptation methods such as LoRA. Our approach reduces memory
  usage by up to 65.5% in optimizer states while maintaining both efficiency and
  performance for pre-training on LLaMA 1B and 7B architectures with C4 dataset
  with up to 19.7B tokens, and on fine-tuning RoBERTa on GLUE tasks. Our 8-bit
  GaLore further reduces optimizer memory by up to 82.5% and total training
  memory by 63.3%, compared to a BF16 baseline. Notably, we demonstrate, for the
  first time, the feasibility of pre-training a 7B model on consumer GPUs with
  24GB memory (e.g., NVIDIA RTX 4090) without model parallel, checkpointing, or
  offloading strategies.
paper_authors._id: 65e944d31a58734e13cfd732
paper_authors.name: Jiawei Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T13:50:24.165Z
paper_authors._id: 64dd8355573d067c9e858262
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64dd8355573d067c9e858262/Bu2kFs6-lcYc93A_SE8WU.jpeg
paper_authors.fullname: Jiawei Zhao
paper_authors.type: user
paper_authors.user: jiaweizhao
paper_authors._id: 65e944d31a58734e13cfd733
paper_authors.name: Zhenyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T09:14:13.350Z
paper_authors._id: 649c888f67fd6c6aa97e5f85
paper_authors.avatarUrl: /avatars/9967b729916d1128773102797fed1673.svg
paper_authors.fullname: Zhenyu Zhang
paper_authors.type: user
paper_authors.user: Kyriection
paper_authors._id: 65e944d31a58734e13cfd734
paper_authors.name: Beidi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T09:37:44.839Z
paper_authors._id: 64b732f832403871593e082c
paper_authors.avatarUrl: /avatars/dd21932b0c167131ee7545a622c46c3c.svg
paper_authors.fullname: Beidi Chen
paper_authors.type: user
paper_authors.user: beidic
paper_authors._id: 65e944d31a58734e13cfd735
paper_authors.name: Zhangyang Wang
paper_authors._id: 65e944d31a58734e13cfd736
paper_authors.name: Anima Anandkumar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T09:38:17.113Z
paper_authors._id: 6532920b3e385cfc6002938d
paper_authors.avatarUrl: /avatars/cb9cc6d2733031582c83f56dc6cd1dd5.svg
paper_authors.fullname: Anima Anandkumar
paper_authors.type: user
paper_authors.user: animakumar
paper_authors._id: 65e944d31a58734e13cfd737
paper_authors.name: Yuandong Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T09:38:34.212Z
paper_authors._id: 6344cf73ee1504dbcd5bdfe7
paper_authors.avatarUrl: /avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
paper_authors.fullname: Yuandong Tian
paper_authors.type: user
paper_authors.user: tydsh
paper_title: GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
paper_summary: Training Large Language Models (LLMs) presents significant memory challenges,
  predominantly due to the growing size of weights and optimizer states. Common
  memory-reduction approaches, such as low-rank adaptation (LoRA), add a
  trainable low-rank matrix to the frozen pre-trained weight in each layer,
  reducing trainable parameters and optimizer states. However, such approaches
  typically underperform training with full-rank weights in both pre-training and
  fine-tuning stages since they limit the parameter search to a low-rank subspace
  and alter the training dynamics, and further, may require full-rank warm start.
  In this work, we propose Gradient Low-Rank Projection (GaLore), a training
  strategy that allows full-parameter learning but is more memory-efficient than
  common low-rank adaptation methods such as LoRA. Our approach reduces memory
  usage by up to 65.5% in optimizer states while maintaining both efficiency and
  performance for pre-training on LLaMA 1B and 7B architectures with C4 dataset
  with up to 19.7B tokens, and on fine-tuning RoBERTa on GLUE tasks. Our 8-bit
  GaLore further reduces optimizer memory by up to 82.5% and total training
  memory by 63.3%, compared to a BF16 baseline. Notably, we demonstrate, for the
  first time, the feasibility of pre-training a 7B model on consumer GPUs with
  24GB memory (e.g., NVIDIA RTX 4090) without model parallel, checkpointing, or
  offloading strategies.
paper_ai_keywords: low-rank adaptation low-rank matrix trainable parameters optimizer states parameter search training dynamics Gradient Low-Rank Projection (GaLore) full-parameter learning memory-efficient optimizer memory total training memory BF16 baseline pre-training LLaMA 1B LLaMA 7B C4 dataset fine-tuning ROBERTa GLUE tasks consumer GPUs NVIDIA RTX 4090 offloading strategies