title: Train Small, Infer Large: Memory-Efficient LoRA Training for Large
    Language Models
summary: Large Language Models (LLMs) have significantly advanced natural language
  processing with exceptional task generalization capabilities. Low-Rank Adaption
  (LoRA) offers a cost-effective fine-tuning solution, freezing the original
  model parameters and training only lightweight, low-rank adapter matrices.
  However, the memory footprint of LoRA is largely dominated by the original
  model parameters. To mitigate this, we propose LoRAM, a memory-efficient LoRA
  training scheme founded on the intuition that many neurons in
  over-parameterized LLMs have low training utility but are essential for
  inference. LoRAM presents a unique twist: it trains on a pruned (small) model
  to obtain pruned low-rank matrices, which are then recovered and utilized with
  the original (large) model for inference. Additionally, minimal-cost continual
  pre-training, performed by the model publishers in advance, aligns the
  knowledge discrepancy between pruned and original models. Our extensive
  experiments demonstrate the efficacy of LoRAM across various pruning strategies
  and downstream tasks. For a model with 70 billion parameters, LoRAM enables
  training on a GPU with only 20G HBM, replacing an A100-80G GPU for LoRA
  training and 15 GPUs for full fine-tuning. Specifically, QLoRAM implemented by
  structured pruning combined with 4-bit quantization, for LLaMA-3.1-70B
  (LLaMA-2-70B), reduces the parameter storage cost that dominates the memory
  usage in low-rank matrix training by 15.81times (16.95times), while
  achieving dominant performance gains over both the original LLaMA-3.1-70B
  (LLaMA-2-70B) and LoRA-trained LLaMA-3.1-8B (LLaMA-2-13B).
paper_authors._id: 67b68f883cd5860d8597eace
paper_authors.name: Jun Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T09:36:13.757Z
paper_authors._id: 63fcb42c987f631186e554f2
paper_authors.avatarUrl: /avatars/5cf87e9fa21c088c0bd8577d651d01f6.svg
paper_authors.fullname: Jun Zhang
paper_authors.type: user
paper_authors.user: junzhang98
paper_authors._id: 67b68f883cd5860d8597eacf
paper_authors.name: Jue Wang
paper_authors._id: 67b68f883cd5860d8597ead0
paper_authors.name: Huan Li
paper_authors._id: 67b68f883cd5860d8597ead1
paper_authors.name: Lidan Shou
paper_authors._id: 67b68f883cd5860d8597ead2
paper_authors.name: Ke Chen
paper_authors._id: 67b68f883cd5860d8597ead3
paper_authors.name: Yang You
paper_authors._id: 67b68f883cd5860d8597ead4
paper_authors.name: Guiming Xie
paper_authors._id: 67b68f883cd5860d8597ead5
paper_authors.name: Xuejian Gong
paper_authors._id: 67b68f883cd5860d8597ead6
paper_authors.name: Kunlong Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T17:43:26.626Z
paper_authors._id: 63db54249f2687298a14dc4d
paper_authors.avatarUrl: /avatars/7fd31c4056be5c304cd79e17e3a6e560.svg
paper_authors.fullname: kunlong zhou
paper_authors.type: user
paper_authors.user: kerlomz
paper_title: Train Small, Infer Large: Memory-Efficient LoRA Training for Large
    Language Models
paper_summary: Large Language Models (LLMs) have significantly advanced natural language
  processing with exceptional task generalization capabilities. Low-Rank Adaption
  (LoRA) offers a cost-effective fine-tuning solution, freezing the original
  model parameters and training only lightweight, low-rank adapter matrices.
  However, the memory footprint of LoRA is largely dominated by the original
  model parameters. To mitigate this, we propose LoRAM, a memory-efficient LoRA
  training scheme founded on the intuition that many neurons in
  over-parameterized LLMs have low training utility but are essential for
  inference. LoRAM presents a unique twist: it trains on a pruned (small) model
  to obtain pruned low-rank matrices, which are then recovered and utilized with
  the original (large) model for inference. Additionally, minimal-cost continual
  pre-training, performed by the model publishers in advance, aligns the
  knowledge discrepancy between pruned and original models. Our extensive
  experiments demonstrate the efficacy of LoRAM across various pruning strategies
  and downstream tasks. For a model with 70 billion parameters, LoRAM enables
  training on a GPU with only 20G HBM, replacing an A100-80G GPU for LoRA
  training and 15 GPUs for full fine-tuning. Specifically, QLoRAM implemented by
  structured pruning combined with 4-bit quantization, for LLaMA-3.1-70B
  (LLaMA-2-70B), reduces the parameter storage cost that dominates the memory
  usage in low-rank matrix training by 15.81times (16.95times), while
  achieving dominant performance gains over both the original LLaMA-3.1-70B
  (LLaMA-2-70B) and LoRA-trained LLaMA-3.1-8B (LLaMA-2-13B).
paper_ai_summary: LoRAM is a memory-efficient fine-tuning method for large language models using pruned low-rank matrices, enabling significant performance gains with reduced memory usage.
paper_ai_keywords: LLMs Low-Rank Adaption LoRA LoRAM memory footprint over-parameterized models prune pruned low-rank matrices minimal-cost continual pre-training pruning strategies structured pruning quantization parameter storage cost