title: Adam-mini: Use Fewer Learning Rates To Gain More
summary: We propose Adam-mini, an optimizer that achieves on-par or better performance
  than AdamW with 45% to 50% less memory footprint. Adam-mini reduces memory by
  cutting down the learning rate resources in Adam (i.e., 1/v). We find
  that geq 90% of these learning rates in v could be harmlessly removed if
  we (1) carefully partition the parameters into blocks following our proposed
  principle on Hessian structure; (2) assign a single but good learning rate to
  each parameter block. We further find that, for each of these parameter blocks,
  there exists a single high-quality learning rate that can outperform Adam,
  provided that sufficient resources are available to search it out. We then
  provide one cost-effective way to find good learning rates and propose
  Adam-mini. Empirically, we verify that Adam-mini performs on par or better than
  AdamW on various language models sized from 125M to 7B for pre-training,
  supervised fine-tuning, and RLHF. The reduced memory footprint of Adam-mini
  also alleviates communication overheads among GPUs and CPUs, thereby increasing
  throughput. For instance, Adam-mini achieves 49.6% higher throughput than AdamW
  when pre-training Llama2-7B on 2times A800-80GB GPUs, which saves 33%
  wall-clock time for pre-training.
paper_authors._id: 667cd8a57bd463a3908111a8
paper_authors.name: Yushun Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T07:31:12.994Z
paper_authors._id: 6548c940e70ffa3c07554102
paper_authors.avatarUrl: /avatars/df7167db084ab6f10b7aed2423d82a83.svg
paper_authors.fullname: yushun zhang
paper_authors.type: user
paper_authors.user: yushun0410
paper_authors._id: 667cd8a57bd463a3908111a9
paper_authors.name: Congliang Chen
paper_authors._id: 667cd8a57bd463a3908111aa
paper_authors.name: Ziniu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T13:52:46.043Z
paper_authors._id: 647c4c901f878439e2fd34d6
paper_authors.avatarUrl: /avatars/b4399d210d7239d4662b11a4ee7b527d.svg
paper_authors.fullname: Ziniu Li
paper_authors.type: user
paper_authors.user: ziniuli
paper_authors._id: 667cd8a57bd463a3908111ab
paper_authors.name: Tian Ding
paper_authors._id: 667cd8a57bd463a3908111ac
paper_authors.name: Chenwei Wu
paper_authors._id: 667cd8a57bd463a3908111ad
paper_authors.name: Yinyu Ye
paper_authors._id: 667cd8a57bd463a3908111ae
paper_authors.name: Zhi-Quan Luo
paper_authors._id: 667cd8a57bd463a3908111af
paper_authors.name: Ruoyu Sun
paper_title: Adam-mini: Use Fewer Learning Rates To Gain More
paper_summary: We propose Adam-mini, an optimizer that achieves on-par or better performance
  than AdamW with 45% to 50% less memory footprint. Adam-mini reduces memory by
  cutting down the learning rate resources in Adam (i.e., 1/v). We find
  that geq 90% of these learning rates in v could be harmlessly removed if
  we (1) carefully partition the parameters into blocks following our proposed
  principle on Hessian structure; (2) assign a single but good learning rate to
  each parameter block. We further find that, for each of these parameter blocks,
  there exists a single high-quality learning rate that can outperform Adam,
  provided that sufficient resources are available to search it out. We then
  provide one cost-effective way to find good learning rates and propose
  Adam-mini. Empirically, we verify that Adam-mini performs on par or better than
  AdamW on various language models sized from 125M to 7B for pre-training,
  supervised fine-tuning, and RLHF. The reduced memory footprint of Adam-mini
  also alleviates communication overheads among GPUs and CPUs, thereby increasing
  throughput. For instance, Adam-mini achieves 49.6% higher throughput than AdamW
  when pre-training Llama2-7B on 2times A800-80GB GPUs, which saves 33%
  wall-clock time for pre-training.
paper_ai_keywords: Adam-mini AdamW learning rate resources Hessian structure parameter blocks learning rates High-quality learning rate memory footprint communication overheads throughput Llama2-7B pre-training supervised fine-tuning RLHF wall-clock time