Adam-mini: Use Fewer Learning Rates To Gain More
We propose Adam-mini, an optimizer that achieves on-par or better performance
than AdamW with 45% to 50% less memory footprint. Adam-mini reduces memory by
cutting down the learning rate resources in Adam (i.e., 1/v). We find
that geq 90% of these learning rates in v could be harmlessly removed if
we (1) carefully partition the parameters into blocks following our proposed
principle on Hessian structure; (2) assign a single but good learning rate to
each parameter block. We further find that, for each of these parameter blocks,
there exists a single high-quality learning rate that can outperform Adam,
provided that sufficient resources are available to search it out. We then
provide one cost-effective way to find good learning rates and propose
Adam-mini. Empirically, we verify that Adam-mini performs on par or better than
AdamW on various language models sized from 125M to 7B for pre-training,
supervised fine-tuning, and RLHF. The reduced memory footprint of Adam-mini
also alleviates communication overheads among GPUs and CPUs, thereby increasing
throughput. For instance, Adam-mini achieves 49.6% higher throughput than AdamW
when pre-training Llama2-7B on 2times A800-80GB GPUs, which saves 33%
wall-clock time for pre-training.
667cd8a57bd463a3908111a8
Yushun Zhang
claimed_verified
2024-06-27T07:31:12.994Z
6548c940e70ffa3c07554102
/avatars/df7167db084ab6f10b7aed2423d82a83.svg
yushun zhang
user
yushun0410
667cd8a57bd463a3908111a9
Congliang Chen
667cd8a57bd463a3908111aa
Ziniu Li
admin_assigned
2024-06-27T13:52:46.043Z
647c4c901f878439e2fd34d6
/avatars/b4399d210d7239d4662b11a4ee7b527d.svg
Ziniu Li
user
ziniuli
667cd8a57bd463a3908111ab
Tian Ding
667cd8a57bd463a3908111ac
Chenwei Wu
667cd8a57bd463a3908111ad
Yinyu Ye
667cd8a57bd463a3908111ae
Zhi-Quan Luo
667cd8a57bd463a3908111af
Ruoyu Sun
Adam-mini: Use Fewer Learning Rates To Gain More
We propose Adam-mini, an optimizer that achieves on-par or better performance
than AdamW with 45% to 50% less memory footprint. Adam-mini reduces memory by
cutting down the learning rate resources in Adam (i.e., 1/v). We find
that geq 90% of these learning rates in v could be harmlessly removed if
we (1) carefully partition the parameters into blocks following our proposed
principle on Hessian structure; (2) assign a single but good learning rate to
each parameter block. We further find that, for each of these parameter blocks,
there exists a single high-quality learning rate that can outperform Adam,
provided that sufficient resources are available to search it out. We then
provide one cost-effective way to find good learning rates and propose
Adam-mini. Empirically, we verify that Adam-mini performs on par or better than
AdamW on various language models sized from 125M to 7B for pre-training,
supervised fine-tuning, and RLHF. The reduced memory footprint of Adam-mini
also alleviates communication overheads among GPUs and CPUs, thereby increasing
throughput. For instance, Adam-mini achieves 49.6% higher throughput than AdamW
when pre-training Llama2-7B on 2times A800-80GB GPUs, which saves 33%
wall-clock time for pre-training.
Adam-mini reduces memory usage by optimizing learning rate allocation in parameter blocks, achieving comparable or better performance than AdamW with increased throughput.
Adam-mini Adam AdamW learning rate resources Hessian structure parameter blocks throughput Llama2-7B A800-80GB GPU