MARS: Unleashing the Power of Variance Reduction for Training Large
  Models
Training deep neural networks--and more recently, large models--demands
efficient and scalable optimizers. Adaptive gradient algorithms like Adam,
AdamW, and their variants have been central to this task. Despite the
development of numerous variance reduction algorithms in the past decade aimed
at accelerating stochastic optimization in both convex and nonconvex settings,
variance reduction has not found widespread success in training deep neural
networks or large language models. Consequently, it has remained a less favored
approach in modern AI. In this paper, to unleash the power of variance
reduction for efficient training of large models, we propose a unified
optimization framework, MARS (Make vAriance Reduction Shine), which reconciles
preconditioned gradient methods with variance reduction via a scaled stochastic
recursive momentum technique. Within our framework, we introduce three
instances of MARS that leverage preconditioned gradient updates based on AdamW,
Lion, and Shampoo, respectively. We also draw a connection between our
algorithms and existing optimizers. Experimental results on training GPT-2
models indicate that MARS consistently outperforms AdamW by a large margin.
673ba53a79b5a48d372c6d2a
Huizhuo Yuan
admin_assigned
2024-11-18T20:57:36.594Z
657be24e8d360b690d5b665f
/avatars/423185c3f386e3fec2e5708c1030b013.svg
Angela Yuan
user
angelahzyuan
673ba53a79b5a48d372c6d2b
Yifeng Liu
claimed_verified
2024-11-19T08:39:41.010Z
653d276681f52ceb4d12bd85
/avatars/56601a25e5f883a8f6dc15f6fd9dcc57.svg
Yifeng Liu
user
Lewis-Lau
673ba53a79b5a48d372c6d2c
Shuang Wu
673ba53a79b5a48d372c6d2d
Xun Zhou
673ba53a79b5a48d372c6d2e
Quanquan Gu
admin_assigned
2024-11-18T20:55:59.308Z
64c039128e2612254356bba5
/avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
Quanquan Gu
user
thughost
MARS: Unleashing the Power of Variance Reduction for Training Large
  Models
Training deep neural networks--and more recently, large models--demands
efficient and scalable optimizers. Adaptive gradient algorithms like Adam,
AdamW, and their variants have been central to this task. Despite the
development of numerous variance reduction algorithms in the past decade aimed
at accelerating stochastic optimization in both convex and nonconvex settings,
variance reduction has not found widespread success in training deep neural
networks or large language models. Consequently, it has remained a less favored
approach in modern AI. In this paper, to unleash the power of variance
reduction for efficient training of large models, we propose a unified
optimization framework, MARS (Make vAriance Reduction Shine), which reconciles
preconditioned gradient methods with variance reduction via a scaled stochastic
recursive momentum technique. Within our framework, we introduce three
instances of MARS that leverage preconditioned gradient updates based on AdamW,
Lion, and Shampoo, respectively. We also draw a connection between our
algorithms and existing optimizers. Experimental results on training GPT-2
models indicate that MARS consistently outperforms AdamW by a large margin.
MARS, a unified optimization framework combining preconditioned gradient methods with variance reduction, improves the training efficiency of large models like GPT-2 compared to AdamW.
Adam AdamW variance reduction adaptive gradient algorithms stochastic optimization convex settings nonconvex settings preconditioned gradient methods scaled stochastic recursive momentum Lion Shampoo GPT-2