title: MARS: Unleashing the Power of Variance Reduction for Training Large
    Models
summary: Training deep neural networks--and more recently, large models--demands
  efficient and scalable optimizers. Adaptive gradient algorithms like Adam,
  AdamW, and their variants have been central to this task. Despite the
  development of numerous variance reduction algorithms in the past decade aimed
  at accelerating stochastic optimization in both convex and nonconvex settings,
  variance reduction has not found widespread success in training deep neural
  networks or large language models. Consequently, it has remained a less favored
  approach in modern AI. In this paper, to unleash the power of variance
  reduction for efficient training of large models, we propose a unified
  optimization framework, MARS (Make vAriance Reduction Shine), which reconciles
  preconditioned gradient methods with variance reduction via a scaled stochastic
  recursive momentum technique. Within our framework, we introduce three
  instances of MARS that leverage preconditioned gradient updates based on AdamW,
  Lion, and Shampoo, respectively. We also draw a connection between our
  algorithms and existing optimizers. Experimental results on training GPT-2
  models indicate that MARS consistently outperforms AdamW by a large margin.
paper_authors._id: 673ba53a79b5a48d372c6d2a
paper_authors.name: Huizhuo Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-18T20:57:36.594Z
paper_authors._id: 657be24e8d360b690d5b665f
paper_authors.avatarUrl: /avatars/423185c3f386e3fec2e5708c1030b013.svg
paper_authors.fullname: Angela Yuan
paper_authors.type: user
paper_authors.user: angelahzyuan
paper_authors._id: 673ba53a79b5a48d372c6d2b
paper_authors.name: Yifeng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-19T08:39:41.010Z
paper_authors._id: 653d276681f52ceb4d12bd85
paper_authors.avatarUrl: /avatars/56601a25e5f883a8f6dc15f6fd9dcc57.svg
paper_authors.fullname: Yifeng Liu
paper_authors.type: user
paper_authors.user: Lewis-Lau
paper_authors._id: 673ba53a79b5a48d372c6d2c
paper_authors.name: Shuang Wu
paper_authors._id: 673ba53a79b5a48d372c6d2d
paper_authors.name: Xun Zhou
paper_authors._id: 673ba53a79b5a48d372c6d2e
paper_authors.name: Quanquan Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-18T20:55:59.308Z
paper_authors._id: 64c039128e2612254356bba5
paper_authors.avatarUrl: /avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
paper_authors.fullname: Quanquan Gu
paper_authors.type: user
paper_authors.user: thughost
paper_title: MARS: Unleashing the Power of Variance Reduction for Training Large
    Models
paper_summary: Training deep neural networks--and more recently, large models--demands
  efficient and scalable optimizers. Adaptive gradient algorithms like Adam,
  AdamW, and their variants have been central to this task. Despite the
  development of numerous variance reduction algorithms in the past decade aimed
  at accelerating stochastic optimization in both convex and nonconvex settings,
  variance reduction has not found widespread success in training deep neural
  networks or large language models. Consequently, it has remained a less favored
  approach in modern AI. In this paper, to unleash the power of variance
  reduction for efficient training of large models, we propose a unified
  optimization framework, MARS (Make vAriance Reduction Shine), which reconciles
  preconditioned gradient methods with variance reduction via a scaled stochastic
  recursive momentum technique. Within our framework, we introduce three
  instances of MARS that leverage preconditioned gradient updates based on AdamW,
  Lion, and Shampoo, respectively. We also draw a connection between our
  algorithms and existing optimizers. Experimental results on training GPT-2
  models indicate that MARS consistently outperforms AdamW by a large margin.
paper_ai_summary: MARS, a unified optimization framework combining preconditioned gradient methods with variance reduction, improves the training efficiency of large models like GPT-2 compared to AdamW.
paper_ai_keywords: Adam AdamW variance reduction adaptive gradient algorithms stochastic optimization convex settings nonconvex settings preconditioned gradient methods scaled stochastic recursive momentum Lion Shampoo GPT-2