title: Cautious Optimizers: Improving Training with One Line of Code
summary: AdamW has been the default optimizer for transformer pretraining. For many
  years, our community searches for faster and more stable optimizers with only
  constraint positive outcomes. In this work, we propose a single-line
  modification in Pytorch to any momentum-based optimizer, which we rename
  Cautious Optimizer, e.g. C-AdamW and C-Lion. Our theoretical result shows that
  this modification preserves Adam's Hamiltonian function and it does not break
  the convergence guarantee under the Lyapunov analysis. In addition, a whole new
  family of optimizers is revealed by our theoretical insight. Among them, we
  pick the simplest one for empirical experiments, showing speed-up on Llama and
  MAE pretraining up to 1.47times. Code is available at
  https://github.com/kyleliang919/C-Optim
paper_authors._id: 67453b867baa7dd7dd0a5e4c
paper_authors.name: Kaizhao Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:52:03.230Z
paper_authors._id: 62140dcdcf7928035e8135ad
paper_authors.avatarUrl: /avatars/146a8ac3fb476e79b6503373df0f38c1.svg
paper_authors.fullname: Kaizhao Liang
paper_authors.type: user
paper_authors.user: kz919
paper_authors._id: 67453b867baa7dd7dd0a5e4d
paper_authors.name: Lizhang Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T08:51:12.625Z
paper_authors._id: 657ac67b5b9c86c08f2c073d
paper_authors.avatarUrl: /avatars/a6770776bba02413123d53eb020240b4.svg
paper_authors.fullname: 'Tim' Lizhang Chen
paper_authors.type: user
paper_authors.user: lzchenut
paper_authors._id: 67453b867baa7dd7dd0a5e4e
paper_authors.name: Bo Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T09:09:57.439Z
paper_authors._id: 64375a5ab8ea86e50329d2da
paper_authors.avatarUrl: /avatars/b3dcef9e2e0234156a22d699a30b6f1f.svg
paper_authors.fullname: Bo Liu
paper_authors.type: user
paper_authors.user: cranial-xix
paper_authors._id: 67453b867baa7dd7dd0a5e4f
paper_authors.name: Qiang Liu
paper_title: Cautious Optimizers: Improving Training with One Line of Code
paper_summary: AdamW has been the default optimizer for transformer pretraining. For many
  years, our community searches for faster and more stable optimizers with only
  constraint positive outcomes. In this work, we propose a single-line
  modification in Pytorch to any momentum-based optimizer, which we rename
  Cautious Optimizer, e.g. C-AdamW and C-Lion. Our theoretical result shows that
  this modification preserves Adam's Hamiltonian function and it does not break
  the convergence guarantee under the Lyapunov analysis. In addition, a whole new
  family of optimizers is revealed by our theoretical insight. Among them, we
  pick the simplest one for empirical experiments, showing speed-up on Llama and
  MAE pretraining up to 1.47times. Code is available at
  https://github.com/kyleliang919/C-Optim
paper_ai_keywords: Cautious Optimizer C-AdamW C-Lion Adam's Hamiltonian function Lyapunov analysis momentum-based optimizer Llama MAE pretraining