Cautious Optimizers: Improving Training with One Line of Code
AdamW has been the default optimizer for transformer pretraining. For many
years, our community searches for faster and more stable optimizers with only
constraint positive outcomes. In this work, we propose a single-line
modification in Pytorch to any momentum-based optimizer, which we rename
Cautious Optimizer, e.g. C-AdamW and C-Lion. Our theoretical result shows that
this modification preserves Adam's Hamiltonian function and it does not break
the convergence guarantee under the Lyapunov analysis. In addition, a whole new
family of optimizers is revealed by our theoretical insight. Among them, we
pick the simplest one for empirical experiments, showing speed-up on Llama and
MAE pretraining up to 1.47times. Code is available at
https://github.com/kyleliang919/C-Optim
67453b867baa7dd7dd0a5e4c
Kaizhao Liang
admin_assigned
2024-11-26T22:52:03.230Z
62140dcdcf7928035e8135ad
/avatars/146a8ac3fb476e79b6503373df0f38c1.svg
Kaizhao Liang
user
kz919
67453b867baa7dd7dd0a5e4d
Lizhang Chen
claimed_verified
2024-12-02T08:51:12.625Z
657ac67b5b9c86c08f2c073d
/avatars/a6770776bba02413123d53eb020240b4.svg
'Tim' Lizhang Chen
user
lzchenut
67453b867baa7dd7dd0a5e4e
Bo Liu
claimed_verified
2024-12-06T09:09:57.439Z
64375a5ab8ea86e50329d2da
/avatars/b3dcef9e2e0234156a22d699a30b6f1f.svg
Bo Liu
user
cranial-xix
67453b867baa7dd7dd0a5e4f
Qiang Liu
Cautious Optimizers: Improving Training with One Line of Code
AdamW has been the default optimizer for transformer pretraining. For many
years, our community searches for faster and more stable optimizers with only
constraint positive outcomes. In this work, we propose a single-line
modification in Pytorch to any momentum-based optimizer, which we rename
Cautious Optimizer, e.g. C-AdamW and C-Lion. Our theoretical result shows that
this modification preserves Adam's Hamiltonian function and it does not break
the convergence guarantee under the Lyapunov analysis. In addition, a whole new
family of optimizers is revealed by our theoretical insight. Among them, we
pick the simplest one for empirical experiments, showing speed-up on Llama and
MAE pretraining up to 1.47times. Code is available at
https://github.com/kyleliang919/C-Optim
A single-line modification to momentum-based optimizers preserves Adam's convergence properties and achieves significant speed-up in transformer pretraining.
AdamW momentum-based optimizer Cautious Optimizer C-AdamW C-Lion Hamiltonian function Lyapunov analysis speed-up Llama MAE pretraining