The Road Less Scheduled
Existing learning rate schedules that do not require specification of the
optimization stopping step T are greatly out-performed by learning rate
schedules that depend on T. We propose an approach that avoids the need for
this stopping time by eschewing the use of schedules entirely, while exhibiting
state-of-the-art performance compared to schedules across a wide family of
problems ranging from convex problems to large-scale deep learning problems.
Our Schedule-Free approach introduces no additional hyper-parameters over
standard optimizers with momentum. Our method is a direct consequence of a new
theory we develop that unifies scheduling and iterate averaging. An open source
implementation of our method is available
(https://github.com/facebookresearch/schedule_free).
66540642607894ea804f1e36
Aaron Defazio
admin_assigned
2024-05-27T08:11:23.555Z
6630035159aea64bf164b77a
/avatars/eef9a3e5bfce4d6481a395e86840d1ab.svg
Aaron Defazio
user
adefazio
66540642607894ea804f1e37
Xingyu
66540642607894ea804f1e38
Yang
66540642607894ea804f1e39
Harsh Mehta
66540642607894ea804f1e3a
Konstantin Mishchenko
admin_assigned
2024-05-27T08:12:26.655Z
645d5b49ce72244df7b347a3
https://cdn-avatars.huggingface.co/v1/production/uploads/645d5b49ce72244df7b347a3/L7UIYte82M64qPHYV5_mN.png
Konstantin Mishchenko
user
konstmish
66540642607894ea804f1e3b
Ahmed Khaled
66540642607894ea804f1e3c
Ashok Cutkosky
admin_assigned
2024-05-27T08:12:54.928Z
64d8517d0d87929689300ae5
/avatars/86ebb95ef46d62b3752fae1c67f82b1d.svg
Ashok Cutkosky
user
cutkosky
The Road Less Scheduled
Existing learning rate schedules that do not require specification of the
optimization stopping step T are greatly out-performed by learning rate
schedules that depend on T. We propose an approach that avoids the need for
this stopping time by eschewing the use of schedules entirely, while exhibiting
state-of-the-art performance compared to schedules across a wide family of
problems ranging from convex problems to large-scale deep learning problems.
Our Schedule-Free approach introduces no additional hyper-parameters over
standard optimizers with momentum. Our method is a direct consequence of a new
theory we develop that unifies scheduling and iterate averaging. An open source
implementation of our method is available
(https://github.com/facebookresearch/schedule_free).
A new schedule-free method achieves state-of-the-art performance in optimization without requiring a stopping time or additional hyper-parameters, by unifying scheduling and iterate averaging.
learning rate schedules optimization stopping step learning rate state-of-the-art hyper-parameters momentum iterate averaging schedule-free