title: The Road Less Scheduled
summary: Existing learning rate schedules that do not require specification of the
  optimization stopping step T are greatly out-performed by learning rate
  schedules that depend on T. We propose an approach that avoids the need for
  this stopping time by eschewing the use of schedules entirely, while exhibiting
  state-of-the-art performance compared to schedules across a wide family of
  problems ranging from convex problems to large-scale deep learning problems.
  Our Schedule-Free approach introduces no additional hyper-parameters over
  standard optimizers with momentum. Our method is a direct consequence of a new
  theory we develop that unifies scheduling and iterate averaging. An open source
  implementation of our method is available
  (https://github.com/facebookresearch/schedule_free).
paper_authors._id: 66540642607894ea804f1e36
paper_authors.name: Aaron Defazio
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T08:11:23.555Z
paper_authors._id: 6630035159aea64bf164b77a
paper_authors.avatarUrl: /avatars/eef9a3e5bfce4d6481a395e86840d1ab.svg
paper_authors.fullname: Aaron Defazio
paper_authors.type: user
paper_authors.user: adefazio
paper_authors._id: 66540642607894ea804f1e37
paper_authors.name: Xingyu
paper_authors._id: 66540642607894ea804f1e38
paper_authors.name: Yang
paper_authors._id: 66540642607894ea804f1e39
paper_authors.name: Harsh Mehta
paper_authors._id: 66540642607894ea804f1e3a
paper_authors.name: Konstantin Mishchenko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T08:12:26.655Z
paper_authors._id: 645d5b49ce72244df7b347a3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/645d5b49ce72244df7b347a3/L7UIYte82M64qPHYV5_mN.png
paper_authors.fullname: Konstantin Mishchenko
paper_authors.type: user
paper_authors.user: konstmish
paper_authors._id: 66540642607894ea804f1e3b
paper_authors.name: Ahmed Khaled
paper_authors._id: 66540642607894ea804f1e3c
paper_authors.name: Ashok Cutkosky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T08:12:54.928Z
paper_authors._id: 64d8517d0d87929689300ae5
paper_authors.avatarUrl: /avatars/86ebb95ef46d62b3752fae1c67f82b1d.svg
paper_authors.fullname: Ashok Cutkosky
paper_authors.type: user
paper_authors.user: cutkosky
paper_title: The Road Less Scheduled
paper_summary: Existing learning rate schedules that do not require specification of the
  optimization stopping step T are greatly out-performed by learning rate
  schedules that depend on T. We propose an approach that avoids the need for
  this stopping time by eschewing the use of schedules entirely, while exhibiting
  state-of-the-art performance compared to schedules across a wide family of
  problems ranging from convex problems to large-scale deep learning problems.
  Our Schedule-Free approach introduces no additional hyper-parameters over
  standard optimizers with momentum. Our method is a direct consequence of a new
  theory we develop that unifies scheduling and iterate averaging. An open source
  implementation of our method is available
  (https://github.com/facebookresearch/schedule_free).
paper_ai_summary: A new schedule-free method achieves state-of-the-art performance in optimization without requiring a stopping time or additional hyper-parameters, by unifying scheduling and iterate averaging.
paper_ai_keywords: learning rate schedules optimization stopping step learning rate state-of-the-art hyper-parameters momentum iterate averaging schedule-free