title: The Surprising Agreement Between Convex Optimization Theory and
    Learning-Rate Scheduling for Large Model Training
summary: We show that learning-rate schedules for large model training behave
  surprisingly similar to a performance bound from non-smooth convex optimization
  theory. We provide a bound for the constant schedule with linear cooldown; in
  particular, the practical benefit of cooldown is reflected in the bound due to
  the absence of logarithmic terms. Further, we show that this surprisingly close
  match between optimization theory and practice can be exploited for
  learning-rate tuning: we achieve noticeable improvements for training 124M and
  210M Llama-type models by (i) extending the schedule for continued training
  with optimal learning-rate, and (ii) transferring the optimal learning-rate
  across schedules.
paper_authors._id: 67a0e794042d0e5936db83cf
paper_authors.name: Fabian Schaipp
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-03T16:00:56.014Z
paper_authors._id: 64a2b68da0696e0a29739349
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a2b68da0696e0a29739349/wUtx3yCcSiN7SZP4nMSPK.png
paper_authors.fullname: Fabian S
paper_authors.type: user
paper_authors.user: fabian-sp
paper_authors._id: 67a0e794042d0e5936db83d0
paper_authors.name: Alexander Hägele
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-03T16:01:03.085Z
paper_authors._id: 65785d22dddc2360b01702e1
paper_authors.avatarUrl: /avatars/8e3ddf25b9c423f57484fddef4f0aafd.svg
paper_authors.fullname: Alexander Hägele
paper_authors.type: user
paper_authors.user: haeggee
paper_authors._id: 67a0e794042d0e5936db83d1
paper_authors.name: Adrien Taylor
paper_authors._id: 67a0e794042d0e5936db83d2
paper_authors.name: Umut Simsekli
paper_authors._id: 67a0e794042d0e5936db83d3
paper_authors.name: Francis Bach
paper_title: The Surprising Agreement Between Convex Optimization Theory and
    Learning-Rate Scheduling for Large Model Training
paper_summary: We show that learning-rate schedules for large model training behave
  surprisingly similar to a performance bound from non-smooth convex optimization
  theory. We provide a bound for the constant schedule with linear cooldown; in
  particular, the practical benefit of cooldown is reflected in the bound due to
  the absence of logarithmic terms. Further, we show that this surprisingly close
  match between optimization theory and practice can be exploited for
  learning-rate tuning: we achieve noticeable improvements for training 124M and
  210M Llama-type models by (i) extending the schedule for continued training
  with optimal learning-rate, and (ii) transferring the optimal learning-rate
  across schedules.
paper_ai_summary: Learning-rate schedules for large deep learning models align closely with non-smooth convex optimization theory, allowing for improved training and tuning.
paper_ai_keywords: learning-rate schedules non-smooth convex optimization linear cooldown optimal learning-rate Llama-type models