title: Stepsize anything: A unified learning rate schedule for
    budgeted-iteration training
summary: The expanding computational costs and limited resources underscore the
  critical need for budgeted-iteration training, which aims to achieve optimal
  learning within predetermined iteration budgets.While learning rate schedules
  fundamentally govern the performance of different networks and tasks,
  particularly in budgeted-iteration scenarios, their design remains largely
  heuristic, lacking theoretical foundations.In addition, the optimal learning
  rate schedule requires extensive trial-and-error selection, making the training
  process inefficient.In this work, we propose the Unified Budget-Aware (UBA)
  schedule, a theoretically grounded learning rate schedule that consistently
  outperforms commonly-used schedules among diverse architectures and tasks under
  different constrained training budgets.First, we bridge the gap by constructing
  a novel training budget-aware optimization framework, which explicitly accounts
  for the robustness to landscape curvature variations.From this framework, we
  derive the UBA schedule, controlled by a single hyper-parameter varphi that
  provides a trade-off between flexibility and simplicity, eliminating the need
  for per-network numerical optimization. Moreover, we establish a theoretical
  connection between varphi and the condition number, adding interpretation
  and justification to our approach. Besides, we prove the convergence for
  different values of varphi.We offer practical guidelines for its selection
  via theoretical analysis and empirical results.xtensive experimental results
  show that UBA consistently surpasses the commonly-used schedules
  across diverse vision and language tasks, spanning network architectures (e.g.,
  ResNet, OLMo) and scales, under different training-iteration budgets.
paper_authors._id: 683e8abd8e6e97efe0bf20d9
paper_authors.name: Anda Tang
paper_authors._id: 683e8abd8e6e97efe0bf20da
paper_authors.name: Yiming Dong
paper_authors._id: 683e8abd8e6e97efe0bf20db
paper_authors.name: Yutao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:39:15.864Z
paper_authors._id: 6371128eafbe42caa5a5222b
paper_authors.avatarUrl: /avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
paper_authors.fullname: Yutao Zeng
paper_authors.type: user
paper_authors.user: Taoer
paper_authors._id: 683e8abd8e6e97efe0bf20dc
paper_authors.name: zhou Xun
paper_authors._id: 683e8abd8e6e97efe0bf20dd
paper_authors.name: Zhouchen Lin
paper_title: Stepsize anything: A unified learning rate schedule for
    budgeted-iteration training
paper_summary: The expanding computational costs and limited resources underscore the
  critical need for budgeted-iteration training, which aims to achieve optimal
  learning within predetermined iteration budgets.While learning rate schedules
  fundamentally govern the performance of different networks and tasks,
  particularly in budgeted-iteration scenarios, their design remains largely
  heuristic, lacking theoretical foundations.In addition, the optimal learning
  rate schedule requires extensive trial-and-error selection, making the training
  process inefficient.In this work, we propose the Unified Budget-Aware (UBA)
  schedule, a theoretically grounded learning rate schedule that consistently
  outperforms commonly-used schedules among diverse architectures and tasks under
  different constrained training budgets.First, we bridge the gap by constructing
  a novel training budget-aware optimization framework, which explicitly accounts
  for the robustness to landscape curvature variations.From this framework, we
  derive the UBA schedule, controlled by a single hyper-parameter varphi that
  provides a trade-off between flexibility and simplicity, eliminating the need
  for per-network numerical optimization. Moreover, we establish a theoretical
  connection between varphi and the condition number, adding interpretation
  and justification to our approach. Besides, we prove the convergence for
  different values of varphi.We offer practical guidelines for its selection
  via theoretical analysis and empirical results.xtensive experimental results
  show that UBA consistently surpasses the commonly-used schedules
  across diverse vision and language tasks, spanning network architectures (e.g.,
  ResNet, OLMo) and scales, under different training-iteration budgets.
paper_ai_summary: A unified budget-aware learning rate schedule is proposed to optimize training within limited iteration budgets, outperforming traditional schedules across various tasks and network architectures.
paper_ai_keywords: budgeted-iteration training learning rate schedules Unified Budget-Aware (UBA) schedule training budget-aware optimization framework robustness to landscape curvature variations condition number convergence ResNet OLMo