Stepsize anything: A unified learning rate schedule for
  budgeted-iteration training
The expanding computational costs and limited resources underscore the
critical need for budgeted-iteration training, which aims to achieve optimal
learning within predetermined iteration budgets.While learning rate schedules
fundamentally govern the performance of different networks and tasks,
particularly in budgeted-iteration scenarios, their design remains largely
heuristic, lacking theoretical foundations.In addition, the optimal learning
rate schedule requires extensive trial-and-error selection, making the training
process inefficient.In this work, we propose the Unified Budget-Aware (UBA)
schedule, a theoretically grounded learning rate schedule that consistently
outperforms commonly-used schedules among diverse architectures and tasks under
different constrained training budgets.First, we bridge the gap by constructing
a novel training budget-aware optimization framework, which explicitly accounts
for the robustness to landscape curvature variations.From this framework, we
derive the UBA schedule, controlled by a single hyper-parameter varphi that
provides a trade-off between flexibility and simplicity, eliminating the need
for per-network numerical optimization. Moreover, we establish a theoretical
connection between varphi and the condition number, adding interpretation
and justification to our approach. Besides, we prove the convergence for
different values of varphi.We offer practical guidelines for its selection
via theoretical analysis and empirical results.xtensive experimental results
show that UBA consistently surpasses the commonly-used schedules
across diverse vision and language tasks, spanning network architectures (e.g.,
ResNet, OLMo) and scales, under different training-iteration budgets.
683e8abd8e6e97efe0bf20d9
Anda Tang
683e8abd8e6e97efe0bf20da
Yiming Dong
683e8abd8e6e97efe0bf20db
Yutao Zeng
claimed_verified
2025-06-03T08:39:15.864Z
6371128eafbe42caa5a5222b
/avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
Yutao Zeng
user
Taoer
683e8abd8e6e97efe0bf20dc
zhou Xun
683e8abd8e6e97efe0bf20dd
Zhouchen Lin
Stepsize anything: A unified learning rate schedule for
  budgeted-iteration training
The expanding computational costs and limited resources underscore the
critical need for budgeted-iteration training, which aims to achieve optimal
learning within predetermined iteration budgets.While learning rate schedules
fundamentally govern the performance of different networks and tasks,
particularly in budgeted-iteration scenarios, their design remains largely
heuristic, lacking theoretical foundations.In addition, the optimal learning
rate schedule requires extensive trial-and-error selection, making the training
process inefficient.In this work, we propose the Unified Budget-Aware (UBA)
schedule, a theoretically grounded learning rate schedule that consistently
outperforms commonly-used schedules among diverse architectures and tasks under
different constrained training budgets.First, we bridge the gap by constructing
a novel training budget-aware optimization framework, which explicitly accounts
for the robustness to landscape curvature variations.From this framework, we
derive the UBA schedule, controlled by a single hyper-parameter varphi that
provides a trade-off between flexibility and simplicity, eliminating the need
for per-network numerical optimization. Moreover, we establish a theoretical
connection between varphi and the condition number, adding interpretation
and justification to our approach. Besides, we prove the convergence for
different values of varphi.We offer practical guidelines for its selection
via theoretical analysis and empirical results.xtensive experimental results
show that UBA consistently surpasses the commonly-used schedules
across diverse vision and language tasks, spanning network architectures (e.g.,
ResNet, OLMo) and scales, under different training-iteration budgets.
A unified budget-aware learning rate schedule is proposed to optimize training within limited iteration budgets, outperforming traditional schedules across various tasks and network architectures.
budgeted-iteration training learning rate schedules Unified Budget-Aware (UBA) schedule training budget-aware optimization framework robustness to landscape curvature variations condition number convergence ResNet OLMo