Optimizing (L0,L1)-Smooth Functions by Gradient Methods
Daniil Vankov Anton Rodomanov Angelia Nedich Lalitha Sankar Sebastian U Stich
We study gradient methods for optimizing $(L_0, L_1)$-smooth functions, a
class that generalizes Lipschitz-smooth functions and has gained attention for
its relevance in machine learning.
We provide new insights into the structure of this function class and develop
a principled framework for analyzing optimization methods in this setting.
While our convergence rate estimates recover existing results for minimizing
the gradient norm in nonconvex problems, our approach significantly improves
the best-known complexity bounds for convex objectives.
Moreover, we show that the gradient method with Polyak stepsizes and the
normalized gradient method achieve nearly the same complexity guarantees as
methods that rely on explicit knowledge of $(L_0, L_1)$.
Finally, we demonstrate that a carefully designed accelerated gradient
method can be applied to $(L_0, L_1)$-smooth functions, further improving all
previous results.
$(L_0, L_1)$-smoothness, gradient methods, convex optimization, worst-case complexity bounds, acceleration, Polyak stepsizes, nonconvex optimization