  Many machine learning applications and tasks rely on the stochastic gradient
descent (SGD) algorithm and its variants. Effective step length selection is
crucial for the success of these algorithms, which has motivated the
development of algorithms such as ADAM or AdaGrad. In this paper, we propose a
novel algorithm for adaptive step length selection in the classical SGD
framework, which can be readily adapted to other stochastic algorithms. Our
proposed algorithm is inspired by traditional nonlinear optimization techniques
and is supported by analytical findings. We show that under reasonable
conditions, the algorithm produces step lengths in line with well-established
theoretical requirements, and generates iterates that converge to a stationary
neighborhood of a solution in expectation. We test the proposed algorithm on
logistic regressions and deep neural networks and demonstrate that the
algorithm can generate step lengths comparable to the best step length obtained
from manual tuning.

Machine Learning;Artificial Intelligence;Computer Vision and Pattern Recognition;Optimization and Control