ONLINE LEARNING RATE ADAPTATION WITH HYPERGRADIENT DESCENT
We introduce a general method for improving the convergence rate of gradientbased optimizers that is easy to implement and works well in practice. We demonstrate the effectiveness of the method in a range of optimization problems by applying it to stochastic gradient descent, stochastic gradient descent with Nesterov momentum, and Adam, showing that it significantly reduces the need for the manual tuning of the initial learning rate for these commonly used algorithms. Our method works by dynamically updating the learning rate during optimization using the gradient with respect to the learning rate of the update rule itself. Computing this "hypergradient" needs little additional computation, requires only one extra copy of the original gradient to be stored in memory, and relies upon nothing more than what is provided by reverse-mode automatic differentiation.
SUMMARY:
The authors reinvent a 20 years old technique for adapting a global or component-wise learning rate for gradient descent. The technique can be derived as a gradient step for the learning rate hyperparameter, or it can be understood as a simple and efficient adaptation technique.
GENERAL IMPRESSION:
One central problem of the paper is missing novelty. The authors are well aware of this. They still manage to provide added value.
Despite its limited novelty, this is a very interesting and potentially impactful paper. I like in particular the detailed discussion of related work, which includes some frequently overlooked precursors of modern methods.
CRITICISM:
The experimental evaluation is rather solid, but not perfect. It considers three different problems: logistic regression (a convex problem), and dense as well as convolutional networks. That's a solid spectrum. However, it is not clear why the method is tested only on a single data set: MNIST. Since it is entirely general, I would rather expect a test on a dozen different data sets. That would also tell us more about a possible sensitivity w.r.t. the hyperparameters \alpha_0 and \beta.
The extensions in section 5 don't seem to be very useful. In particular, I cannot get rid of the impression that section 5.1 exists for the sole purpose of introducing a convergence theorem. Analyzing the actual adaptive algorithm would be very interesting. In contrast, the present result is trivial and of no interest at all, since it requires knowing a good parameter setting, which defeats a large part of the value of the method.
MINOR POINTS:
page 4, bottom: use \citep for Duchi et al. (2011).
None of the figures is legible on a grayscale printout of the paper. Please do not use color as the only cue to identify a curve.
In figure 2, top row, please display the learning rate on a log scale.
page 8, line 7 in section 4.3: "the the" (unintended repetition)
End of section 4: an increase from 0.001 to 0.001002 is hardly worth reporting - or am I missing something?