L4: Practical loss-based stepsize adaptation for deep learning
We propose a stepsize adaptation scheme for stochastic gradient descent. It operates directly with the loss function and rescales the gradient in order to make fixed predicted progress on the loss. We demonstrate its capabilities by conclusively improving the performance of Adam and Momentum optimizers. The enhanced optimizers with default hyperparameters consistently outperform their constant stepsize counterparts, even the best ones, without a measurable increase in computational cost. The performance is validated on multiple architectures including dense nets, CNNs, ResNets, and the recurrent Differential Neural Computer on classical datasets MNIST, fashion MNIST, CIFAR10 and others.
The paper proposes a scheme for adaptive choice of learning rate for stochastic gradients descent and its variants. The key idea is very simple and easy to implement: given the loss value L at the global minimum, L_min, the idea is to choose learning rate eta, such that the update along the gradient reaches L_min from the current point i.e. solving L(theta-eta*v)=L_min in eta, where v is for example dL/dtheta in gradient descent. Due to nonlinearity of L, the authors suggest linearizing L around theta: L(theta)+(-eta*v)^T dL/dtheta=L_min which yields eta=(L(theta)-L_min)/(v^T dL/dtheta). Finally, to make the adaptive learning rate pessimistic to the possible linearization error, the authors introduce a coefficient alpha, so the effective learning rate used by the optimizer is eta*alpha.  The authors empirically show (on badly conditioned regression, MNIST, CIFAR-10, and neural computer) that using such adaptive scheme helps in two ways: 1. the optimization performance is less sensitive to the choice of the coefficient alpha vs the learning rate (in non-adaptive setting), and 2. the optimizer can reduce the loss faster or at worst in equal speed with commonly used optimizers.  At the same time, the paper has some shortcomings as admitted by the authors: 1. There is no theoretical justification for the proposed scheme. 2. Despite training faster, the resulted model may generalize worse than models trained with non-adaptive step size.  In spite of the above shortcomings, I think this is a nice paper which may motivate further research around the proposed idea and eventually create an impact.  Some comments that may improve the manuscript: 1. I think providing a graphical illustration on the trajectories taken by this scheme vs plain gradient descent can be very insightful and maybe provide some intuition why the idea works. Perhaps this can be shown on a simple task: a poorly conditioned quadratic function. We know that in this case the gradient may not necessarily point toward the minimum. So it is interesting to visually inspect how by merely modifying the step size, and yet moving along the gradient directions over this poorly condition surface, the adaptive step size method can reach the minimum faster. 2. For CIFAR-10 graphs (Figure 5), what happens after 250 epochs? Is it that all methods eventually converge to the same loss value or some methods get stuck at larger values?  One major concern I have about this paper is the way L_min is used. Is L_min defined locally (i.e. the minimal point along the gradient ray only) or globally (over the entire space)? I suppose it is global, because the notation does not show any dependency of L_min on theta. If it is defined globally, then how would you justify solving L(theta-eta*v)=L_min, because this equation may not even have a solution( and I think often it does not!). For example, for a poorly conditioned quadratic, the direction of gradient may not necessarily point toward the minimum. Hence, no matter how much you move along the gradient direction, you never reach the minimum value L_min. Therefore, It seems the idea of solving  L(theta-eta*v)=L_min has some fundamental problem.  POST REBUTTAL COMMENTS: I read author's response. I was not convinced about L_min justification. Specially, the fact that setting L_min to "0" did not work shows the crticality of L_min for the algorithm to work. However, I am also not aware of a cheap workaround, so I believe the setting is okay although not idea. I keep my rating as it was.