ABSTRACT Wilson et al. (2017) showed that, when the stepsize schedule is properly designed, stochastic gradient generalizes better than ADAM (Kingma & Ba, 2014). In light of recent work on hypergradient methods (Baydin et al., 2018), we revisit these claims to see if such methods close the gap between the most popular optimizers.
As a byproduct, we analyze the true benefit of these hypergradient methods compared to more classical schedules, such as the fixed decay of Wilson et al. (2017). In particular, we observe they are of marginal help since their performance varies significantly when tuning their hyperparameters. Finally, as robustness is a critical quality of an optimizer, we provide a sensitivity analysis of these gradient based optimizers to assess how challenging their tuning is.
Clarity: Below average
- The introduction would be easier to follow if you named Baydin's approach and your own approach, because in the 2-4 bullet points you say "this online scheme", and "the learning rate schedule", without being perfectly clear what you are talking about
- The last sentence of the introduction is meant to clearly state your hypothesis, so I was expecting "emphasize the value of *", i.e. either adaptive or non-adaptive methods, rather than just general 'tuning', which is self-apparently important.
Quality: Below average
This is a purely empirical study that does not go too deep. It is not quite a review paper, but only compares previous methods.
Pros:
I especially appreciate the sensitivity analysis, ie Fig 6. If only all ML papers had something like this to suggest the difficulty of setting hyperparameters for their proposed methods.
Cons:
- You should use mathematics to describe what you are talking about with adaptive stepsize in Sec 2.1. "these methods multiply the gradient with a matrix". Just giving one equation would be extremely helpful.
- If I understand correctly, you are interpreting the inverse-Hessian as used in Newton's method and other non-diagonal 'gradient conditioners' as types of stepsize. This is definitely interesting, but again it would be very simple to see what you are saying with an equation instead of starting with the phrase "stepsize" which is generally understood to be a scalar multiple on the gradient.
- I'm surprised you jump right into experiements after your background settings. It's apparent that this paper fundamentally relies on the Wilson (2017) hypergradient paper. Your paper should be more self-contained: 'hypergradient' is not even defined in this paper, is it?...
Especially:
How do you know that if you change the model architecture, data, and loss, that a similar result will occur? I imagine that it heavily relies on the data and model-- in other words, that the sensitivity is dependent on "how an algorithm reacts to a certain data/loss/model landscape". I'm trying to say that I'm not convinced these results generalize to any other situation than the one presented here (so does it really say anything about the different stepsize selection rules?)
Random side note:
Since your appendix is only a few lines, you could consider succinctly listing learning rates with set notation, for example {1e-n,5e-n : -5<n<1}.