Training Deep Networks without Learning Rates Through Coin Betting
Deep learning methods achieve state-of-the-art performance in many application scenarios. Yet, these methods require a significant amount of hyperparameters tuning in order to achieve the best results. In particular, tuning the learning rates in the stochastic optimization process is still one of the main bottlenecks. In this paper, we propose a new stochastic gradient descent procedure for deep networks that does not require any learning rate setting. Contrary to previous methods, we do not adapt the learning rates nor we make use of the assumed curvature of the objective function. Instead, we reduce the optimization process to a game of betting on a coin and propose a learning-rate-free optimal algorithm for this scenario. Theoretical convergence is proven for convex and quasi-convex functions and empirical evidence shows the advantage of our algorithm over popular stochastic gradient algorithms.
This paper presents an optimization strategy using coin betting, and a variant which works well for training neural networks.  The optimizer is tested and compared to various stochastic optimization routines on some simple problems.
I like this paper because it's an unusual optimization method which surprisingly seems to work reasonably well.  It also has fewer tunable parameters than existing stochastic optimizers, which is nice.  However, I'm giving it a marginal accept for the following reasons:
- The experiments aren't very convincing.  They're training some very simple models on some very small datasets, and results in this regime do not necessarily translate over to "real" problems/models.  Further, success on these problems mostly revolves around preventing overfitting.  I'd actually suggest additional experiments in _both_ directions - additional simple unit test-like experiments (like the |x - 10| experiment, but more involved - see e.g. the "unit tests for stochastic optimization" paper), and experiments on realistically large models on realistically large datasets.
- I don't 100% buy the "without learning rates" premise, for the following reason:  If you divide the right-most term of line 10 of Algorithm 2 by L_{t, i}, then the denominator simplifies to max(G_{t, i}/L_{t, i} + 1, \alpha).  So, provided that \alpha is bigger than G_{t, i}/L_{t, i}, the updates effectively are getting scaled by \alpha.  In general I would expect G_{t, i}/L_{t, i} to be smaller than \alpha = 100 particularly at the beginning of training, and as a result I'd expect the setting of \alpha to have a considerable effect on training.  Of course, if \alpha = 100 well in any experiment of interest, we can ignore it, but arguably the default learning rate setting of Adam works reasonably well in most problems of interest too -but of course, we wouldn't call it  "without learning rates" method.
- Overall, this is an unusual and unconventional idea (which is great!), but it is frankly not presented in a clear way.  I do not get a lot of intuition from the paper about _why_ this works, how it is similar/different from SGD, how the different quantities being updated (G, L, reward, \theta, w, etc) evolve over the course of a typical training run, etc. despite spending considerable time with the paper.  I would suggest adding an additional section to the appendix, or something, which gives a lot more intuition about how and why this actually works.
More specific comments:
- Very high level: "backprop without learning rates" is a strange phrase.  Backprop has no learning rate.  It's an efficient algorithm for finding the gradients with respect to all parameters in your model with respect to the output.  SGD has a learning rate.  SGD is often used for training neural networks.  In neural networks backprop is often used for finding the gradients necessary for SGD.  But you don't need a learning rate to backprop; they are disjoint concepts.
- In your introduction, it might be worth mentioning results that show that the learning rate is one of the most "important" hyperparameters, in the sense that if it's set wrong the model may not work at all, so its correct setting can have a strong effect on the outcome.  
- The relation of the second and third inequality in the proof after line 112 took me about 15 minutes to figure out/verify.  It would be helpful if you broke this into a few steps.
- Algorithm 1 is missing some input; e.g. it does not take "T" or "F" as input.
- Calling \beta_{t, i} the "fraction to bet" is odd because it can be negative, e.g. if the gradients are consistently negative then \theta_{t, i} will be negative and 2\sigma(...) - 1 will be close to -1.  So you are allowed to bet a negative amount?  Further, it seems that in general 2\theta_{t, i} can be substantially smaller than G_{t, i} + L_i
- I think you have redefined w_t.  When defining coin betting you use w_t to refer to the bet at round t.  In COCOB w_t are the model parameters, and the bet at round t is (I think) \beta_t, i (L_i + Reward_t, i).
- I think most readers will be most familiar with (S)GD as an optimization workhorse.  Further presentation of COCOB vs. variants of SGD would be helpful, e.g. drawing specific analogies between each step of COCOB and each step of some SGD-based optimizer.  Or, perhaps showing the behavior/updates of COCOB vs. SGD on additional simple examples.
- "Rather, it is big when we are far from the optimum and small when close to it."  Not necessarily - for example, in your |x - 10| example, at the beginning you are quite far from the optimum but \sum_i g_i^2 is quite small.  Really the only thing you can say about this term is that it grows with iteration t, I think.
- In Algorithm 2, the loop is just "repeat", so there is no variable T defined anywhere.  So you can't "Return w_T".  I think you mean "return w_t".  You also never increment, or initialize, t.
- Setting L_{t, i} to the running max of the gradients seems like a bad idea in practice with neural networks (particularly for RNNs) because if at some iteration gradients explode in a transient manner (a common occurrence), then for all subsequent iterations the update will be very small due to the quadratic L_{t, i} term in the denominator of line 10 of algorithm 2.  It seems like you would want to set an explicit limit as to the values of |g_{t, i}| you consider, e.g. setting L_{t, i} <- max(L_{t - 1, i}, min(|g_{t, i}|, 10)) or something.
- What version of MNIST are you using that has 55k training samples?  It technically has 60k training images, typically split into 50k for training and 10k for validation.
- I would suggest using an off-the-shelf classifier for your MNIST experiments too, since you are missing some experiment details (how were the weights initialized, etc).
- You say your convolutional kernels are of shape 5 x 5 x 3 and your pooling is 3 x 3 x 3, I think you mean 5 x 5 and 3 x 3 respectively.
- What learning rates did you use for the different optimizers and did you do any kind of search over learning rate values?  This is absolutely critical!