Hyperparameter tuning is a bothersome step in the training of deep learning models. One of the most sensitive hyperparameters is the learning rate of the gradient descent. We present the All Learning Rates At Once (Alrao) optimization method for neural networks: each unit or feature in the network gets its own learning rate sampled from a random distribution spanning several orders of magnitude. This comes at practically no computational cost. Perhaps surprisingly, stochastic gradient descent (SGD) with Alrao performs close to SGD with an optimally tuned learning rate, for various architectures and problems. Notably, Alrao is robust: all runs were able to learn without any tuning. This makes Alrao a candidate for situations in which a neural network has to work from the first try. This text comes with a PyTorch implementation of the method, which can be plugged on an existing PyTorch model.
POST REBUTTAL: I think the paper is decent, there are some significant downsides to the method but it could constitute a first step towards a more mature learning-rate-free method. However, in its current state the paper is left with some gaping holes in its experiment section. The authors tried to add experiments on Imagenet, but these experiments apparently didn't finish before the end of the rebuttal period. For that reason, the paper probably should not be accepted for publication (even if the authors manage to finish running these experiments, we would not have a chance to review these results). 
------
Original review
------
In this paper, the authors present a method for training deep networks with randomly sampled feature-wise learning rates, removing the need for fixed learning rates and their tuning. The method is shown to perform comparatively to SGD with a learning rate roughly optimized with regards to validation performance. The method applies to the most popular types of deep learning architectures, which includes fully connected layers, convolutional layers and recurrent cells. 
Quality: The paper is of a decent quality in general, I noticed no glaring omissions while reading the paper. However, I do worry that the method provides little gain for a lot of work. It is becoming more and more easy to tune the learning rate of deep learning models with strategies such as early stopping, and this method comes at a high cost for models with a big final layer. 
Clarity: The paper is well written, but the reader is often (too often?) sent to the Appendix, which is itself ordered in a strange way (e.g., the first reference to the Appendix in the paper refers to Appendix F?). If some sections of the Appendix are not needed, I would remove them. 
Originality: The work is original in the approach, i.e. randomization as a way to get rid of learning rates is a novel method. However, there was one work presented last year at NIPS which concerns itself with the same problem, which is getting rid of learning rates:
“Training Deep Networks without Learning Rates Through Coin Betting” by Francesco Orabona and Tatiana Tommasi, NIPS, 2017.
They don’t compare on the same methods and the same datasets, but I think the authors should be aware of this work and perhaps compare themselves with it. The work takes a very different approach to solve the problem so I don’t think it’s an issue for this paper. 
Significance: I think the work is important, in that it adds another tool to solve the learning rate problem. I would not say it is likely to have a very high impact, because it involves a lot of work, for little benefit. Furthermore, the cost of reproducing multiple times the last layer of the network will be prohibitive in many cases for NLP. 
The method feels ad-hoc in many respects, and there are no guarantees that it would work any better than Adam does on pathological cases. Perhaps some mathematical analysis on simpler problems would help make the contributions stronger. 
The authors state that the learning rate range has little impact on performance, yet it still has enough impact to justify tuning it for different models and datasets (on CIFAR it is 10^-5 to 10^1, on Pennbank it is 10^-3 to 10^2). I would tend to agree that the alrao method is more robust to the choice of learning rate than plain SGD, however the fact of the matter is that there are still parameters to tune. 
Figure 5. also seems to suggest that the range is important, although the models were not trained until the end, so it is not clear.
Some additional comments:
Nitpicking: In Section 2, most sub-sections (or paragraphs titles?) have the name of the method in them. That’s redundant. Instead of “Alrao principle”, “Alrao update”, etc., just write “Principle.”, “Update.”.
Is there a justification for using the same learning rate for all weights in an LSTM unit? 
I believe there is a mistake in Equation 2. The denominator should be log(\eta_{max}) - log(\eta_{min})
[second paragraph on page 4.] Once again nitpicking for the sake of clarity: “For each classifier Cθ cl j, we set a learning rate log eta_j = …” this reads as if the learning rate would be set to log eta_j, but you probably mean you will set the learning rate to eta_j = exp(...).
Figure 5b in the appendix does not specify which curve has which learning rate interval.