One very important hyperparameter for training deep neural networks is the learning rate of the optimizer. The choice of learning rate schedule determines the computational cost of getting close to a minima, how close you actually get to the minima, and most importantly the kind of local minima (wide/narrow) attained. The kind of minima attained has a significant impact on the generalization accuracy of the network. Current systems employ hand tuned learning rate schedules, which are painstakingly tuned for each network and dataset. Given that the state space of schedules is huge, finding a satisfactory learning rate schedule can be very time consuming. In this paper, we present AutoLR, a method for auto-tuning the learning rate as training proceeds. Our method works with any optimizer, and we demonstrate results on SGD, Momentum, and Adam optimizers. We extensively evaluate AutoLR on multiple datasets, models, and across multiple optimizers. We compare favorably against state of the art learning rate schedules for the given dataset and models, including for ImageNet on Resnet-50, Cifar-10 on Resnet-18, and SQuAD fine-tuning on BERT. For example, AutoLR achieves an EM score of 81.2 on SQuAD v1.1 with BERT BASE compared to 80.8 reported in (Devlin et al. (2018)) by just auto-tuning the learning rate schedule. To the best of our knowledge, this is the first automatic learning rate tuning scheme to achieve state of the art generalization accuracy on these datasets with the given models.
The authors propose an automatic learning rate schedule based on an explore (always increase LR initially) and then exploit (more typical patience based decay) strategy. The strategy seems to factor in recent understanding of deep learning optimization, and I was very much convinced by the overall idea, even if not fully convinced by the motivation. The main issues with the paper I have is that (1) I does not compare to a strong enough baseline (one cycle), and (2) the schedule does not feel automatic enough to call it "automatic"; see detailed comments. Finally, I found the remarks and analysis regarding width of the minima, and the stipulation that there are more narrow minimas, not substantiated and even contradicatory to some of the results in the literature. Based on this, I am currently leaning towards rejecting the paper. I am willing to raise my score if issues with the experimental setting are addressed.
Detailed comments:
1. I am not convinced that the baseline is strong enough.
While I appreciate the extensive range of architecture and datasets, I am not convinced that the baseline schedule tested against in each setting is actually "state of the art" as stated in the abstract. To the best of my knowledge, the state-of-the-art learning rate schedule, which uses a similar computational budget as the proposed method, is one cycle. In particular, one cycle uses LR range test to select the appropriate starting learning rate and then warmups to it gradually; see [1] for more details (see also [2]). 
Note that one cycle includes warming up the learning rate, which is another motivation for including it. As seen for instance in Fig. 4 the proposed AutoLR can increase learning rate initially, while the hand tuned schedules don't. This does not seem to be a fair comparison, given that the state of the art schedules in vision models very often do warm-up the learning rate.
2. The learning rate schedule requires feeding in "seed" learning rate. Is it an automatic learning rate schedule then? I am a bit confused by this claim. Could you please clarify what is the claim about the method?
3. To sum up point 1 and 2, while I like what the paper set out to do, I think that it is key to either (1) demonstrate that the automatic schedule is substantially better than the state of the art schedule, or (2) demonstrate that initial seed learning rate is not an important hyperparameter. If both are not true, then how would you convince the reader to use the method in practice?
4. It is implicitely assumed in the analysis that wide minima are good for generalization. It seems to me that recent evidence points towards the direction that width of the minima is a epiphenomenon, see [3,4,5,6] experiments. I think at least a discussion of (some) of these papers is very important. Also it is worth noting that [5] studied how a high learning rate (or a small batch size) enters wide regions of the loss surface early.  
5. "In that respect, an interesting intuitive observation is that a large learning rate can escape narrow minima “valleys” easily (as the optimizer can jump out of them with large steps), however once it reaches a wide minima “valley”, it is likely to get stuck in it (if the “width” of the wide valley is large compared to the step size).". I think [5,8] should be cited here. Both papers studied how learning rate selects minima shape.
6. At the same time, I am not convinced by the experimental data for "We hypothesize that for deep neural networks, narrow minimas far outnumber the wide minimas.":
6a) Taking on the face value that the experiment shows that a low learnring rate can find a wide minima, this doesn't prove the hypothesis. At best (i.e. assuming the indeed there is a strong correlation between width of the low lr minima, and generalization) it proves that low learning rate can find wide minima. It doesn't establish any bound/estimation on the relative number of narrow to wide minima. What if low learning rate by default is driven to narrow minima, and high learning rate is driven to wide minima, but the two are equal in number? I think a similar experimental data would be observed in such a null world.
6b) More importantly, it is not demonstrated that the one experiment that worked better had lower curvature. It seems implicitely assumed that if it worked better then it for sure ended up in a wider minima. I do not think this is a valid reasoning given the weak data for a causal relationship between curvature and generalization (see 2). Could you please report curvature explicitely, rather than making this assumption?
References:
[1] Fast.ai documentation on one cycle method, https://docs.fast.ai/callbacks.one_cycle.html
[2] Arpit et al, Walk with SGD, https://arxiv.org/abs/1802.08770
[3] Golatkar et al, Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence, https://arxiv.org/abs/1905.13277
[4] Yoshida et al, Spectral Norm Regularization for Improving the Generalizability of Deep Learning, https://arxiv.org/pdf/1705.10941.pdf
[5] Jastrzebski et al, On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length, https://arxiv.org/abs/1807.05031
[6] Guiroy et al, Towards Understanding Generalization in Gradient-Based Meta-Learning, https://arxiv.org/abs/1907.07287
[7] Wang et al, Identifying generalization properties in neural networks, https://arxiv.org/pdf/1809.07402.pdf
[8] Wu et al, How SGD Selects the Global Minima in Over-parameterized Learning: A Dynamical Stability Perspective, https://papers.nips.cc/paper/8049-how-sgd-selects-the-global-minima-in-over-parameterized-learning-a-dynamical-stability-perspective.pdf