In this paper, we show a phenomenon, which we named "super-convergence", where residual networks can be trained using an order of magnitude fewer iterations than is used with standard training methods. The existence of super-convergence is relevant to understanding why deep networks generalize well. One of the key elements of super-convergence is training with cyclical learning rates and a large maximum learning rate. Furthermore, we present evidence that training with large learning rates improves performance by regularizing the network. In addition, we show that super-convergence provides a greater boost in performance relative to standard training when the amount of labeled training data is limited. We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate. The architectures to replicate this work will be made available upon publication.
In this paper, the authors analyze training of residual networks using large cyclic learning rates (CLR). The authors demonstrate (a) fast convergence with cyclic learning rates and (b) evidence of large learning rates acting as regularization which improves performance on test sets – this is called “super-convergence”. However, both these effects are only shown on a specific dataset, architecture, learning algorithm and hyper parameter setting. 
Some specific comments by sections:
2. Related Work: This section loosely mentions other related works on SGD, topology of loss function and adaptive learning rates. The authors mention Loshchilov & Hutter in next section but do not compare it to their work. The authors do not discuss a somewhat contradictory claim from NIPS 2017 (as pointed out in the public comment): http://papers.nips.cc/paper/6770-train-longer-generalize-better-closing-the-generalization-gap-in-large-batch-training-of-neural-networks.pdf
3. Super-convergence: This is a well explained section where the authors describe the LR range test and how it can be used to understand potential for super-convergence for any architecture. The authors also provide sufficient intuition for super-convergence. Since CLRs were already proposed by Smith (2015), the originality of this work would be specifically tied to their application to residual units. It would be interesting to see a qualitative analysis on how the residual error is impacting super-convergence.
4. Regularization: While Fig 4 demonstrates the regularization property, the reference to Fig 1a with better test error compared to typical training methods could simply be a result of slower convergence of typical training methods. 
5. Optimal LRs: Fig.5b shows results for 1000 iterations whereas the text says 10000 (seems like a typo in scaling the plot). Figs 1 and 5 illustrate only one cycle (one increase and one decrease) of CLR. It would be interesting to see cases where more than one cycle is required and to see what happens when the LR increases the second time.
6. Experiments: This is a strong section where the authors show extensive reproducible experimentation to identify settings under which super-convergence works or does not work. However, the fact that the results only applies to CIFAR-10 dataset and could not be observed for ImageNet or other architectures is disappointing and heavily takes away from the significance of this work. 
Overall, the work is presented as a positive result in very specific conditions but it seems more like a negative result. It would be more appealing if the paper is presented as a negative result and strengthened by additional experimentation and theoretical backing.