A CLOSER LOOK AT DEEP LEARNING HEURISTICS: LEARNING RATE RESTARTS, WARMUP AND DISTILLA- TION
The convergence rate and final performance of common deep learning models have significantly benefited from heuristics such as learning rate schedules, knowledge distillation, skip connections, and normalization layers. In the absence of theoretical underpinnings, controlled experiments aimed at explaining these strategies can aid our understanding of deep learning landscapes and the training dynamics. Existing approaches for empirical analysis rely on tools of linear interpolation and visualizations with dimensionality reduction, each with their limitations. Instead, we revisit such analysis of heuristics through the lens of recently proposed methods for loss surface and representation analysis, viz., mode connectivity and canonical correlation analysis (CCA), and hypothesize reasons for the success of the heuristics. In particular, we explore knowledge distillation and learning rate heuristics of (cosine) restarts and warmup using mode connectivity and CCA. Our empirical analysis suggests that: (a) the reasons often quoted for the success of cosine annealing are not evidenced in practice; (b) that the effect of learning rate warmup is to prevent the deeper layers from creating training instability; and (c) that the latent knowledge shared by the teacher is primarily disbursed to the deeper layers.
Summary:
This paper uses the recently proposed techniques of mode connectivity and CCA to analyze two different popular heuristics in deep learning: 
(1) SGDR (stochastic gradient descent with restarts/cosine annealing of learning rate) 
(2) Learning rate warmup
(3) Model distillation
For (1) they visualize 1d and 2d slices of the loss surface either using mode connectivity, or parameter points immediately before restarts to try and understand if the parameters sit in different local minima. For (2), they study the effect of learning rate warmup using CCA, coming to the conclusion that learning rate warmup helps stabilize the fully connected layers. (3) They also study model distillation with CCA, finding out that the higher layers are the most similar to the teacher model. 
Clarity: The paper is clearly written, cites lots of relevant work, and describes the experiments in detail.
Originality: This paper seems original, while the techniques used are established, they conduct thorough experiments on phenomena in deep learning that haven't been studied.
 
Comments on Significance and Quality:
I liked parts (2), (3) of the paper most, as it seemed like conclusions from these parts were fairly clear: 
Figures 4, 5 make the effect of warm restarts in the large batch setting on FC layers clear: the restarts help the layers stabilize better. I really liked the experiment in 4(d), where they tested this hypothesis by freezing the fully connected layers for the duration of the warmup.  It was interesting to see that this had no effect on the remainder of the trajectory. This seemed to be a good demonstration and investigation of the effect of warm restarts, and I appreciate the tests on different architectures in the supplementary material. I'd be curious to see if there's some way to further incorporate this into learning rate schedules.
I also liked Figure 6, exploring Model distillation, which showed that the higher layers of the shallower network were the most affected by the teacher network. The authors cite related work which suggests only training higher layers, and I'd be curious to see how only training higher layers affects accuracy.
While I thought the experiments for part (1) SGD with Restarts were thorough, and appreciated Figure 1, which experimentally validated the use of mode connectivity, I felt there was some difficulty in interpreting the results. 
Firstly, in Figure 2, the claim is that SGD with Restarts does possibly bridge local minima as the mode connectivity curves increase between the two convergence points. However, we see in both 2(b) and 2(c) that the linear interpolation between both convergence points does *not* increase in loss. In which case is there any reason to believe that the increase of MC in the middle means that SGDR is climbing a basin? How do we know that the linear combination isn't closer to the path followed by SGDR? 
For additional comparisons, it would be good to have the linear combination plots for Figure 1 also.
In general, it seems hard to make meaningful conclusions with low dimensional projections of a very high dimensional loss surface. We'd have to know some kind of theoretical property of MC to be able to do so.
Minor Comments
I think the figures in this paper could be much clearer. In Figure 2 for example, the legend blocks some of the main areas of interest of the plot. I would recommend cutting some of the raw learning rate figures and making all figures much bigger.
In figure 4(d), the text describes the process in training steps (200 training steps), but the plot is in epochs -- it would be better if the text and axis were consistent in units.
Conclusion:
Despite my concerns on the first part of this paper, I think the very thorough experiments, clear presentation and the interesting results on learning rate warmups and model distillation merit its acceptance.