On the relation between the sharpest direc- tions of DNN loss and the SGD step length
Stochastic Gradient Descent (SGD) based training of neural networks with a large learning rate or a small batch-size typically ends in well-generalizing, flat regions of the weight space, as indicated by small eigenvalues of the Hessian of the training loss. However, the curvature along the SGD trajectory is poorly understood. An empirical investigation shows that initially SGD visits increasingly sharp regions, reaching a maximum sharpness determined by both the learning rate and the batch-size of SGD. When studying the SGD dynamics in relation to the sharpest directions in this initial phase, we find that the SGD step is large compared to the curvature and commonly fails to minimize the loss along the sharpest directions. Furthermore, using a reduced learning rate along these directions can improve training speed while leading to both sharper and better generalizing solutions compared to vanilla SGD. In summary, our analysis of the dynamics of SGD in the subspace of the sharpest directions shows that they influence the regions that SGD steers to (where larger learning rate or smaller batch size result in wider regions visited), the overall training speed, and the generalization ability of the final model.
Update after author response: I am changing my rating from 4 to 6 in light of the clarification and new experiments.
-------
In this paper the authors study the relationship between the SGD step size and the curvature of the loss surface, empirically showing that: 1) SGD is guided towards sharp regions of the loss surface at the start especially with a large learning rate or a small batch size. 2) Loss increases on average when taking a SGD step in the sharpest directions. 3) Modifying the SGD step size in the sharp directions (for example removing its component in the sharpest direction), can lead to substantial changes in both the quality and the local landscape of the minima (for the example mentioned, leading to a better and sharper minima). Motivated by these observations, the authors propose a variant of SGD that leads to better performance on the datasets considered.
Deep learning theory is a very important frontier for machine learning and one that’s needed to make the practice be guided more by the foundational principles than incessant tweaks. The paper makes some very interesting observations and uses those insights to improve the widely used SGD. However, I have a few concerns which leave me unconvinced about the impact of the contributions in the paper. My biggest problem is the use of second order information in the algorithm which makes the optimization process computationally cumbersome, and raises the question as to why might this approach be preferable to any other second order approach (the authors touch on Newton method in the appendix but the discussion far from settles the matter). Similar questions arise in considering the merit of the proposed methods in comparison to a host of other well-studied augmentations to SGD like momentum, Adam or AdaGrad. The quality of presentation is also a problem, and both the organization of the main matter as well as of the figures can use some polishing. The latter specifically sometimes lacked legends (Fig. 3 and 4), and some other times had legends covering a quarter of the plot (Fig. 5). Lastly, even though the claims sound theoretical, they are not derived from any set of first principles but come from observations on a few datasets. While this may after all be how SGD behaves in general, currently the paper doesn’t provide any evidence to believe that. 
Minor issues: “withe” (page 2, spelling), “\alpha = 0.5, 1, 2 corresponding to red, green, and blue” (page 4, I believe it should be “blue, green and red”).
In summary, even though I liked what the paper set out to do, I am not convinced on the generalizability of these results and subsequently the rationale for using the proposed method over other competing options. A revised version of the paper with either validation on more datasets or sound theory generalizing the results to some extent would make for a much nicer contribution.