The non-convex nature of the loss landscape of deep neural networks (DNN) lends them the intuition that over the course of training, stochastic optimization algorithms explore different regions of the loss surface by entering and escaping many local minima due to the noise induced by mini-batches. But is this really the case? This question couples the geometry of the DNN loss landscape with how stochastic optimization algorithms like SGD interact with it during training. Answering this question may help us qualitatively understand the dynamics of deep neural network optimization. We show evidence through qualitative and quantitative experiments that mini-batch SGD rarely crosses barriers during DNN optimization. As we show, the mini-batch induced noise helps SGD explore different regions of the loss surface using a seemingly different mechanism. To complement this finding, we also investigate the qualitative reason behind the slowing down of this exploration when using larger batch-sizes. We show this happens because gradients from larger batch-sizes align more with the top eigenvectors of the Hessian, which makes SGD oscillate in the proximity of the parameter initialization, thus preventing exploration.
The subject of how a given algorithm explores the landscape is still a poorly understood area in training neural networks. There is a large body of recent work that attempts to shed light on this puzzle, and each one tries to claim their share in the furthering of the understanding of the relationship between the geometry of the landscape and the dynamics that one chooses in optimization. The present paper is a fine addition to the literature with interesting observations and novel questions, however, it falls short in many core areas: An apparent work in progress that has a great potential. 
It is safe to say that "A walk with SGD" has an important single focus in mind: Does the SGD cross over barriers in the weight space of the underlying neural network? This question, at its heart, is intimately linked with the many properties that are attributed to the modern algorithm of the choice and the way it navigates a given non-convex landscape. The paper claims to provide an almost negative answer to the question and thereby busting several myths that are attributed to the "trick" part of SGD algorithm. As good as it sounds, unfortunately, the paper falls short of providing a convincing evidence (be it theoretical or empirical), and the way it tries to frame itself unique and different in relation to related works only indicate a lack of deep understanding of the existing literature. Therefore, I think there are several ways the paper should be improved before it is ready.
A major question (that I hope will easily be addressed) is on the definition of the barrier itself. According to the text, a barrier is defined judging by the minima of two 1-dimensional segments that connect weights connecting three consecutive steps: if the minimum of the line segment defined by the latter step is larger than the former, then it declared that a barrier is crossed. In a low dimensional world, this makes total sense, however, I fail to understand what kind of barrier it implies on the geometry of the landscape: Can the 1-dimensional lines be on the sides of a valley? Can one find *another* 1-dimensional projection for which the inequality is broken? How do such dependencies change the understanding of the problem? And if one is indeed only interested in the flat line segments (since SGD is making discrete steps), then one can, in principle, observe barrier crossing in a convex problem, as well? Is there an argument for otherwise? Or if it is a notion that applies equally well in a convex case then how should we really think about the barrier crossing? On the opposite point of view, can one not imagine a barrier crossing that doesn't appear in this triangular inequality above?
The paper is full of empirical evidence that is guided by a simple observable that is very intuitive, however, it lacks a comprehensive discussion on the new quantity they propose that I consider a major flaw, but that I think (hope) that the authors can fix very easily. Some minor points that would improve the readability and clarity for the reader:
- The figures are not very reader-friendly, this can be improved by better using the whitespaces in the paper but it can also be improved by finding further observables that would summarize the observations instead of showing individual consecutive line interpolations.
- What are the values of the y-axis in Figure 5 and 6? Are they the top eigenvalues of the Hessian?
- In the models that are compared in Figure 7, what are their generalization properties (early stopping and otherwise)?
- The interpretation at the end of p. 6 may be a good motivation for the reader if it had been introduced earlier for that section.
Finally, Section 5 reads very strangely, I have hard times understanding why certain phrases exist the way they are in this part. Here are further notes on Section 5:
- Why the whole first paragraph focused on hot the current paper is different than Goodfellow et al (2014) (it is obvious that it is for a different purpose), or why do we read the sentence "Li et al (2017b) also visualize...." What way do they visualize, is it the only paper that does visualization, what's the relation with the current paper and barrier crossing? 
- For the second paragraph, I can suggest another paper, https://arxiv.org/abs/1803.06969, that was at ICML which also looks at the diffusion process through the parameter distance at different times which is similar to Hoffer et al. which also claims no barrier crossing similar to the present paper. 
- However, my main issue is the exact connection between diffusion and no barrier crossing and it's connection to SGD preferring wide local minima instead of a narrow one. The second paragraph of the conclusion touches upon this subject. But it is not entirely clear how they are linked (except for the brittle SDE approximation at Li et al (see https://arxiv.org/abs/1810.00004)). Overall, the paper would benefit a lot from the discussion on why it is preferable to have SGD choose one basin over another in the beginning, as it is, it looks like the paper has another agenda behind the scenes.
- In the fourth paragraph of the conclusion, the paper refers to three papers that link DNN to spin glasses, in two of the (older) references the networks are far from what we have today, and the third one is far from "showing" anything between DNN and spin glass. In any case, what's the link between the aspects studied there with the present paper?
- Finally, the paper claims at the last few sentences that the works referred a little bit earlier look at the loss surface "in isolation from the optimization dynamics", however, many of those works cited have their empirical observations much like the current paper, and clearly they all "study the DNN loss surface along the trajectory of SGD" necessarily as it is the way to find local minima, saddle points, paths, curvature etc... The present paper is already very interesting and full of novel insight, I fail to see the value of struggling to stand out like this.
Overall, I think the paper is a very interesting step forward in understanding SGD dynamics on the DNN landscape. And, even though it has many shortcomings as it currently stands, I think it has a lot of room to improve.