A Walk with SGD: How SGD Explores Regions of Deep Network Loss?
The non-convex nature of the loss landscape of deep neural networks (DNN) lends them the intuition that over the course of training, stochastic optimization algorithms explore different regions of the loss surface by entering and escaping many local minima due to the noise induced by mini-batches. But is this really the case? This question couples the geometry of the DNN loss landscape with how stochastic optimization algorithms like SGD interact with it during training. Answering this question may help us qualitatively understand the dynamics of deep neural network optimization. We show evidence through qualitative and quantitative experiments that mini-batch SGD rarely crosses barriers during DNN optimization. As we show, the mini-batch induced noise helps SGD explore different regions of the loss surface using a seemingly different mechanism. To complement this finding, we also investigate the qualitative reason behind the slowing down of this exploration when using larger batch-sizes. We show this happens because gradients from larger batch-sizes align more with the top eigenvectors of the Hessian, which makes SGD oscillate in the proximity of the parameter initialization, thus preventing exploration.
 I like the idea of trying to qualitatively illustrate the behavior of SGD when optimizing parameters of complex models, such as Deep and Conv Nets, but I think that the contribution is not very substantial. The connection between SGD and diffusion has been pointed out in previous papers, as acknowledged by the Authors. The study of the effect of batch size is interesting, but again somewhat derived from previous works. It would helpful to illustrate the difference between "crossing" and "moving over" a barrier with a simple figure. The experimental validation is interesting, although I think it is limited and perhaps the conclusions that can be drawn from it are not so surprising. I believe it would have been interesting to study other important factors that affect the behavior of SGD, such as learning rate and type of momentum. For example, a larger learning rate might allow for more crossing of barriers. Also, different SGD algorithms (ADAGRAD, ADAM, etc...) would behave considerably differently I expect. At the moment these important factors are overlooked. It is not clear to me why we would want to avoid larger batch sizes. A larger batch size allows for a lower variance of stochastic gradients, and therefore faster convergence. I think this point requires elaboration, because this forms the motivation behind theoretically grounded and successful SGD works, such as SAGA and the like. I agree that a smaller batch-size is preferable at the beginning of the optimization, but again this is a well known fact (again, see SAGA) and it is for computational reasons mostly (being far away from the (local) mode, a noisy gradient is enough to move in the right direction - no need to spend computations to use an accurate gradient). There is no guarantee that the local optimum close to initialization is a bad local optimum in general, so I don*t think that using a large batch size at the beginning is a bad idea for this reason - again it is just computational. Another thing missing I think is the discussion around why it is potentially a good thing to cross the barrier, either at the beginning of the exploration or towards convergence to a local optimum. At the moment, the paper seems to report the behavior of SGD without key insights on the importance of crossing or avoiding crossing barriers. As a concluding remark - there has been a lot of work on the connections between diffusions and MCMC algorithms (see e.g., the Metropolis Adjusted Langevin Algorithm - MALA) and a lot of the considerations made in the paper are somewhat known. That is, random walk/diffusion type MCMC (and even gradient-based MCMC like Hybrid Monte Carlo) struggle a lot in non-convex problems and they hardly move across modes of a posterior distribution (equivalent to crossing barriers of potential). So I*m not at all surprised that SGD does not cross barriers during optimization and I would challenge the statement in the introduction saying "Intuitively, when performing random walk on a potential, one would expect barriers being crossed quite often during the process."
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
A Walk with SGD: How SGD Explores Regions of Deep Network Loss?
The non-convex nature of the loss landscape of deep neural networks (DNN) lends them the intuition that over the course of training, stochastic optimization algorithms explore different regions of the loss surface by entering and escaping many local minima due to the noise induced by mini-batches. But is this really the case? This question couples the geometry of the DNN loss landscape with how stochastic optimization algorithms like SGD interact with it during training. Answering this question may help us qualitatively understand the dynamics of deep neural network optimization. We show evidence through qualitative and quantitative experiments that mini-batch SGD rarely crosses barriers during DNN optimization. As we show, the mini-batch induced noise helps SGD explore different regions of the loss surface using a seemingly different mechanism. To complement this finding, we also investigate the qualitative reason behind the slowing down of this exploration when using larger batch-sizes. We show this happens because gradients from larger batch-sizes align more with the top eigenvectors of the Hessian, which makes SGD oscillate in the proximity of the parameter initialization, thus preventing exploration.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 I like the idea of trying to qualitatively illustrate the behavior of SGD when optimizing parameters of complex models, such as Deep and Conv Nets, but I think that the contribution is not very substantial. The connection between SGD and diffusion has been pointed out in previous papers, as acknowledged by the Authors. The study of the effect of batch size is interesting, but again somewhat derived from previous works. It would helpful to illustrate the difference between "crossing" and "moving over" a barrier with a simple figure. The experimental validation is interesting, although I think it is limited and perhaps the conclusions that can be drawn from it are not so surprising. I believe it would have been interesting to study other important factors that affect the behavior of SGD, such as learning rate and type of momentum. For example, a larger learning rate might allow for more crossing of barriers. Also, different SGD algorithms (ADAGRAD, ADAM, etc...) would behave considerably differently I expect. At the moment these important factors are overlooked. It is not clear to me why we would want to avoid larger batch sizes. A larger batch size allows for a lower variance of stochastic gradients, and therefore faster convergence. I think this point requires elaboration, because this forms the motivation behind theoretically grounded and successful SGD works, such as SAGA and the like. I agree that a smaller batch-size is preferable at the beginning of the optimization, but again this is a well known fact (again, see SAGA) and it is for computational reasons mostly (being far away from the (local) mode, a noisy gradient is enough to move in the right direction - no need to spend computations to use an accurate gradient). There is no guarantee that the local optimum close to initialization is a bad local optimum in general, so I don*t think that using a large batch size at the beginning is a bad idea for this reason - again it is just computational. Another thing missing I think is the discussion around why it is potentially a good thing to cross the barrier, either at the beginning of the exploration or towards convergence to a local optimum. At the moment, the paper seems to report the behavior of SGD without key insights on the importance of crossing or avoiding crossing barriers. As a concluding remark - there has been a lot of work on the connections between diffusions and MCMC algorithms (see e.g., the Metropolis Adjusted Langevin Algorithm - MALA) and a lot of the considerations made in the paper are somewhat known. That is, random walk/diffusion type MCMC (and even gradient-based MCMC like Hybrid Monte Carlo) struggle a lot in non-convex problems and they hardly move across modes of a posterior distribution (equivalent to crossing barriers of potential). So I*m not at all surprised that SGD does not cross barriers during optimization and I would challenge the statement in the introduction saying "Intuitively, when performing random walk on a potential, one would expect barriers being crossed quite often during the process."
assistant
 I like the idea of trying to qualitatively illustrate the behavior of SGD when optimizing parameters of complex models, such as Deep and Conv Nets, but I think that the contribution is not very substantial. The connection between SGD and diffusion has been pointed out in previous papers, as acknowledged by the Authors. The study of the effect of batch size is interesting, but again somewhat derived from previous works. It would helpful to illustrate the difference between "crossing" and "moving over" a barrier with a simple figure. The experimental validation is interesting, although I think it is limited and perhaps the conclusions that can be drawn from it are not so surprising. I believe it would have been interesting to study other important factors that affect the behavior of SGD, such as learning rate and type of momentum. For example, a larger learning rate might allow for more crossing of barriers. Also, different SGD algorithms (ADAGRAD, ADAM, etc...) would behave considerably differently I expect. At the moment these important factors are overlooked. It is not clear to me why we would want to avoid larger batch sizes. A larger batch size allows for a lower variance of stochastic gradients, and therefore faster convergence. I think this point requires elaboration, because this forms the motivation behind theoretically grounded and successful SGD works, such as SAGA and the like. I agree that a smaller batch-size is preferable at the beginning of the optimization, but again this is a well known fact (again, see SAGA) and it is for computational reasons mostly (being far away from the (local) mode, a noisy gradient is enough to move in the right direction - no need to spend computations to use an accurate gradient). There is no guarantee that the local optimum close to initialization is a bad local optimum in general, so I don*t think that using a large batch size at the beginning is a bad idea for this reason - again it is just computational. Another thing missing I think is the discussion around why it is potentially a good thing to cross the barrier, either at the beginning of the exploration or towards convergence to a local optimum. At the moment, the paper seems to report the behavior of SGD without key insights on the importance of crossing or avoiding crossing barriers. As a concluding remark - there has been a lot of work on the connections between diffusions and MCMC algorithms (see e.g., the Metropolis Adjusted Langevin Algorithm - MALA) and a lot of the considerations made in the paper are somewhat known. That is, random walk/diffusion type MCMC (and even gradient-based MCMC like Hybrid Monte Carlo) struggle a lot in non-convex problems and they hardly move across modes of a posterior distribution (equivalent to crossing barriers of potential). So I*m not at all surprised that SGD does not cross barriers during optimization and I would challenge the statement in the introduction saying "Intuitively, when performing random walk on a potential, one would expect barriers being crossed quite often during the process."