Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks
Stochastic gradient descent (SGD) is widely believed to perform implicit regularization when used to train deep neural networks, but the precise manner in which this occurs has thus far been elusive. We prove that SGD minimizes an average potential over the posterior distribution of weights along with an entropic regularization term. This potential is however not the original loss function in general. So SGD does perform variational inference, but for a different loss than the one used to compute the gradients. Even more surprisingly, SGD does not even converge in the classical sense: we show that the most likely trajectories of SGD for deep networks do not behave like Brownian motion around critical points. Instead, they resemble closed loops with deterministic components. We prove that such out-of-equilibrium behavior is a consequence of highly non-isotropic gradient noise in SGD; the covariance matrix of mini-batch gradients for deep networks has a rank as small as 1% of its dimension. We provide extensive empirical validation of these claims, proven in the appendix.
 The authors discuss the regularized objective function minimized by standard SGD in the context of neural nets, and provide a variational inference perspective using the Fokker-Planck equation. They note that the objective can be very different from the desired loss function if the SGD noise matrix is low rank, as evidenced in their experiments. Overall the paper is written quite well, and the authors do a good job of explaining their thesis. However I was unable to identify any real novelty in the theory: the Fokker-Planck equation has been widely used in analysis of stochastic noise in MCMC samplers in recent years, and this paper mostly rephrases those results. Also the fact that SGD theory only works for isotropic noise is well known, and that there is divergence from the true loss function in case of low rank noise is obvious. Thus I found most of section 3 to be a reformulation of known results, including Theorem 5 and its proof. Same goes for section 5; the symmetric- anti symmetric split is a common technique used in the stochastic MCMC literature over the last few years, and I did not find any new insight into those manipulations of the Fokker-Planck equation from this paper. Thus I think that although this paper is written well, the theory is mostly recycled and the empirical results in Section 4 are known; thus it is below acceptance threshold due to lack of novelty.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks
Stochastic gradient descent (SGD) is widely believed to perform implicit regularization when used to train deep neural networks, but the precise manner in which this occurs has thus far been elusive. We prove that SGD minimizes an average potential over the posterior distribution of weights along with an entropic regularization term. This potential is however not the original loss function in general. So SGD does perform variational inference, but for a different loss than the one used to compute the gradients. Even more surprisingly, SGD does not even converge in the classical sense: we show that the most likely trajectories of SGD for deep networks do not behave like Brownian motion around critical points. Instead, they resemble closed loops with deterministic components. We prove that such out-of-equilibrium behavior is a consequence of highly non-isotropic gradient noise in SGD; the covariance matrix of mini-batch gradients for deep networks has a rank as small as 1% of its dimension. We provide extensive empirical validation of these claims, proven in the appendix.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The authors discuss the regularized objective function minimized by standard SGD in the context of neural nets, and provide a variational inference perspective using the Fokker-Planck equation. They note that the objective can be very different from the desired loss function if the SGD noise matrix is low rank, as evidenced in their experiments. Overall the paper is written quite well, and the authors do a good job of explaining their thesis. However I was unable to identify any real novelty in the theory: the Fokker-Planck equation has been widely used in analysis of stochastic noise in MCMC samplers in recent years, and this paper mostly rephrases those results. Also the fact that SGD theory only works for isotropic noise is well known, and that there is divergence from the true loss function in case of low rank noise is obvious. Thus I found most of section 3 to be a reformulation of known results, including Theorem 5 and its proof. Same goes for section 5; the symmetric- anti symmetric split is a common technique used in the stochastic MCMC literature over the last few years, and I did not find any new insight into those manipulations of the Fokker-Planck equation from this paper. Thus I think that although this paper is written well, the theory is mostly recycled and the empirical results in Section 4 are known; thus it is below acceptance threshold due to lack of novelty.
assistant
 The authors discuss the regularized objective function minimized by standard SGD in the context of neural nets, and provide a variational inference perspective using the Fokker-Planck equation. They note that the objective can be very different from the desired loss function if the SGD noise matrix is low rank, as evidenced in their experiments. Overall the paper is written quite well, and the authors do a good job of explaining their thesis. However I was unable to identify any real novelty in the theory: the Fokker-Planck equation has been widely used in analysis of stochastic noise in MCMC samplers in recent years, and this paper mostly rephrases those results. Also the fact that SGD theory only works for isotropic noise is well known, and that there is divergence from the true loss function in case of low rank noise is obvious. Thus I found most of section 3 to be a reformulation of known results, including Theorem 5 and its proof. Same goes for section 5; the symmetric- anti symmetric split is a common technique used in the stochastic MCMC literature over the last few years, and I did not find any new insight into those manipulations of the Fokker-Planck equation from this paper. Thus I think that although this paper is written well, the theory is mostly recycled and the empirical results in Section 4 are known; thus it is below acceptance threshold due to lack of novelty.