Published as a conference paper at ICLR 2018 STOCHASTIC GRADIENT DESCENT PERFORMS VARIATIONAL INFERENCE, CONVERGES TO LIMIT CYCLES FOR DEEP NETWORKS
Stochastic gradient descent (SGD) is widely believed to perform implicit regularization when used to train deep neural networks, but the precise manner in which this occurs has thus far been elusive. We prove that SGD minimizes an average potential over the posterior distribution of weights along with an entropic regularization term. This potential is however not the original loss function in general. So SGD does perform variational inference, but for a different loss than the one used to compute the gradients. Even more surprisingly, SGD does not even converge in the classical sense: we show that the most likely trajectories of SGD for deep networks do not behave like Brownian motion around critical points. Instead, they resemble closed loops with deterministic components. We prove that such "out-of-equilibrium" behavior is a consequence of highly non-isotropic gradient noise in SGD; the covariance matrix of mini-batch gradients for deep networks has a rank as small as 1% of its dimension. We provide extensive empirical validation of these claims, proven in the appendix.
The paper takes a closer look at the analysis of SGD as variational inference, first proposed by Duvenaud et al. 2016
and Mandt et al. 2016. In particular, the authors point out that in general, SGD behaves quite differently from Langevin diffusion due to the multivariate nature of the Gaussian noise. As the authors show based on the Fokker-Planck equation of the underlying stochastic process, there exists a conservative current (a gradient of an underlying potential) and a non-conservative current (which might induce stationary persistent currents at long times). The non-conservative part leads to the fact that the dynamics of SGD	may show oscillations, and these oscillations may even prevent the algorithm from converging to the 'right' local optima. The theoretical analysis is carried-out very nicely, and the theory is supported by experiments on two-dimensional toy examples, and Fourier-spectra of the iterates of SGD.
This is a nice paper which I would like to see accepted. In particular I appreciate that the authors stress the importance
of 'non-equilibrium physics' for understanding the SGD process. Also, the presentation is quite clear and the paper well written.
There are a few minor points which I would like to ask the authors to address:
1. Why cite Kingma and Welling as a source for variational inference in	section 3.1? VI is a much older	field, and Kingma and Welling proposed a very special form of VI, namely amortized VI with inference networks. A better citation would be Jordan et	al 1999.
2. I'm not sure how much to trust the Fourier-spectra. In particular, perhaps the deviations from Brownian motion could also be due to the discrete	nature of SGD (i.e. that the continuous-time formalism is only an approximation of a discrete process). Could you elaborate on this?
3. Could you give the reader more details on how the uncertainty estimates on the Fourier transformations were obtained?
Thanks.