FLUCTUATION-DISSIPATION RELATIONS FOR STOCHASTIC GRADIENT DESCENT
The notion of the stationary equilibrium ensemble has played a central role in statistical mechanics. In machine learning as well, training serves as generalized equilibration that drives the probability distribution of model parameters toward stationarity. Here, we derive stationary fluctuation-dissipation relations that link measurable quantities and hyperparameters in the stochastic gradient descent algorithm. These relations hold exactly for any stationary state and can in particular be used to adaptively set training schedule. We can further use the relations to efficiently extract information pertaining to a loss-function landscape such as the magnitudes of its Hessian and anharmonicity. Our claims are empirically verified.
The paper introduces the concept of fluctuation-dissipation relations to stochastic gradient descent. These relations hold for certain observables in physical systems in equilibrium. In the context of SGD as a non-equilibrium process with a stationary density, they allow to quantify how far away this process is from its stationary state. 
One of the strengths of the paper is that it works in the discrete-time formalism and uses the master equation, as opposed to other recent works that used the continuous-time limit of SGD to derive related (yet different) results. Furthermore, the formalism does not even rely on a locally quadratic approximation of the loss function, or on any Gaussian assumptions of the SGD noise. To the best of my knowledge, all of this is very innovative. Ultimately, the authors propose a practical algorithm to adaptively lowering the learning rate based on testing fluctuation-dissipation relations.
This is an interesting paper which I recommend to accept. It not only shows new theoretical results, but also conforms their validity in real-world experiments.
I have only a few questions / comments:
1. In Eq. 17 and others where the scalar product of theta and grad(f) occurs, is it implicitly assumed that the optimum of f is at theta=0?
2. In Fig. 2, the distinction between solid and dotted curves could be made better visible.
3. For completeness, it would be good to add the following citation:
Stephan Mandt, Matthew D. Hoffman, and David M. Blei. "Continuous-time limit of stochastic gradient descent revisited." NIPS 2015 Workshop on Optimization for Machine Learning.