Variance Reduction in Stochastic Gradient Langevin Dynamics
Stochastic gradient-based Monte Carlo methods such as stochastic gradient Langevin dynamics are useful tools for posterior inference on large scale datasets in many machine learning applications. These methods scale to large datasets by using noisy gradients calculated using a mini-batch or subset of the dataset. However, the high variance inherent in these noisy gradients degrades performance and leads to slower mixing. In this paper, we present techniques for reducing variance in stochastic gradient Langevin dynamics, yielding novel stochastic Monte Carlo methods that improve performance by reducing the variance in the stochastic gradient. We show that our proposed method has better theoretical guarantees on convergence rate than stochastic Langevin dynamics. This is complemented by impressive empirical results obtained on a variety of real world datasets, and on four different machine learning tasks (regression, classification, independent component analysis and mixture modeling). These theoretical and empirical contributions combine to make a compelling case for using variance reduction in stochastic Monte Carlo methods.
This paper proposes two stochastic gradient sampling methods, motivated by combining the approach of Stochastic Gradient Langevin Dynamics with recent variance reduction techniques from the stochastic approximation literature.
The overall structure of the paper is reasonably clear, and the algorithms themselves are stated reasonably precisely (although there are inconsistencies in the notation used within the algorithm boxes), with clear motivation from the introductory material.  There are some typos (see minor points below), and quite a few instances where notation is used without being defined – in some circumstances it is clear what is meant, but in several circumstances it impeded my understanding of the paper. For example, the quantities \bar{\phi} and \hat{\phi} that appear in Theorems 1, 2, and 3 are not defined.  An important point which I don’t believe has been picked up in the paper is that the newly proposed methods are not MCMC methods as claimed in the abstract - the fact that gradients are stored between timesteps breaks the Markovian structure of the process.  In relation to the “Analysis” section of the paper, the two main theoretical results supporting the algorithms are clearly stated (although as noted above some notation is undefined).  There are some particularly strong conditions in assumption [A2], particularly the condition on uniform boundedness of the gradient log-prior and log-likelihoods, which restricts the applicability of Theorems 2 and 3. The condition rules out (for example) Gaussian priors or likelihoods, and therefore the theorems subsequently proven do not apply to any of the models in the experiments. It would be useful to explain what kinds of models the theoretical results can be applied to, and what kinds of models are excluded by these strong conditions.  I found the proofs of Theorems 2 and 3 in the appendix difficult to follow, due to undefined notation (e.g. C^\prime appearing in Eqn (16)), and missing notation (presumably the norms appearing in the displayed equations between lines 329 and 330 should have expectations taken over them).  Additionally, there appears to be a mistake in the displayed equations between lines 329 and 330; the expected value of the squared norm of eta_j, a d-dimensional multivariate normal random variable (with mean vector 0 and covariance matrix \sqrt{h} I), appears to be bounded by h. In fact, the expected value is exactly hd, where d is the dimension of the variable.  A general point about the final paragraph of Section 4 is that the bounds obtained in Theorems 1, 2, and 3 all have 1/(Th) and h^2 terms in common - would this mean that the bounds all behave similarly asymptotically, even if the first term in the bounds of Theorems 2 and 3 decays quicker than that of Theorem 1? If this is the case, then the claim that convergence of the MSE of SAGA-LD and SVRG-LD is significantly faster than that of SGLD might not hold.  In relation to the “Experiments” section of the paper, there is a good range of experiments (Bayesian linear regression, logistic regression, and ICA) which clearly demonstrate that SAGA-LD performs well in comparison to SGLD in several circumstances. It is a shame that no experiments relating to the SVRG-LD method were included.  The SAGA-LD algorithm begins with a full pass through the data to calculate the gradient log posterior at the initial parameter value - would this materially affect the comparisons drawn for the experiments if the methods were compared on a wall-clock basis, rather than number of passes through the data?  As a minor note, it appears as though in several of the regression and classification experiments, SAGA-LD and SGLD reach a similar level of MSE after several passes through the data set. It would be interesting to see whether initialising both algorithms at the MAP estimate of the posterior (say) would reduce the difference in performance between the two methods. Comparing the performance of the sampling schemes with pure optimisation algorithms such as SGD/SAGA could also provide a useful benchmark for the results obtained.  The mixture model experiment only shows results for SGLD (although perhaps this is a typo in line 258?), so it is not clear to me why it has been included in the paper.  Minor points: Throughout: “Robbins-Monroe” should be “Robbins-Monro” Line 23: the word “method” is repeated Line 50: “success” should be “successes” Line 76: U is the unnormalized *negative* log posterior Eqn (3): In the definition of the generator, the expectation should be conditioned on \theta_0 = \theta References: The 1951 Robbins-Monro paper appears twice in the list of references Appendix Section E: the “protein” graph appears twice