Smoothed Gradients for Stochastic Variational Inference
Stochastic variational inference (SVI) lets us scale up Bayesian computation to massive data. It uses stochastic optimization to fit a variational distribution, following easy-to-compute noisy natural gradients. As with most traditional stochastic optimization methods, SVI takes precautions to use unbiased stochastic gradients whose expectations are equal to the true gradients. In this paper, we explore the idea of following biased stochastic gradients in SVI. Our method replaces the natural gradient with a similarly constructed vector that uses a fixed-window moving average of some of its previous terms. We will demonstrate the many advantages of this technique. First, its computational cost is the same as for SVI and storage requirements only multiply by a constant factor. Second, it enjoys significant variance reduction over the unbiased estimates, smaller bias than averaged gradients, and leads to smaller mean-squared error against the full gradient. We test our method on latent Dirichlet allocation with three large corpora.
OverviewThe authors proposed a new approach of smoothing gradients for stochastic variational inference by averaging the sufficient statistics across a fixed time window. With the proposed method and a proper chosen time window size, the variance of stochastic gradients can be reduced, and the convergence speed of the algorithm can be improved. However, the sufficient statistics averaging introduce a bias to the gradients, which might lead to a less optimal solution. QualityThis work proposes a gradient smoothing scheme via averaging the sufficient statistics. The gradient smoothing reduces variance of gradients, while paying the price by introducing a bias into gradients. It is an interesting idea to average the sufficient statistics instead of averaging gradients which has been studied before. The experiments with LDA show a good result empirically. It has been shown that the gradient smoothing scheme can reduce variance of gradients and improve model performance with a proper time-window size. The choice of the time-window size becomes a tuning parameter for the tradeoff between bias and variance. In general, introducing a small bias to the model is not a big problem, since variational inference approaches introduce an intrinsic bias while optimizing the evidence lower bound.I have several reservations regarding this work. Different from introducing a bias into the objective function of model, the biases in gradients might be accumulated throughout learning, which potentially lead to a solution very far from local optima. It would be interesting to study the properties of the bias term, and see the new objective function that the bias term leads to. As mentioned in the discussion section, the convergence of the proposed method has not been proved. Some demonstration of convergence, at least empirically, needs to be shown. According to the experimental results, the choice of the time window size is crucially to the performance of the method. It would be useful to discuss the principal of choosing the right window size.ClarityThis paper is, in general, well written, and easily understandable. The equation reference on line 240 is missing. In figure 3, the results of SVI are not converged. Please show the results until convergence. It will be also good to show the performance of original LDA as a baseline, to have an idea of the performance difference between SVI-based approaches and original LDA for general audience.OriginalityThis work looks like an original work with an interesting idea and good performance.SignificanceThis work presents a new gradient smoothing scheme for stochastic variational inference. It would potentially be very useful for the variational inference community.