Variance Reduction for Stochastic Gradient Optimization
Stochastic gradient optimization is a class of widely used algorithms for training machine learning models. To optimize an objective, it uses the noisy gradient computed from the random data samples instead of the true gradient computed from the entire dataset. However, when the variance of the noisy gradient is large, the algorithm might spend much time bouncing around, leading to slower convergence and worse performance. In this paper, we develop a general approach of using control variate for variance reduction in stochastic gradient. Data statistics such as low-order moments (pre-computed or estimated online) is used to form the control variate. We demonstrate how to construct the control variate for two practical problems using stochastic gradient optimization. One is convex-the MAP estimation for logistic regression, and the other is non-convex-stochastic variational inference for latent Dirichlet allocation. On both problems, our approach shows faster convergence and better performance than the classical approach.
quality: 5 (out of 10) clarity: 6 originality: 
      6 significance: 9 SUMMARY: The authors propose to accelerate 
      the stochastic gradient optimization algorithm by reducing the variance of 
      the noisy gradient estimate by using the 'control variate' trick (a 
      standard variance reduction technique for Monte Carlo simulations, 
      explained in [3] for example). The control variate is a vector which 
      hopefully has high correlation with the noisy gradient but for which the 
      expectation is easier to compute. Standard convergence rates for 
      stochastic gradient optimization depend on the variance of the gradient 
      estimates, and thus a variance reduction technique should yield an 
      acceleration of convergence. The authors give examples of control variates 
      by using Taylor approximations of the gradient estimate for the 
      optimization problem arising in regularized logistic regression as well as 
      for MAP estimation for the latent Dirichlet Allocation (LDA) model. They 
      compare constant step-size SGD with and without variance reduction for 
      logistic regression on the covtype dataset, claiming that the variance 
      reduction allows to use bigger step-sizes without having the problem of 
      high variance and thus yields faster empirical convergence. For LDA, they 
      compare the adaptive step-size version of the stochastic optimization 
      method of [10] with and without variance reduction, showing a faster 
      convergence on the held-out test log-likelihood on three large corpora. 
      EVALUATION: Pros: - I like the general idea of variance 
      reduction for SGD using control variates -- it could have a big impact 
      given the popularity of SGD. - The motivation is compelling; the 
      concrete examples of control variates are convincing; and the the general 
      idea (Taylor approximation to define them) seems generalizable - The 
      paper is fairly easy to read. Cons: - The experiments are 
      somewhat weak: only one dataset for logistic regression; and a lack of 
      standardized setup for LDA. - The related work is not covered. 
      QUALITY: The theoretical motivation for the approach is compelling 
      (reducing the variance of the gradient estimates reduces the constant in 
      the convergence rate), but the execution in the empirical section is 
      fairly weak. 1) For logistic regression, they only consider one 
      dataset (covtype). As the previous SGD optimization literature has showed, 
      there can be significant variations of behavior between different datasets 
      (and step-sizes) -- see for example figure 1 and 2 of "A Stochastic 
      Gradient Method with an Exponential Convergence Rate for Finite Training 
      Sets", in the arXiv:1202.6258v4 version which compare SGD with different 
      methods on covtype and other datasets for regularized logistic regression. 
      2) For the LDA experiments, I am puzzled why the authors didn't re-use 
      exactly the same setup as in [10] and [4] so that their results could be 
      comparable? Why using a different mini-batch size (500 vs. 100); a 
      different held-out test set (2k vs. 10k), etc.? It is suspicious that the 
      results of their baseline on the held-out test set (which is the 
      state-of-the-art method in 10] are all systematically worse than what was 
      presented in [10]. The authors should clarify this in their rebuttal. 
      3) Another compelling experimental baseline which is missing is to 
      compare using different mini-batch sizes (which is also a variance 
      reduction technique mentioned in the introduction) vs. their control 
      variate method -- as shown in [4], the CLARITY: The paper is 
      fairly easy to read. I like figure 1. An important point which needs to be 
      clarified is how do they estimate the covariance quantities in their 
      experiments to compute a* (do they use the empirical covariances on the 
      mini-batch? This should be repeated in the experiments section -- and 
      perhaps a discussion of how its cost compare to the standard SGD method 
      should be included). Figure 2 is very hard to read -- the authors should 
      add *markers* to identify the different lines. See also below for more 
      suggestions. ORIGINALITY: I am not familiar of any work using such 
      variance reduction techniques for SGD in such generality. On the other 
      hand, the paper is lacking a coverage of related work. An important 
      missing reference is the paper "Variational Bayesian Inference with 
      Stochastic Search" by John Paisley, David Blei, Michael Jordan ICML 2012 
      which also uses a control variate method to reduce the variance of a 
      stochastic optimization approach to variational mean-field to do 
      variational Bayes (they consider both Bayesian logistic regression and 
      HDPs). The authors should explain in their rebuttal what novel 
      contributions they make in comparison to this prior work. Another relevant 
      piece of work (which is a useful pointer to mention though it doesn't 
      compete with the novelty of this submission) is "Variance Reduction 
      Techniques for Gradient Estimates in Reinforcement Learning" by Evan 
      Greensmith, Peter L. Bartlett, Jonathan Baxter, JMLR 2004. 
      SIGNIFICANCE: In addition to the two applications mentioned in 
      this paper, the approach presented could be most probably generalized to 
      many other settings where SGD has been used. Given the popularity of SGD 
      for large-scale optimization, the potential impact of this work is quite 
      significant. The empirical evidence presented is somewhat weak, but the 
      theoretical intuition is fairly compelling and I could believe that a more 
      thorough empirical comparison could also show significant improvements. I 
      note that their theoretical argument didn't depend on having a finite 
      training set; it would thus be interesting to see this approach used as 
      well in the real stochastic optimization setting (where the full 
      expectation cannot be computed) and where running averages are used to 
      estimate the quantities. == Other detailed comments == 
      line 058: 'discuss' - > discussion line 089: The 
      authors should be clearer that the matrix A *depends on w* as well. 
      equation (5): Cov(g, h) is not symmetric in general -- so the 2nd 
      term should be -(cov(g,h) + cov(g,h)^T). equation (6): it should 
      be cov(h,g) on the RHS [or cov(g,h)^T], not cov(g,h) Paragraph 
      150-154: It might be worthwhile to point out that in the maximal 
      correlation case, one could set h_d = g_d and then the variance becomes 
      zero (but obviously, we cannot compute E[h_d] efficiently in this case). 
      lines 295-296: "This is different from the case in Eq. 11 for 
      logistic regression, which is explicit." - > this sentence is 
      ambiguous. What is explicit? figure 3: For which w did they 
      compute the Pearson's coefficient? Is this using the true covariances or 
      the estimated covariances from the mini-batch? === Update after 
      rebuttal == The authors should carefully cover the related work in 
      their final version of the paper, as well as implement the other 
      corrections mentioned above (I will double check!). I agree that they make 
      a nice contribution over the work by [Paisley et al. 2012]; on the other 
      hand, they should also be clear that [Paisely et al. 2012] were already 
      using control variates to improve the *optimization* of their variational 
      objective; just that perhaps they were not using it in such generality as 
      in this submission. I still think that the experiments are on the weak 
      side, which is why my recommendation for acceptance is not stronger.