Streaming Variational Bayes
We present SDA-Bayes, a framework for (S)treaming, (D)istributed, (A)synchronous computation of a Bayesian posterior. The framework makes streaming updates to the estimated posterior according to a user-specified approximation batch primitive. We demonstrate the usefulness of our framework, with variational Bayes (VB) as the primitive, by fitting the latent Dirichlet allocation model to two large-scale document collections. We demonstrate the advantages of our algorithm over stochastic variational inference (SVI) by comparing the two after a single pass through a known amount of data-a case where SVI may be applied-and in the streaming setting, where SVI does not apply.
The authors present a Streaming Distributed 
      Asynchronous (SDA) framework for variational Bayesian computation aimed at 
      analysis of large datasets. They assume a variational form q(θ) for the 
      posterior on model parameters that is exponential family, which they 
      express as q(θ) ∝ exp[ξ . T(θ)]. Note that this parameter vector ξ = (χ ν) 
      contains both the cumulant of the conjugate sufficient statistics χ and 
      the scale parameter ν. The authors then define a set of parameters ξ_0 for 
      the variational approximation of the prior p(θ), and make the observation 
      that when the data is split up into batches, independent posteriors ξ_b 
      for each batch may be combined to obtain a approximation for the posterior 
      on the aggregate data ξ = ξ_0 + Σ_b (ξ_b - ξ_0). Τhey propose an algorithm 
      where each new batch is used to update ξ_0, i.e. for each new batch prior 
      is taken to be the posterior after observation of the previous batch. Τhis 
      algorithm can be run in a distributed setting where each worker returns 
      Δξ_b = (ξ_b - ξ_0) to the master, which then updates its copy of ξ_0 <- 
      ξ_0 + Δξ_b accordingly. Τhe proposed methodology is simple but 
      useful, primarily because it presents a trivial parallelization strategy 
      that may be applied to any algorithm that obtains an exponential family 
      estimate of a posterior p(θ | x). The main competitor to this method is 
      stochastic variational inference (SVI), a method that uses a stochastic 
      optimization based on a gradient estimate of the lower bound evidence. A 
      disadvantage of SVI, as the authors note, is that it is sensitive to 
      parameters for the total number of expected documents, batch size, and 
      step size. However I am not entirely convinced this is always a bad thing. 
      While tuning parameters can indeed be costly, having such tunable 
      parameters also makes a method more flexible. The presented results on LDA 
      are promising, but they are not sufficient to demonstrate that SDA can 
      match or exceed SVI results in a variety of domains while eliminating the 
      need for such tunable settings. I would have also liked to have 
      seen a clearer motivation for the proposed distributed update scheme. When 
      comparing their chosen update scheme with one where ξ_0 is held constant 
      for each of the workers, the authors note that the "key difference between 
      the first and second frameworks proposed above is that, in the second, the 
      latest posterior is used as a prior. This latter framework is more in line 
      with the streaming update of Eq. (2) but introduces a new layer of 
      approximation". Unfortunately the implications of this approximation are 
      left unexplored, save for the otherwise unqualified statement that "we 
      find that the latter framework performs better in practice, so we focus on 
      it exclusively in what follows". It would seem that the essential 
      difference between SDA and SVI is that in SDA ξ grows with the number of 
      documents, whereas the variational parameters have a fixed strength in 
      SVI. This implies that in SDA (1) the entropy of the posterior q(θ) 
      decreases as more data is seen, and that (2) convergence is ensured by 
      virtue of the fact that in later updates (ξ_b - ξ_0) << ξ_0. In SVI 
      convergence is achieved by annealing the step size to 0 and ξ has a fixed 
      strength determined by the total number of documents D, which in an 
      on-line setting one might interpret as a buffer size. It would be 
      interesting to see a more detailed analysis of which of these two 
      strategies is more effective in terms of ensuring rapid convergence while 
      avoiding local maxima. In short I believe this may be a promising 
      approach to variational inference in large data sets, chiefly because it 
      is by design suitable to distributed inference in a map-reduce type 
      setting. However I do feel more thorough testing is needed before this 
      method can be considered a drop-in replacement for SVI. -- 
      Minor points / Questions • I see no reason why SVI could not 
      be performed with distributed updates precisely in the manner proposed 
      here for SDA. Have the authors considered this scenario? • Are the 
      authors using their own implementation of SVI in the analysis comparison, 
      Matt Hoffman's code 
      (http://www.cs.princeton.edu/~mdhoffma/code/onlineldavb.tar) or the Vowpal 
      Wabbit implementation (http://hunch.net/~vw)? Given that a run-time 
      comparison is performed, it would be desirable to state this explicitly 
      • The authors write that in SVI "the stochastic gradient is 
      computed for a single data point (document) at a time". This statement is 
      technically correct but somewhat confusing, since it suggests SVI is 
      performed on single data-points, not batches.