Online Learning of Nonparametric Mixture Models via Sequential Variational Approximation
Reliance on computationally expensive algorithms for inference has been limiting the use of Bayesian nonparametric models in large scale applications. To tackle this problem, we propose a Bayesian learning algorithm for DP mixture models. Instead of following the conventional paradigm -random initialization plus iterative update, we take an progressive approach. Starting with a given prior, our method recursively transforms it into an approximate posterior through sequential variational approximation. In this process, new components will be incorporated on the fly when needed. The algorithm can reliably estimate a DP mixture model in one pass, making it particularly suited for applications with massive data. Experiments on both synthetic data and real datasets demonstrate remarkable improvement on efficiency -orders of magnitude speed-up compared to the state-of-the-art.
In this paper the authors presented a sequential 
      variational inference algorithm for Dirichlet process mixture models. The 
      authors used a posterior characterization of normalized random measures 
      with independent increments as the basis for a variational distribution 
      that was then used on a sequential decomposition of the posterior. The 
      algorithm was demonstrated on a Gaussian mixture model applied to real and 
      synthetic data, and a non-conjugate DP mixture of Dirichlet distributions 
      to cluster text documents. This is a nice paper that aims to make 
      a particular Bayesian nonparametric model useful to analyze massive data 
      sets. The overall presentation is clear and well-motivated. The 
      experiments are explained clearly and demonstrate advantages of the 
      proposed algorithm. I think that the impact of the paper in 
      expanding the usefulness of Bayesian nonparametric models could be greatly 
      improved by including a supplementary materials section with some proofs 
      and derivations so that a wider audience can understand how the algorithm 
      works and adapt it for their problem. Specifically, including the 
      following would make the paper more self-contained and much clearer: 
      1. A proof of Lemma 1. The theorem that the proof is a result from is 
      in a pretty dense paper. Restating the theorem and showing how Lemma 1 
      follows would be very helpful. 2. Derive Equation 12. The variational 
      distribution is non-standard and it's pretty tedious to derive Equation 
      12. 3. Derive Equations 15 and 16 from Equation 14. The variational 
      distribution is non-standard, the notation for the variational 
      distribution changed from q(D) to q(z,phi), and the sequential variational 
      framework is different enough from the standard setup that it would save 
      some mathematical headaches for the reader to see how Equations 15 and 16 
      arise. A few minor comments on the paper that should be addressed: 
      This work should situate itself with respect to the work on sequential 
      Monte Carlo algorithms for Dirichlet process mixture models which use the 
      same sequential decomposition of the posterior. I think Equation 14 is 
      missing the \phi's in the variational posterior. There are a few 
      parameters of the algorithm, namely \epsilon_d and \epsilon_r, what were 
      these set to for the experiments, or were they found with 
      cross-validation? Lastly, the abbreviation CRF is not standard, why not 
      CRP?