Adaptive Low-Complexity Sequential Inference for Dirichlet Process Mixture Models
We develop a sequential low-complexity inference procedure for Dirichlet process mixtures of Gaussians for online clustering and parameter estimation when the number of clusters are unknown a-priori. We present an easily computable, closed form parametric expression for the conditional likelihood, in which hyperparameters are recursively updated as a function of the streaming data assuming conjugate priors. Motivated by large-sample asymptotics, we propose a novel adaptive low-complexity design for the Dirichlet process concentration parameter and show that the number of classes grow at most at a logarithmic rate. We further prove that in the large-sample limit, the conditional likelihood and data predictive distribution become asymptotically Gaussian. We demonstrate through experiments on synthetic and real data sets that our approach is superior to other online state-of-the-art methods.
- 'Theorem 1', is a well-know result in the field. It should be easy to find references from any introductory lecture notes in the field. There seems to be nothing original about the proof, and proper attribution should be made anyways.- SUGS updates parameters (step 2) but they are never used since the parameters are marginalized via conjugacy in step 1. This could be misleading.- Moreover, the following argument starting at line 221 is hand-wavy and should be formalized since the following depends on it.- It should be clearly pointed out that the mean of the posterior on alpha as a point estimates is an ad-hoc, non-Bayesian procedure.- The argument in Section 4 assumes a parametric model. This should be clearly acknowledged as a limitation of the argument. Moreover the assumption 1 in the theorem seems fairly strong. The following paragraph mentions that they hold numerically, but the results do not seem very conclusive (given the large peak towards the end of the run). One or way or the other, the language should be changed, as it seems to suggest that their numerical results formally establish that the condition holds, which can never be strictly the case for an a.s. statement.- Claims of 'stability' (in the title of section 4): what is established in this section is that the number of cluster will not grow too fast under strong assumptions. This is not the key issue: one could just put a hard limit of alpha K1 log(n) + K2 if really needed. What one would expect stability to mean is a bound on the closeness of the greedy posterior to the true posterior that do not blow up exponentially in the number of step size. It is unclear that ASUGS has this property even from the numerical example given the small datasets considered.- There are large portions of the work that reproduce known results in the field. For example, section 5 is about standard results on matrix t-distributions, and much of the supplement contains standard, introductory level results in conjugacy. Unless there is something new there, a citation to existing in-depth tutorials (e.g. http://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf) would be more useful.- The evaluation is clearly rushed and incomplete. First, since the motivation is computational, a key missing piece is some experiments that provides information on the trade-off between posterior approximation quality vs. compute cost. - At least one classical, non-sequential methods should be included for completeness, e.g. the split-merge method of R. Neal.- Since an important motivation of the paper is scalability, it would be good to use the full 60k in Section 6.2. - There are not much useful information or insight gained in Section 6.2. More convincing data analysis would strengthen the paper.- Code availability should be clarified. More info on the experimental setup should be added at the very minimum for reproducibility purpose.  Minor presentation issue: 'define the auxiliary variables': I would avoid calling these auxiliary variables, since auxiliary variables have a distinct, specialized meaning in computational Bayesian statistics.