Online Bayesian Moment Matching for Topic Modeling with Unknown Number of Topics
Latent Dirichlet Allocation (LDA) is a very popular model for topic modeling as well as many other problems with latent groups. It is both simple and effective. When the number of topics (or latent groups) is unknown, the Hierarchical Dirichlet Process (HDP) provides an elegant non-parametric extension; however, it is a complex model and it is difficult to incorporate prior knowledge since the distribution over topics is implicit. We propose two new models that extend LDA in a simple and intuitive fashion by directly expressing a distribution over the number of topics. We also propose a new online Bayesian moment matching technique to learn the parameters and the number of topics of those models based on streaming data. The approach achieves higher log-likelihood than batch and online HDP with fixed hyperparameters on several corpora. The code is publicly available at https://github.com/whsu/bmm.
The paper aims to propose a simpler model than HDP which will explicitly control the number of topics. The paper is confusing in many parts and lacks minimum essential description to validate the contribution. 
 Although the argument around simplicity appears multiple times in the paper but it is kept vague. I assume it means less number of random variables in the joint distribution of the model. However, line 19 indicates me that HDP is complex due to difficulty in understanding its theoretical structure.   Then it implies that the aim is to extend LDA generative process by sampling the number of topics.  The strategy of the method is to compute likelihood by varying number of topics from 1 to K and pick the one with maximum likelihood. This in some sense does the standard process of selecting number of topics using cross-validation i.e. learn a model with a value for number of topics and then check likelihood on some held-out data and pick the one with maximum likelihood. Here the validation and training sets are same.  But due to prior over T, the algorithm can provide some initial bias.   The concept is interesting, but initial paragraphs could create confusion. Some examples below.  (i) line 5-6, "it is a complex model and it is difficult to incorporate prior knowledge since the distribution over topics is implicit." -- next lines indicate that authors wanted to say the number of topics is not explicitly controlled in HDP, but these lines make it confusing.   (ii) "Uniform process may also be used to avoid priors with rich getting richer property" (line 26-27) -- this is correct but is misleading as the context is topic modeling and uniform process does not apply for grouped datasets. The mechanism to share support among multiple uniform processes is still unknown. Such statements should be carefully made.  In Eq. 1, the challenge is to compute c_n the normalization factor. In sec 4.1 variational approximation has been introduced, then corresponding sequential update equation to Eq. 1 should be made clear. When P_n is made as prior for (n+1)th sample as mentioned in line 147, then there is some difference with exact posterior which is not discussed here. In this context it is relevant to discuss Online Learning for Latent Dirichlet Allocation, Hoffman et al, NIPS 2010 and  Online variational inference for the hierarchical Dirichlet process, Wang et al, AISTATS 2011. Through moment matching the paper could possibly propose alternatives but the essential details have not been discussed here.   In Algorithm 1, value of \gamma depends on \alpha and \beta which in turn depend on \gamma,it is not clear how the values are computed for each sample. Currently it seems there is no iteration and they are initialized only once at the beginning that means truncation level for document n depends on other variables till document n-1. Then it confuses me how we are maximizing likelihood at each step.  This could be improved by discussing sequential update considering variational approximation and moment matching together. In that sense lines 153-154 is misleading. As this paper is about online method based on moment matching then the content lacks required details.   Authors should also note that the number of topics here is controlled by the choice of prior over T or initial value of \gamma, which is regarded as explicit control I think. It is important to discuss that similar behaviour is observed in HDP through the scale variable of DP. That controls the growth in number of topics. If the paper is about explicitness in control over number of topics, then this part requires some discussion.