Rethinking LDA: Moment Matching for Discrete ICA
We consider moment matching techniques for estimation in latent Dirichlet allocation (LDA). By drawing explicit links between LDA and discrete versions of independent component analysis (ICA), we first derive a new set of cumulantbased tensors, with an improved sample complexity. Moreover, we reuse standard ICA techniques such as joint diagonalization of tensors to improve over existing methods based on the tensor power method. In an extensive set of experiments on both synthetic and real datasets, we show that our new combination of tensors and orthogonal joint diagonalization techniques outperforms existing moment matching methods.
The paper builds on the moment matching methods for standard LDA, providing related algorithms that estimate topic token distributions in a fully specified generative LDA model (one with a specific prior distribution for document length). I found the paper quite clear, the approach interesting, and the results convincing. My only suggestion is to clean up the notation a bit - the zoo of models, variables, and indices makes it more difficult to parse than it should be.Specifically:I don't think the Discrete PCA section is necessary.It seems like the GP model is just a stepping-stone to the more general DICA model for which results are presented, linking DICA to the natural choice of Poisson distribution for L.However, it might be more clear to skip directly to the DICA model, and bring up earlier the meaning of alpha's (topic intensities).At the end of section 4, the authors mention that the violation of the non-negativity constraint is practically significant.What consequence does that have on convergence, stability?This issue seems important enough to expand a bit.Regarding the last point made in Conclusions, is it at least possible to estimate the relative scales of topic intensities given assumptions on D?Minor things:line 137, 669 "low of total" to "law of total"line 298 "some up" to "sum up"line 366 we set b in (4)(?)