Scalable Inference for Logistic-Normal Topic Models
Logistic-normal topic models can effectively discover correlation structures among latent topics. However, their inference remains a challenge because of the non-conjugacy between the logistic-normal prior and multinomial topic mixing proportions. Existing algorithms either make restricting mean-field assumptions or are not scalable to large-scale applications. This paper presents a partially collapsed Gibbs sampling algorithm that approaches the provably correct distribution by exploring the ideas of data augmentation. To improve time efficiency, we further present a parallel implementation that can deal with large-scale applications and learn the correlation structures of thousands of topics from millions of documents. Extensive empirical results demonstrate the promise.
The authors describe a two novel inference methods for 
      the correlated topic model (CTM). They build on analytic results for 
      the conditional logistic normal likelihood to arrive at a fast, 
      easily parallelized exact inference. This leads to an approximate 
      sampling method for producing Polya-Gamma variates. Finally, they 
      propose a method for efficiently drawing samples in the presence of 
      sparsity. To demonstrate the utility of the topic correlations 
      inferred by the correlated topic model, the authors show a 
      hierarchical clustering of 1000 topics learned from the 300k document 
      New York Times dataset. While the figure makes a good case for the 
      usefulness of these sort of correlations, it seems a bit out of place 
      in the present paper, where the focus is on speeding up inference of 
      an existing model as opposed to a suggestion of a new model. This page of 
      the paper could have been better used further expanding on the details 
      of the inference. To assess the effectiveness of their 
      inference, the authors compare against a variational CTM inference on 
      the New York Times dataset. The authors examine perplexity and 
      training time of the variational method and both serial and parallel 
      runs of their inference, in both cases finding little degradation in 
      perplexity performance while showing up to two orders of magnitude 
      faster training times. To show the scalability of the 
      approach, the authors compare the training time and perplexity of 
      their approach with that of Y!LDA, both running on 480 distributed 
      processors. Here they demonstrate similar perplexity and while Y!LDA 
      significantly outperforms CTM in training runtime, this must be 
      weighed against the value of the correlation structure that CTM 
      produces. The authors give a brief discussion to the effect of the 
      various burn-in times of the inference and show that the priors have 
      no measurable effect on perplexity. That being said, this is 
      demonstrated only over one small corpus with 100 topics. How robust 
      are these parameters to other corpus sizes and numbers of topics? 
      Finally, the authors note that the training time of their 
      parallel inference is nearly constant as document count and number of 
      compute nodes are increased in proportion. This is a clear 
      strength of the method. The authors put forth a compelling 
      inference for a challenging model. They demonstrate prediction 
      performance comparable to existing, but much slower, variational 
      methods while significantly improving runtime. While the discussion of 
      the inference itself could be expanded (especially in light of the 
      page spent on Figure 3), the contribution is clear and enables the CTM 
      to be used on datasets far larger than previously feasible. 
      This paper comprises a solid contribution to the topic modeling 
      community. So far, the slow inference of CTMs have prevented the community 
      from building on it. This paper has the potential to change this. The 
      derivation is sound and the evaluation is convincing. 
      -------------------- Update of comments after the 
      discussion period: In my understanding Y!LDA refers to a 
      non-correlated version of LDA. I was impressed that this model is able to 
      learn this more complicated model for a computational price that is 
      comparable to Y!LDA, which is heavily tuned and less expressive. I 
      prefer papers that attempt a new approach, even if they do not strictly 
      outperform a heavily tuned baseline. I think this approach is worthwhile 
      considering and may lead to further improvements as people pick the idea 
      up and refine it. The fact that neither me nor 
      Assigned_Reviewer_7 was not familiar the Polya-Gamma distribution is a 
      sign that this knowledge should be promoted in the community, even if it 
      is a straight forward application of work on Polya Gamma distributions. I 
      would appreciate if relations to the paper of Polson, Scott and Windle, 
      2013 would be clarified in the writeup. I have not seen many 
      practical applications of the correlated topic model, and I think this is 
      partially due to problems with the slow inference. I am hoping that this 
      paper will motivate some new inference methods for this and other complex 
      topic models, and that's why I would like to see it at the conference.