Dirichlet belief networks for topic structure learning
Recently, considerable research effort has been devoted to developing deep architectures for topic models to learn topic structures. Although several deep models have been proposed to learn better topic proportions of documents, how to leverage the benefits of deep structures for learning word distributions of topics has not yet been rigorously studied. Here we propose a new multi-layer generative process on word distributions of topics, where each layer consists of a set of topics and each topic is drawn from a mixture of the topics of the layer above. As the topics in all layers can be directly interpreted by words, the proposed model is able to discover interpretable topic hierarchies. As a self-contained module, our model can be flexibly adapted to different kinds of topic models to improve their modelling accuracy and interpretability. Extensive experiments on text corpora demonstrate the advantages of the proposed model.
The paper introduces the Dirichlet Belief Net as a general and deeper drop-in replacement for Dirichlet prior distributions for multinomial data. The author(s) derive an elegant data augmented Gibbs sampler to sample the parameters in what they call the Dirichlet Belief Net. They also give three application examples where this prior can replace standard Dirichlet distribution models, such as in Poisson Factor Analysis, MetaLDA, and the Gamma Belief Networks. The work is motivated by the fact that in the three mentioned models, previous work in deep topic modeling has not been focused on deep structures in the word-topic distribution. I’m no expert in the field of Deep Belief Net, so I’m not sure that similar approaches have not been proposed previously. I’m also not sure exactly how big a contribution this is compared to other similar approaches. But I think the idea is both novel, general and the proposed data augmented Gibbs sampling algorithm make the paper an attractive contribution that could be studied and developed further in other model settings. The proposed approach also seems to produce a better perplexity as well as better topic coherence.  Quality:  The technical content of the paper appears to be correct and the experimental result is convincing that this is a promising approach. There may be a couple of minor typos in the Inference section in the paper:: i) At line 115, I suspect that $q^\beta_{\cdot k}$ should be $q^\psi_{\cdot k_1}}$ ii) At line 116, can it be so that $z_{vk_{2}k_{1}}^{(1)}$ should be $\sum_{k_{2}}^{K_{2}}z_{vk_{2}k_{1}}^{(1)}=y_{vk_{1}}^{(1)}$? That would make more sense for me, but I may be wrong.  One issue is the question of using documentation classification as a method for evaluating topic models. The documentation classification does not show any significant results with the new DirBN. I wonder about the argument of including this as an evaluation metric in the paper. Why would we expect a better classification accuracy if we have a better WP part of the model? I think this should be explained and/or argued for in the paper.  Clarity: The paper is, in general, is well written, I think there are some parts that can be improved to make it easier and more clear for the reader. As an example, I find the Related Work being partly overlapping with the introduction. As an example, see paragraph 2 in the introduction and the last paragraph of the related work section.  Figure 1 was initially a little difficult to follow and understand, but after a while, I think it was quite informative. But maybe the author(s) could add an explanation in the caption, helping the reader to understand the Figure easier, maybe by explaining what the three slides are, since they are different things, even though they look similar.  There are parts in Table 1 that are not explained anywhere in the text, such as the different percentage of the different corpora (I guess this is the percentage of the corpus used as training set?). This should be explained in the paper or in the title caption. Also, I guess that the +- indicate the standard deviation of the different estimates. This should also be stated together with how this was computed in the experiment section. This is not possible to follow or reproduce in the current version.  Minor things: i) At line 80, define what k_{t+1} is in a similar way as k is defined previously. ii) There are also errors in the boldfacing of the results in Table 1 (a) I think MetaLDA DirBN3 has better performance than the boldfaced results for WS20% and TM20%.  Originality: Since I’m not an expert in the field of deep probabilistic models, I am not sure if this work has been done before. But the idea of Dirichlet distributed hidden layers seem to be a good idea and the results are promising.  Significance: The experimental results using the DirBN is promising. Including the DirBN in three different models all improve the model perplexity and topic coherence. The only question is how well the approach would work compared with other similar approaches that has been used to do deep modeling of TP.   The straight-forward MCMC sampling scheme using local conjugacy makes the approach promising and straightforward to use in other models, this in turn make the approach an attractive contribution. The approach can easily be incorporated in other models, studied and developed further.  UPDATE AFTER REBUTTAL:  I think most of my issues were explained. But it did not clarify \emph{why} DPN for WP should increase classification accuracy. Why would we expect this to happen? But this is a minor issue.