Capturing Semantically Meaningful Word Dependencies with an Admixture of Poisson MRFs
We develop a fast algorithm for the Admixture of Poisson MRFs (APM) topic model [1] and propose a novel metric to directly evaluate this model. The APM topic model recently introduced by Inouye et al. [1] is the first topic model that allows for word dependencies within each topic unlike in previous topic models like LDA that assume independence between words within a topic. Research in both the semantic coherence of a topic models [2,3,4,5] and measures of model fitness [6] provide strong support that explicitly modeling word dependencies-as in APM-could be both semantically meaningful and essential for appropriately modeling real text data. Though APM shows significant promise for providing a better topic model, APM has a high computational complexity because O(p 2 ) parameters must be estimated where p is the number of words ([1] could only provide results for datasets with p = 200). In light of this, we develop a parallel alternating Newton-like algorithm for training the APM model that can handle p = 10 4 as an important step towards scaling to large datasets. In addition, Inouye et al.
[1] only provided tentative and inconclusive results on the utility of APM. Thus, motivated by simple intuitions and previous evaluations of topic models, we propose a novel evaluation metric based on human evocation scores between word pairs (i.e. how much one word "brings to mind" another word [7]). We provide compelling quantitative and qualitative results on the BNC corpus that demonstrate the superiority of APM over previous topic models for identifying semantically meaningful word dependencies. (MATLAB code available at:
The paper addresses the computational challenge for a recently proposed admixture models of Poisson MRFs to model word dependencies in topic models. The authors proposed a parallelizable Newton-like algorithm which would speed up the computation. Although the paper is technically sound, I do have couple of concerns regarding to the proposed method and experiment results.- The paper would be much stronger if comparing admixture model with Poisson MRF with some other models. Many other topic models also model the correlation on either document and/or words. For example, hierarchical topic models with nested CRP by Wang et al. models the hierarchies on the topics, hence the correlations among topics and words. Boyd-Graber et al. also proposed topic models for word sense disambiguation using WordNet. Comparison against these models would be a much stronger evidence for the model.- This paper uses the evocation evaluation metric. It would be great to include some other standard evaluation of correlations, such as PMI (Newman, et al.) of coherence score (Mimno, et al.).- One other big concern I have regarding to the model is the scalability of the algorithm. The experiments are running on only one dataset, and it is relatively small, e.g., the BNC dataset contains less than 5k documents, with about 1500 word types only. Even though, from the experiment results, it seems like the model took around half an hour to train, after parallelization. It would be great to demonstrate that the model is able to scale to much larger datasets and hence, be potentially more useful to the research community. In addition, the authors conduct experiments only on the BNC dataset, it would be great to include experiments evaluation on other larger datasets to demonstrate the model is efficient and effective.The paper is considerably well written, and well organized, except couple of comments.- The paper heavily rely on the knowledge of the recent proposed admixture model of Poisson MRFs, which is a very strong assumption about the readers. Authors should include more explanation about the background knowledge. - Authors could draw connections with other topic models extensions for correlation modeling.In terms of significance, the paper address the scalability issue of a recent proposed admixture model of Poisson MRFs, but the experimental results seem less convincing. Author should also focus on the scalability of the model, and examine the algorithm on a much larger dataset to demonstrate the effectiveness and efficiency of the model. Without these, it is hard to see whether the paper would make a significant impact to the research community.%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%After check Authors' feedback:One of my biggest concern is the real scalability of this algorithm. The authors scaled out the inference, but the experiments with only one small dataset are not convincing to show the scalability, which make me doubt how well the proposed algorithm can scale up.In the author's feedback, they provide a little bit more results with 10k vocab. This is a reasonable size. If they integrate the new results to final version and release code, this is more convincing to me now. But on the other hand, the evocation evaluation should be better explained and supported, and it would be great if standard evaluation scores like PMI are included as well.