Fixed-Length Poisson MRF: Adding Dependencies to the Multinomial
We propose a novel distribution that generalizes the Multinomial distribution to enable dependencies between dimensions. Our novel distribution is based on the parametric form of the Poisson MRF model [1] but is fundamentally different because of the domain restriction to a fixed-length vector like in a Multinomial where the number of trials is fixed or known. Thus, we propose the Fixed-Length Poisson MRF (LPMRF) distribution. We develop AIS sampling methods to estimate the likelihood and log partition function (i.e. the log normalizing constant), which was not developed for the Poisson MRF model. In addition, we propose novel mixture and topic models that use LPMRF as a base distribution and discuss the similarities and differences with previous topic models such as the recently proposed Admixture of Poisson MRFs [2]. We show the effectiveness of our LPMRF distribution over Multinomial models by evaluating the test set perplexity on a dataset of abstracts and Wikipedia. Qualitatively, we show that the positive dependencies discovered by LPMRF are interesting and intuitive. Finally, we show that our algorithms are fast and have good scaling (code available online).
This paper proposes a tractable extension of the Poisson MRF model that can accommodate positive dependencies. This model, the Fixed-Length Poisson MRF (LPMRF), is then used as the emission distribution for an admixture model similar to LDA. In this model, each topic carries additional information about pairwise dependencies (correlations) between word types. Applying this model to text, the authors obtain interesting and competitive results using only 3 topics. My primary technical question for this work is: Where is the line between modeling topics and modeling word dependencies? If we had a perfect model of word dependencies, would we need a mixture model over topics? Are the topics simply accounting for higher-order (ternary, etc) dependencies that cannot be captured easily by this particular model of word dependencies? With enough topics, the top words are more than enough to look at as results from LDA. The intuitive nature of word type dependencies might be much more questionable if you used your model with more topics. This is especially true because each topic is capturing certain positive and negative dependencies which might interact across topics in ways that are very hard to analyze or present intuitively.A key concern is scalability of the approach. How well can this method scale with vocabulary size? Especially as the number of topics increases, it seems like the memory requirements might be prohibitive. How many word types were in your data set? Why didn't you test your model with more topics?This paper is very interesting in terms of its modeling contribution and the development of a fixed-length PMRF with different properties from the original PMRF distribution. The paper has a relatively small contribution in terms of inference, because existing techniques which optimize a pseudo-likelihood are applied. Why did you choose to optimize rather than sample your parameters?You mention the Admixture of Poisson MRFs model several times in your paper, but you do not discuss any details. Please address this in a related work discussion. In addition, please cite collocation literature from linguistics and contrast your approach; you look for long-range dependences rather than words following one another .The output of the model looks similar, except the order of words in the pairs don't matter.In section 3, you say that LDAdoes not "capture dependencies between words because there is only one word being drawn at a time". While this is true from a generative point of view, I would argue that from an inferential point of view, allowing enough topics helps us to capture many types of positive dependencies.The proposed model shows only small gains over LDA in terms of perplexity. Perplexity is known to be a bad indicator of topic quality, and there are many existing topic models with better perplexity scores than LDA.The negative dependencies in Table 1 don't make nearly as much sense as the positive dependences. Please comment on this.More interesting applications for your model might actually be outside of topic modeling, e.g. in event count modeling for political science where the relationships are less well understood and where the relationships themselves are more relevant as direct objects of study.Small commentsBring Figure 1 up to page 2. Figure 1 is very effective. What L did you use to produce these graphs?Standardize your model names and acronyms. What does CPMRF stand for in Eq 1?What size of dependency matrix did you use to produce Figure 3?For perplexity results, Figure 4, note in the caption that lower is better.Increase your font size in all figures!Do not use \theta and \Theta as different parameters! Likewise, don't use 'e' both as the well-known constant and as a vector of all 1's in your proof. These notations are hard to follow.Move your Monte Carlo sampling section before your Upper Bound section in 2.1.On Page 5, use bold font for "LPMRF" in the generative process to emphasize that this model is your paper's contribution.In section 4 you say you optimize the Dirichlet parameter \beta. Do you mean \alpha?