Deep Poisson Factor Modeling
We propose a new deep architecture for topic modeling, based on Poisson Factor Analysis (PFA) modules. The model is composed of a Poisson distribution to model observed vectors of counts, as well as a deep hierarchy of hidden binary units. Rather than using logistic functions to characterize the probability that a latent binary unit is on, we employ a Bernoulli-Poisson link, which allows PFA modules to be used repeatedly in the deep architecture. We also describe an approach to build discriminative topic models, by adapting PFA modules. We derive efficient inference via MCMC and stochastic variational methods, that scale with the number of non-zeros in the data and binary units, yielding significant efficiency, relative to models based on logistic links. Experiments on several corpora demonstrate the advantages of our model when compared to related deep models.
This paper proposes a novel probabilistic deep architecture for modelling count data. In addition the authors propose a modelling extension for multi-class classification problems (i.e., discriminative modelling). The authors derive two inference techniques, one sampling one and a variational inference one, and show empirical gains compared to unsupervised methods using several large-scale datasets.Questions/Comments: - I found the model to be clearly explained and motivated. I enjoyed reading Section 2.3. I think adding a figure to illustrate Equations 3 would be useful. - I think it may be slightly misleading to imply that [23] does not scale wrt to the number of zeros. My understanding is that using a Poisson likelihood allows all methods to effectively ignore the zeros. Further it seems like most of the computational gains come from that rather than from higher-layers (which are typically much smaller). [23] also does allow  non-linear activation functions. - Computation. It appears that your model scales well. It would be interesting to have an idea of how computation scales and how long it takes to learn on these larger datasets. Providing a rough comparison to competing models (docNADE, LDA, and replicated softmax) would also be useful. - When reporting the results of the classification experiments it seems like you are only comparing to unsupervised techniques. In that sense the comparison is not absolutely fair. It would be good to add, at least, one simple supervised baseline (e.g., a small neural net with a softmax output and the word frequencies as inputs).Other comments: - line 165: Sparsity of the Dirichlet relies on your choice of parameters (eta). I think it would be good to make it clear.