The Poisson Gamma Belief Network
To infer a multilayer representation of high-dimensional count vectors, we propose the Poisson gamma belief network (PGBN) that factorizes each of its layers into the product of a connection weight matrix and the nonnegative real hidden units of the next layer. The PGBN's hidden layers are jointly trained with an upward-downward Gibbs sampler, each iteration of which upward samples Dirichlet distributed connection weight vectors starting from the first layer (bottom data layer), and then downward samples gamma distributed hidden units starting from the top hidden layer. The gamma-negative binomial process combined with a layer-wise training strategy allows the PGBN to infer the width of each layer given a fixed budget on the width of the first layer. The PGBN with a single hidden layer reduces to Poisson factor analysis. Example results on text analysis illustrate interesting relationships between the width of the first layer and the inferred network structure, and demonstrate that the PGBN, whose hidden units are imposed with correlated gamma priors, can add more layers to increase its performance gains over Poisson factor analysis, given the same limit on the width of the first layer.
Acknowledge the author's rebuttal.I overall maintain my general sentiment about the paper and look forward to more discussion of this work's relationship to traditional neural / deep networks.---In this paper the authors present a deep belief network in which the intermediate hidden layers are represented by nonnegative weights.They apply this model to text documents as a "deep belief topic model" (my own phrasing).They derive an inference / update step, in which dirichlet vectors are propagated up the network and gamma weights are propagated back down it, and perform an empirical evaluation of the model.Overall this paper was written fairly clearly.I thought the paper was mostly fine, but I believe that the empirical analysis could be improved (more on that shortly).I also would be shocked if nobody has tried to implement a neural network with weights outside the range [0, 1] before; it would be good to see some background material discussed even if it wasn't used to motivate this model.I also had a few other minor comments: - It's not clear whether a fixed budget is appropriate for fixing the size of the various layers (and, in the absence of more detail about the method, I am skeptical).It still seems that these layers could easily be too large or too small despite the fixed budget approach.I suspect that some held-out subset of data could be used to decide whether to grow or shrink the model. - It's not clear whether adding any layers beyond the second layer help.The performance in the figures indicates that they're marginally better, but it's difficult to be confident given how much variance there is in the plots (with the possible exception of Figure 3).Further, the topic descriptions from sample words (line 417) aren't substantially different; it's very possible that the authors' observation that the topics are more specific at the top layer are simply because that layer is largest. - The authors describe a process to jointly train all layers in contrast to training in a greedy fashion (line 264).In this respect it resembles the way a traditional neural network (from 20 years ago) was fit, i.e., forward/back propagation, except that it is with samples. - It seems that this paper could introduce the idea of the Gamma-Poisson belief network without needing to use an application which uses the CRT (i.e., the application itself is complicated, and it's not clear that this paper necessitates a complicated application to evaluate a neural network architecture.Regarding the experimental validation, I believe it could be improved.First, it would be useful to see more comparisons with baselines.For the classification task, for example, it would be good to see an SVM or ridge regression using word counts as covariates, which I have seen outcompete topic models for classification. (N.B. I don't think the model needs to beat these baselines).It would be good to see some comparison with e.g. a typical neural network, in which weights are in [0,1], perhaps with the bottom layer adapted to suit the Poisson/Multinomial observations.Particularly, claims like that on 145 ("..clearly shows that the gamma distributed nonnegative hidden units could carry richer information than the binary hidden units and model more complex nonlinear functions") could an should be backed up with a comparison with a model using binary hidden units.It's not clear that a binary network couldn't express similar nonlinear functions with the right architecture.Is it necessary to remove stopwords?Topic models like LDA can handle them fine if you're okay with a stopword topic (and it's not clear why that would hurt an evaluation).Nit picks:  - The authors discuss classification in the experiment section (line 349) before the actual task is introduced in 353, which was a bit confusing. - Line 56: "budge" -> "budget"