Cooperative neural networks (CoNN): Exploiting prior independence structure for improved classification
We propose a new approach, called cooperative neural networks (CoNN), which uses a set of cooperatively trained neural networks to capture latent representations that exploit prior given independence structure. The model is more flexible than traditional graphical models based on exponential family distributions, but incorporates more domain specific prior structure than traditional deep networks or variational autoencoders. The framework is very general and can be used to exploit the independence structure of any graphical model. We illustrate the technique by showing that we can transfer the independence structure of the popular Latent Dirichlet Allocation (LDA) model to a cooperative neural network, CoNNsLDA. Empirical evaluation of CoNN-sLDA on supervised text classification tasks demonstrates that the theoretical advantages of prior independence structure can be realized in practice -we demonstrate a 23% reduction in error on the challenging MultiSent data set compared to state-of-the-art.
This paper presents a novel approach for combining neural networks with probabilistic graphical models such as LDA through learning Hilbert space embeddings. The idea is to incorporate domain specific prior structure into the more flexible NN-based learning. There is a growing body of literature in this area, and this work introduces some interesting ideas in this domain.   Main strengths:  - Well motivated work  - Interesting new modeling approach, joint constraint networks - Experimental comparisons with other competing approaches  Main weaknesses: - Description of model derivations are vague and grandiose in some places (see below for examples)  - Experimental results are not too impressive, and  some of the strengths of the proposed model, such as ability to inject prior domain structure and need for reduced training data are not demonstrated effectively   The modeling approach is very well motivated with the promise of combining best of both worlds of graphical models and deep learning. Specifically, the need to adapt to limited domain learning problems and to generalize over the vocabulary is certainly important. However, the effectiveness of your model along these dimensions are not effectively demonstrated in the experimental work. For example, it is not clear what domain prior structure is injected into the model, other than selecting the size of Hilbert space dimension. Similarly, the need for reduced training data can be shown by plotting learning curves. I recognize that this is a general missing trend in this area of research, but it would be worth considering some of these additional experiments. Also, given that the results on 20 NG are so close, you might consider also testing your approach on a few other text classification corpora, such as RCV1, and others.   Here are some parts of the model description that are vague and require more clarifications or support of evidence:   127-128: although their actual form is complex, maintaining this complete form is unnecessary because it will be different after the embedding anyway.  130: Here, f_1 and f_2 denote the complex dependency structure of equations (5) and (6), respectively 144-146: ..... in order to classify documents accurately naturally causes the embeddings to be injective exactly where they need to be.  169-170: many of the properties and intuitions behind the original LDA model are preserved with the embedded model (this claim needs to be backed by scientific experiments)   Typographical/grammatical errors:  21: constraint  114: corresponding probability density