Natural-Parameter Networks: A Class of Probabilistic Neural Networks
Neural networks (NN) have achieved state-of-the-art performance in various applications. Unfortunately in applications where training data is insufficient, they are often prone to overfitting. One effective way to alleviate this problem is to exploit the Bayesian approach by using Bayesian neural networks (BNN). Another shortcoming of NN is the lack of flexibility to customize different distributions for the weights and neurons according to the data, as is often done in probabilistic graphical models. To address these problems, we propose a class of probabilistic neural networks, dubbed natural-parameter networks (NPN), as a novel and lightweight Bayesian treatment of NN. NPN allows the usage of arbitrary exponential-family distributions to model the weights and neurons. Different from traditional NN and BNN, NPN takes distributions as input and goes through layers of transformation before producing distributions to match the target output distributions. As a Bayesian treatment, efficient backpropagation (BP) is performed to learn the natural parameters for the distributions over both the weights and neurons. The output distributions of each layer, as byproducts, may be used as second-order representations for the associated tasks such as link prediction. Experiments on real-world datasets show that NPN can achieve state-of-the-art performance.
The paper presents a new Bayesian treatment for probabilistic neural networks. The approach allows to specify and estimate distributions within the exponential family for all parameters of the model, as well as its inputs an outputs. Besides, the framework allows for multi-layer architectures and nonlinear activation functions, akin to standard neural networks. With their approach, the authors aim to alleviate two well-know problems of standard neural networks, namely, overfitting when data is insufficient and lack of flexibility in terms of available distributions for weights and neurons.  The authors formulate neural networks in terms of parameters' distributions rather than the parameters themselves, which in principle allow the model to capture a richer representation of the data by modeling uncertainty. For additional flexibility, nonlinear activation functions are defined for a variety of distributions within the exponential family.  Learning is done via back-propagation and the authors provide details for 3 instances of their framework, namely, gamma, Gaussian and Poisson.  Experiments focus on handwritten digits classification (MNIST) and second-order representation learning for text documents using bag-of-words representations.   
My main concern with the paper, on a conceptual level, is that the authors claim to use a Bayesian treatment of neural networks however, the authors make no mention whatsoever to posterior/predictive distributions and the only mention to priors is reserved to Gaussian distributions in the context of regularization. Also, all parameters of the model (weights and neurons) are specified via fully factorized univariate distributions. How is this different from a straightforward mean-filed variational Bayes treatment of neural networks with moment matching for activation functions? One of the most appealing elements of Bayesian analysis is the ability of estimating posterior/predictive distributions to quantify uncertainty via capturing the covariance structure of the data, however such task is very difficult with fully factorized specifications as it is well-known from the variational inference literature.  In the experiments, it is not clear to me what is the added value of the approach presented by the authors for three reasons: i) it is difficult to evaluate the significance of the results in Table 3 without error bars. Besides, for Size = 100, although NPN outperforms Dropout, it is not entirely obvious that the latter is overfitting considerably more than the former. ii) Why are the results for unsupervised models evaluated in terms on (indirect measure) link prediction rather than (direct measure) say perplexities or marginal likelihood estimates? iii) One of the main points of the authors is that standard neural networks are limited in terms of distribution choices, yet the authors show that on MNIST a Gaussian model outperforms the Gamma model which in principle is better suited for non-negative data. Besides on the unsupervised task, the Gaussian model compares favorably against the Poisson model, but again results in Table 4 are difficult to interpret as they indirectly quantify the representational capabilities of the models being considered.    It would be nice to get some clarity in the following points:  - How are the parameters of the activation functions in Table 3 selected/tuned in practice.  - How good is the approximation in (5), from the supplementary material, it seems that (5) comes from a well-known result for the probit link function, however, no details are provided about the accuracy of the approximation.  - In (6), what prevents the variance, a_s, from becoming a negative number? and if that is the case how it is handled in practice?  - In the Gaussian NPN model, outputs, y, are assumed Gaussian with variance epsilon, how is epsilon selected and how it affects performance?   - For Poisson NPN, how are counts obtained from the (continuous) approximation proposed by the authors?  - The authors should avoid using the term calibration so liberally, as the authors do not provide results to directly ascertain calibration, instead results simply indicate correlation between uncertainty (variance) and accuracy.