We introduce our Distribution Regression Network (DRN) which performs regression from input probability distributions to output probability distributions. Compared to existing methods, DRN learns with fewer model parameters and easily extends to multiple input and multiple output distributions. On synthetic and real-world datasets, DRN performs similarly or better than the state-of-the-art. Furthermore, DRN generalizes the conventional multilayer perceptron (MLP). In the framework of MLP, each node encodes a real number, whereas in DRN, each node encodes a probability distribution.
Summary:
This paper presents a new network architecture for learning a regression of probability distributions.
The distribution output from a given node is defined in terms of a learned conditional probability function, and the output distributions of its input nodes. The conditional probability function is an unnormalized distribution with the same form as the Boltzman distribution, and distributions are approximated from point estimates by discretizing the finite support into predefined equal-sized bins. By letting the conditional distribution between nodes be unnormalized, and using an energy function that incorporates child nodes independently, the approach admits efficient computation that does not need to model the interaction between the distributions output by nodes at a given level.
Under these dynamics and discretization, the chain rule can be used to derive a matrix of gradients at each node that denotes the derivative of the discretized output distribution with respect to the current node's discretized distribution. These gradients are in turn used to calculate updates for the network parameters with respect to the Jensen Shannon divergence between the predicted distribution and a target distribution.
The approach is evaluated on three tasks, two synthetic and one real world. The baselines are the state of the art triple basis estimator (3BE) or a standard MLP that represents the output distribution using a softmax over quantiles. On both of the synthetic tasks --- which involve predicting gaussians --- the proposed approach can fit the data reasonably using far fewer parameters than the baselines, although 3BE does achieve better overall performance. On a real world task that involves predicting a distribution of future stock market prices from multiple input stock marked distributions, the proposed approach significantly outperforms both baselines. However, this experiment uses 3BE outside of its intended use case --- which is for a single input distribution --- so it's not entirely clear how well the very simple proposed model is doing.
Notes to authors:
I'm not familiar with 3BE but the fact that it is used outside of its intended use case for the stock data is worrying. How does 3BE perform at predicting the FTSE distribution at time t + k from the FTSE distribution at time t only? Do the multiple input distributions actually help?
You use a kernel density estimate with a Gaussian kernel function to estimate the stock market pdf, but then you apply your network directly to this estimate. What would happen if you built more complex networks using the kernel values themselves as inputs?
Could you also run experiments on the real-world datasets used by the 3BE paper?
What is the structure of the DRN that uses > 10^3 parameters (from Fig. 4)? The width of the network is bounded by the two input distributions, so is this network just incredibly deep? Also, is it reasonable to assume that both the DRN and MLP are overfitting the toy task when they have access to an order of magnitude more parameters than datapoints.
It would be nice if section 2.4 was expanded to actually define the cost gradients for the network parameters, either in line or in an appendix.