DISCO Nets: DISsimilarity COefficient Networks
We present a new type of probabilistic model which we call DISsimilarity COefficient Networks (DISCO Nets). DISCO Nets allow us to efficiently sample from a posterior distribution parametrised by a neural network. During training, DISCO Nets are learned by minimising the dissimilarity coefficient between the true distribution and the estimated distribution. This allows us to tailor the training to the loss related to the task at hand. We empirically show that (i) by modeling uncertainty on the output value, DISCO Nets outperform equivalent non-probabilistic predictive networks and (ii) DISCO Nets accurately model the uncertainty of the output, outperforming existing probabilistic models based on deep neural networks.
This paper proposes a new way to introduce uncertainty estimation into deep models. It does so by optimising a different cost, the dissimilarity coefficient, which allows for any arbitrary cost to be considered while still capturing the uncertainty of the samples into account.  They apply it on the NYU Hand Pose estimation dataset and compare to GAN and cGAN.
[EDIT] I'd like to thank the authors for their response, which addressed most of my concerns and made me more confident about the paper.  - The connection between using U[0,1] noise and performing Inverse transform sampling makes total sense and is actually quite interesting, it would be really useful to state it in the paper. - The loss restriction point was helpful as well thanks, it is indeed general enough then, no issues. - I completely agree with the assessment of the differences/difficulties in comparing DISCO Net and VAEs. A comparison would be great, but perhaps it would actually be more meaningful and easier to perform it on another dataset entirely, which may prove quite a lot of work just for this paper. I think mentioning the similarities/differences clearly but leaving it for further work would be fine.  [Original review] I'm not sure how I feel about this paper.  I wasn't aware of the dissimilarity coefficient metrics, but they seems to be supersets of Jensen-shannon divergences, so at least they are expressive enough. I'm not sure how useful they would be without actual proofs that P_G(y|x) actually converges to the true one. Hence most of the motivation for the paper is conditioned on the end of Section 3.2 "Proper scoring rule". Hence actually, the distances that can be used are restricted, and they consider Euclidean distance only. I would have appreciated more space to be spent on explaining exactly how well the distributions are properly captured. The authors just inject uniform noise, without any proof that this will be sufficient to capture the correct distribution.  Variational Autoencoders (VAE) are mentioned in the introduction, but are not used for comparison, and this is quite lacking. Indeed at least for VAE the assumptions about the approximations to the posterior are known and well captured. It would have been great to see how a VAE (not really autoencoder, but say a network which takes the depth map as input for the Encoder, and then Decodes back to the poses, while still having a Gaussian prior over the hidden code).  Indeed, I actually think that the particular architecture shown in Figure 2 is really just a slightly different way to represent a VAE (although using a different noise prior and a different optimisation function, which could be either good or bad depending on how much you want to keep the theoretical proofs of VAEs): The Encoder in VAE usually output two estimates: a vector for the mean of the posterior, and another vector for the covariance of the Gaussian posterior (a diagonal covariance actually for efficiency reasons). When training, the reparametrization trick uses samples from a unit N(0,I) gaussian, multiplies it by the covariance posterior estimator and adds it to the mean. This is functionally equivalent to the architecture shown in Figure 2, if we consider the output of the Convolutional layers of DISCO-nets as the mean of the posterior. The concatenated noise sample will be multiplied by a weight matrix, and then added to the next the layers alongside the mean (which will get its own weight matrix as well). Hence the input, as seen from the first dense layer of the decoder, is exactly the same as for VAE: learnt_mean + sampled noise * learnt_variance.  The difference, as said before, is in the form of the sampled noise, what the cost is, and how it's estimated. DISCO-net assume that K samples are used in order to evaluate the cost. This is, again, similar to extensions to VAE which use multiple samples in order to improve the variational lower bound (https://arxiv.org/abs/1509.00519).  So, this is not really a problem, but more an observation, which would be interesting to discuss.   The comparison with the BASE model are good, but I think \sigma should really have been optimized as an hyperparameter to make it fair. At least I'd like to see how a BASE model with \sigma=0 performs on metrics that did not require multiple samples.  Another aspect of my unease with trusting the overall model is the fact that no checks are done on other datasets which require to model the full distribution. The authors could have used more standard datasets (usually MNIST or OMNIGLOT now) for that, or synthetic distributions like done in [http://arxiv.org/abs/1505.05770].  I commend the authors on their attempts to make GAN and cGAN work. They did seem to run into the usual stability issues with them, but as far as I can tell they tried their best and made a fair comparison. Unfortunately, I felt the authors didn't cover the other models as well, in Section 4.4. This section lacks interpretation and discussion, and Figure 4 is neither presented nor commented.   Considering that DISCO Nets do not achieve SOTA compared to these, it would have required a bit more explanation.  The rest of the paper is clear, the supplementary material covers additional training details, and all the information to reproduce their results is easy to find.  In conclusion, I am happy to see another method to construct DNN which capture uncertainty, but given this paper I am not certain of how correct their uncertainty estimate really is.