Obtaining reliable uncertainty estimates of neural network predictions is a long standing challenge. Bayesian neural networks have been proposed as a solution, but it remains open how to specify their prior. In particular, the common practice of a standard normal prior in weight space imposes only weak regularities, causing the function posterior to possibly generalize in unforeseen ways on inputs outside of the training distribution. We propose noise contrastive priors (NCPs) to obtain reliable uncertainty estimates. The key idea is to train the model to output high uncertainty for data points outside of the training distribution. NCPs do so using an input prior, which adds noise to the inputs of the current mini batch, and an output prior, which is a wide distribution given these inputs. NCPs are compatible with any model that can output uncertainty estimates, are easy to scale, and yield reliable uncertainty estimates throughout training. Empirically, we show that NCPs prevent overfitting outside of the training distribution and result in uncertainty estimates that are useful for active learning. We demonstrate the scalability of our method on the flight delays data set, where we significantly improve upon previously published results.
The paper considers the problem of obtaining reliable predictive uncertainty estimates. The authors propose noise contrastive priors — the idea being to explicitly encourage high uncertainties for out of distribution (OOD) data through a loss in the data space.  OOD data is simulated by adding noise to existing data and the model is trained to maximize the likelihood wr.t. training data while being close in the KL sense to a (wide) conditional prior p(y | x) on the OOD responses (y).  The authors demonstrate that the procedure leads to improved uncertainty estimates on toy data and can better drive active learning on a large flight delay dataset.
The paper is well written and makes for a nice read. I like the idea of using “pseudo” OOD data for encouraging better behaved uncertainties away from the data. It is nice to see that even simple schemes for generating OOD data (adding iid noise) lead to improved uncertainty estimates. 
My main concern about this work stems from not knowing how sensitive the recovered uncertainties are to the OOD data generating mechanism and the parameters thereof. The paper provides little evidence to conclude one way or the other.  The detailed comments below further elaborate on this concern.
Detailed Comments: 
a) I like the sensitivity analysis presented in Figure 4, and it does show for the 1D sine wave the method is reasonably robust to the choice of \sigma_x. However, it is unclear how problem dependent the choice of sigma_x is. From the experiments, it seems that \sigma_x needs to be carefully chosen for different problems, \sigma^2_x < 0.3 seems to not work very well for BBB + NCP for the 1D sine data, but for the flight delay data \sigma^2_x is set to 0.1 and seems to work well. How was \sigma_x chosen for the different experiments?
b) It is also interesting that noise with a shared scale is used for all 8 dimensions of the flight dataset. Is this choice mainly governed by convenience — easier to select one hyper-parameter rather than eight? 
c) Presumably, the predictive uncertainties are also strongly affected by both the weighting parameter \gamma and the prior variance sigma^2_y . How sensitive are the uncertainties to these and how were these values chosen for the experiments presented in the paper? 
d) It would be really interesting to see how well the approach extends to data with more interesting correlations. For example, for image data would using standard data-augmentation techniques (affine transformations) for generating OOD data help over adding iid noise. In general, it would be good to have at least some empirical validation of the proposed approach on moderate-to-high dimensional data (such as images).
==============
Overall this is an interesting paper that could be significantly strengthened by addressing the comments above and a more careful discussion of how the procedure for generating OOD data affects the corresponding uncertainties.