As deep learning applications are becoming more and more pervasive, the question of evaluating the reliability of a prediction becomes a central question in the machine learning community. This domain, known as predictive uncertainty, has come under the scrutiny of research groups developing Bayesian approaches to deep learning such as Monte Carlo Dropout. Unfortunately, for the time being, the real goal of predictive uncertainty has been swept under the rug. Indeed, Bayesian approaches are solely evaluated in terms of raw performance of the prediction, while the quality of the estimated uncertainty itself is not assessed. One contribution of this article is to draw attention on existing metrics developed in the forecast community, designed to evaluate both the sharpness and the calibration of predictive uncertainty. Sharpness refers to the concentration of the predictive distributions and calibration to the consistency between the predicted uncertainty level and the actual errors. We further analyze the behavior of these metrics on regression problems when deep convolutional networks are involved and for several current predictive uncertainty approaches. A second contribution of this article is to propose an alternative metric that is more adapted to the evaluation of relative uncertainty assessment and directly applicable to regression with deep learning. This metric is evaluated and compared with existing ones on a toy dataset as well as on the problem of monocular depth estimation.
The authors propose mean rescaled confidence interval (MERCI) as a way to measure the quality of predictive uncertainty for regression problems. The main idea is to rescale confidence intervals, and use average width of the confidence intervals as a measure for calibration. Due to the rescaling, the MERCI score is insensitive to the absolute scale; while this could be a feature in some cases, it can also be problematic in applications where the absolute scale of uncertainty matters. 
Overall, the current draft feels a bit preliminary. The current draft misses discussion of other relevant papers, makes some incorrect claims, and the experiments are a bit limited. I encourage the authors to revise and submit to a different venue. 
	
There’s a very relevant ICML 2018 paper on calibrating regression using similar idea: 
Accurate Uncertainties for Deep Learning Using Calibrated Regression
https://arxiv.org/pdf/1807.00263.pdf
Can you clarify if/how the proposed work differs from this? I’d also like to see a discussion of calibration post-processing methods such as Platt scaling and isotonic regression.
The paper unfairly dismisses prior work by making factually incorrect claims, e.g. Section 2 claims
“Indeed, papers like (Hernandez-Lobato & Adams, 2015; Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017; Kendall & Gal, 2017) propose quantitative evaluations on several datasets, those classically used for evaluating the task, but only compare their average test performances in terms of RMSE. It is the quality of the prediction which is measured, and not the quality of the estimated uncertainty. They also show some qualitative results, where maps of the estimated uncertainty are displayed as images and visually evaluated. Yet, to the best of our knowledge, the literature on deep neural networks does not propose any method for the quantitative evaluation of the uncertainty estimates.”
This is incorrect.  To just name a few examples of prior work quantitatively evaluating the quality of uncertainty: (Hernandez-Lobato & Adams, 2015) and (Gal & Ghahramani, 2016) report log-likelihoods on regression tasks, (Lakshminarayanan et al. 2017) report log-likelihoods and Brier score on classification and regression tasks. There are many more examples. 
The experiments are a bit limited. Figure 1 is a toy dataset and Table 2 / Figure 4 focus on a single test case which does not seem like a fair comparison of the different methods. The authors should at least compare their method to other work on the UCI regression benchmarks used by (Hernandez-Lobato & Adams, 2015).