Predicting not only the target but also an accurate measure of uncertainty is important for many applications and in particular safety-critical ones. In this work we study the calibration of uncertainty prediction for regression tasks which often arise in real-world systems. We show that the existing definition for calibration of a regression uncertainty (Kuleshov et al., 2018) has severe limitations in distinguishing informative from non-informative uncertainty predictions. We propose a new definition that escapes this caveat and an evaluation method using a simple histogram-based approach inspired by reliability diagrams used in classification tasks. Our method clusters examples with similar uncertainty prediction and compares the prediction with the empirical uncertainty on these examples. We also propose a simple scaling-based calibration that preforms well in our experimental tests. We show results on both a synthetic, controlled problem and on the object detection bounding-box regression task using the COCO (Lin et al., 2014) and KITTI (Geiger et al., 2012)  datasets.
This paper focuses on the calibration for the regression problem. First, it investigates the shortcomings of a recently proposed calibration metric [1] for regression, and show theoretically where this metric can be fooled. Later, it introduces new metrics for measuring the calibration in regression problem which are instantiated from similar idea as the ECE calibration metric [2] used for classification problem. The paper defines the uncertainty as the mean square error and like the ECE idea, it divides the samples into different uncertainty bins and for each bin, it calculates RMSE of network output uncertainty. The RMSE versus variance of estimated uncertainty is depicted as the reliability diagram.
To summarize the calibration error as the numbers, the paper proposes two other metrics as ENCE and CV that explain the overall calibration error for the network and calibration error for each sample, respectively. It also shows the effectiveness of the metrics, on the synthetic data where the calibration metric proposed in [1] can be fooled VS. more informative metrics like proposed reliability diagram with RMSE and mVAR.  The other experiment is conducted on object detection problem where the uncertainty of bounding box positional outputs is considered as the calibration for the regression problem. 
Pros:
1- Investigating theoretically the shortcoming of the calibration metric for regression is interesting
2- Proposing the new metrics specifically  designed for regression in novel, however it needs more accurate investigation. 
Cons:
The paper is not well-written and the experiments and discussions are not support the ideas, more specifically I can mention several concerns:
1- Motivation: The flaw of the previously proposed calibration metric is not explained clearly. The paper only discusses in which scenario, the metric cannot work properly considering. However, the considered assumption (for instance the output of uncertainty estimation from the network has uniform distribution) can never happen in real scenario. Then the importance of redefining of calibration method is not clear.
2-  Accuracy of Proposed Method: The measure proposed for calibration (Eq. 8) should be shown it will converge to the definition of calibration proposed in Eq.5, which is missing in the paper. 
3- Lack of Clarity in some Parts: The paper introduces the  new architecture for adding the uncertainty output to the regression network. But this new architecture is not explained clearly. I suggest the authors add more details about this part.
4- Justification: The paper trains the uncertainty part of the network with NLL loss function and later fine-tunes the output with temperature scaling method. But this calibration  method is not related to the proposed metric which is claimed it has better calibration clarity. Then the importance of using the new metric to define the calibration is not clear. 
5- Experiments: The experiment setup needs more accuracy. The paper should investigate the shortcomings of the previous metric in the same setting as the new proposed metric. In experiment Sec. 4.1, the settings for obtaining the output of the network uncertainty is different then comparing the results are not fare.  The experiments for real scenario is not wide enough. It just shows that the parameters get calibrated. However we expect to see more results about the CV metric and its importance to define.  
  
Overall, I think this paper should be rejected as 1) the novelty of proposed metrics are not enough. 2) the motivation and justification of why the proposed metrics is better than previous metrics is not clear. 3) the experiments are not supporting the idea.
References:
[1]  Kuleshov, Volodymyr, Nathan Fenner, and Stefano Ermon. "Accurate Uncertainties for Deep Learning Using Calibrated Regression." International Conference on Machine Learning. 2018.
[2] Naeini, Mahdi Pakdaman, Gregory Cooper, and Milos Hauskrecht. "Obtaining well calibrated probabilities using bayesian binning." Twenty-Ninth AAAI Conference on Artificial Intelligence. 2015.