MERCI: A NEW METRIC TO EVALUATE THE CORRELATION BETWEEN PREDICTIVE UNCERTAINTY AND TRUE ERROR
As deep learning applications are becoming more and more pervasive, the question of evaluating the reliability of a prediction becomes a central question in the machine learning community. This domain, known as predictive uncertainty, has come under the scrutiny of research groups developing Bayesian approaches to deep learning such as Monte Carlo Dropout. Unfortunately, for the time being, the real goal of predictive uncertainty has been swept under the rug. Indeed, Bayesian approaches are solely evaluated in terms of raw performance of the prediction, while the quality of the estimated uncertainty is not assessed. One contribution of this article is to draw attention on existing metrics developed in the forecast community, designed to evaluate both the sharpness and the calibration of predictive uncertainty. Sharpness refers to the concentration of the predictive distributions and calibration to the consistency between the predicted uncertainty level and the actual errors. We further analyze the behavior of these metrics on regression problems when deep convolutional networks are involved and for several current predictive uncertainty approaches. A second contribution of this article is to propose an alternative metric that is more adapted to the evaluation of relative uncertainty assessment and directly applicable to regression with deep learning. This metric is evaluated and compared with existing ones on a toy dataset as well as on the problem of monocular depth estimation.
 This works presents an overview of different techniques to obtain uncertainty estimates for regression algorithms, as well as metrics to assess the quality of these uncertainty estimates. It then introduces MeRCI, a novel metric that is more suitable for deep learning applications. Being able to build algorithms that are good not only at making predictions, but also at reliably assessing the confidence of these predictions is fundamental in any application. While this is often a focus in many communities, in the deep learning community however this is not the case, so I really like that the authors of this paper want to raise awareness on these techniques. The paper is well written and I enjoyed reading it. I feel that to be more readable for a broader audience it would be relevant to introduce more in depth key concepts such as sharpness and calibration, an not just in a few lines as done in the end of page 2. While I found the theoretical explanation interesting, I feel that the experimental part does not support strongly enough the claims made in the paper. First of all, for this type of paper I would have expected more real-life experiments, and not just the monocular depth estimation one. This is in fact the only way to assess if the findings of the paper generalize. Then, I am not convinced that keeping the networks predictions fixed in all experiments is correct. The different predictive uncertainty methods return both a mean and a variance of the prediction, but it seems that you disregard the information on the mean in you tests. If I understood correctly, I would expect the absolute errors to change for each of the methods, so the comparisons in Figure 4 can be very misleading. With which method did you obtain the predictions in Figure 4.c? Typos: - "implies" -> "imply" in first line of page 3 - "0. 2" -> "0.2" in pag 6, also you should clarify if 0.2 refers to the fraction of units that are dropped or that are kept
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
MERCI: A NEW METRIC TO EVALUATE THE CORRELATION BETWEEN PREDICTIVE UNCERTAINTY AND TRUE ERROR
As deep learning applications are becoming more and more pervasive, the question of evaluating the reliability of a prediction becomes a central question in the machine learning community. This domain, known as predictive uncertainty, has come under the scrutiny of research groups developing Bayesian approaches to deep learning such as Monte Carlo Dropout. Unfortunately, for the time being, the real goal of predictive uncertainty has been swept under the rug. Indeed, Bayesian approaches are solely evaluated in terms of raw performance of the prediction, while the quality of the estimated uncertainty is not assessed. One contribution of this article is to draw attention on existing metrics developed in the forecast community, designed to evaluate both the sharpness and the calibration of predictive uncertainty. Sharpness refers to the concentration of the predictive distributions and calibration to the consistency between the predicted uncertainty level and the actual errors. We further analyze the behavior of these metrics on regression problems when deep convolutional networks are involved and for several current predictive uncertainty approaches. A second contribution of this article is to propose an alternative metric that is more adapted to the evaluation of relative uncertainty assessment and directly applicable to regression with deep learning. This metric is evaluated and compared with existing ones on a toy dataset as well as on the problem of monocular depth estimation.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This works presents an overview of different techniques to obtain uncertainty estimates for regression algorithms, as well as metrics to assess the quality of these uncertainty estimates. It then introduces MeRCI, a novel metric that is more suitable for deep learning applications. Being able to build algorithms that are good not only at making predictions, but also at reliably assessing the confidence of these predictions is fundamental in any application. While this is often a focus in many communities, in the deep learning community however this is not the case, so I really like that the authors of this paper want to raise awareness on these techniques. The paper is well written and I enjoyed reading it. I feel that to be more readable for a broader audience it would be relevant to introduce more in depth key concepts such as sharpness and calibration, an not just in a few lines as done in the end of page 2. While I found the theoretical explanation interesting, I feel that the experimental part does not support strongly enough the claims made in the paper. First of all, for this type of paper I would have expected more real-life experiments, and not just the monocular depth estimation one. This is in fact the only way to assess if the findings of the paper generalize. Then, I am not convinced that keeping the networks predictions fixed in all experiments is correct. The different predictive uncertainty methods return both a mean and a variance of the prediction, but it seems that you disregard the information on the mean in you tests. If I understood correctly, I would expect the absolute errors to change for each of the methods, so the comparisons in Figure 4 can be very misleading. With which method did you obtain the predictions in Figure 4.c? Typos: - "implies" -> "imply" in first line of page 3 - "0. 2" -> "0.2" in pag 6, also you should clarify if 0.2 refers to the fraction of units that are dropped or that are kept
assistant
 This works presents an overview of different techniques to obtain uncertainty estimates for regression algorithms, as well as metrics to assess the quality of these uncertainty estimates. It then introduces MeRCI, a novel metric that is more suitable for deep learning applications. Being able to build algorithms that are good not only at making predictions, but also at reliably assessing the confidence of these predictions is fundamental in any application. While this is often a focus in many communities, in the deep learning community however this is not the case, so I really like that the authors of this paper want to raise awareness on these techniques. The paper is well written and I enjoyed reading it. I feel that to be more readable for a broader audience it would be relevant to introduce more in depth key concepts such as sharpness and calibration, an not just in a few lines as done in the end of page 2. While I found the theoretical explanation interesting, I feel that the experimental part does not support strongly enough the claims made in the paper. First of all, for this type of paper I would have expected more real-life experiments, and not just the monocular depth estimation one. This is in fact the only way to assess if the findings of the paper generalize. Then, I am not convinced that keeping the networks predictions fixed in all experiments is correct. The different predictive uncertainty methods return both a mean and a variance of the prediction, but it seems that you disregard the information on the mean in you tests. If I understood correctly, I would expect the absolute errors to change for each of the methods, so the comparisons in Figure 4 can be very misleading. With which method did you obtain the predictions in Figure 4.c? Typos: - "implies" -> "imply" in first line of page 3 - "0. 2" -> "0.2" in pag 6, also you should clarify if 0.2 refers to the fraction of units that are dropped or that are kept