Distance-Based Learning from Errors for Confidence Calibration
Deep neural networks (DNNs) are poorly-calibrated when trained in conventional ways. To improve confidence calibration of DNNs, we propose a novel training method, distance-based learning from errors (DBLE). DBLE bases its confidence estimation on distances in the representation space. We first adapt prototypical learning for training of a classification model for DBLE. It yields a representation space where a test sample*s distance to its ground-truth class center can calibrate the model*s performance. At inference, however, these distances are not available due to the lack of ground-truth label. To circumvent this by approximately inferring the distance for every test sample, we propose to train a confidence model jointly with the classification model, by merely learning from mis-classified training samples, which we show to be highly-beneficial for effective learning. On multiple data sets and DNN architectures, we demonstrate that DBLE outperforms alternative single-modal confidence calibration approaches. DBLE also achieves comparable performance with computationally-expensive ensemble approaches with lower computational cost and lower number of parameters.
The paper describes an approach to improve the confidence on deep neural networks (DNN). The proposed approach uses a distance-based approach (distance to prototypes) and train using a confidence model with the classification model using mis-classified examples. It first learns using a loss function based on pre-computed centroids of each class evaluated from the training examples. At inference it assigns the class with the closest center. To estimate a confidence level, it learns a new model to estimate the distance using only the misclassified examples. The authors experimentally showed the benefits of the proposed approach over several methods. Finding prototypes and training with distances, on one hand, and evaluating confidence with Gaussian, on the other, assumes that the classes are *well defined*. Although the authors show promising results, it is not clear how well the proposed method will behave in more challenging classification tasks where the classes are mixed. Although the authors show that estimating the confidence with misclassified examples works better I will like to see a further analysis in the paper.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Distance-Based Learning from Errors for Confidence Calibration
Deep neural networks (DNNs) are poorly-calibrated when trained in conventional ways. To improve confidence calibration of DNNs, we propose a novel training method, distance-based learning from errors (DBLE). DBLE bases its confidence estimation on distances in the representation space. We first adapt prototypical learning for training of a classification model for DBLE. It yields a representation space where a test sample*s distance to its ground-truth class center can calibrate the model*s performance. At inference, however, these distances are not available due to the lack of ground-truth label. To circumvent this by approximately inferring the distance for every test sample, we propose to train a confidence model jointly with the classification model, by merely learning from mis-classified training samples, which we show to be highly-beneficial for effective learning. On multiple data sets and DNN architectures, we demonstrate that DBLE outperforms alternative single-modal confidence calibration approaches. DBLE also achieves comparable performance with computationally-expensive ensemble approaches with lower computational cost and lower number of parameters.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper describes an approach to improve the confidence on deep neural networks (DNN). The proposed approach uses a distance-based approach (distance to prototypes) and train using a confidence model with the classification model using mis-classified examples. It first learns using a loss function based on pre-computed centroids of each class evaluated from the training examples. At inference it assigns the class with the closest center. To estimate a confidence level, it learns a new model to estimate the distance using only the misclassified examples. The authors experimentally showed the benefits of the proposed approach over several methods. Finding prototypes and training with distances, on one hand, and evaluating confidence with Gaussian, on the other, assumes that the classes are *well defined*. Although the authors show promising results, it is not clear how well the proposed method will behave in more challenging classification tasks where the classes are mixed. Although the authors show that estimating the confidence with misclassified examples works better I will like to see a further analysis in the paper.
assistant
The paper describes an approach to improve the confidence on deep neural networks (DNN). The proposed approach uses a distance-based approach (distance to prototypes) and train using a confidence model with the classification model using mis-classified examples. It first learns using a loss function based on pre-computed centroids of each class evaluated from the training examples. At inference it assigns the class with the closest center. To estimate a confidence level, it learns a new model to estimate the distance using only the misclassified examples. The authors experimentally showed the benefits of the proposed approach over several methods. Finding prototypes and training with distances, on one hand, and evaluating confidence with Gaussian, on the other, assumes that the classes are *well defined*. Although the authors show promising results, it is not clear how well the proposed method will behave in more challenging classification tasks where the classes are mixed. Although the authors show that estimating the confidence with misclassified examples works better I will like to see a further analysis in the paper.