To Trust Or Not To Trust A Classifier
Knowing when a classifier's prediction can be trusted is useful in many applications and critical for safely using AI. While the bulk of the effort in machine learning research has been towards improving classifier performance, understanding when a classifier's predictions should and should not be trusted has received far less attention. The standard approach is to use the classifier's discriminant or confidence score; however, we show there exists an alternative that is more effective in many situations. We propose a new score, called the trust score, which measures the agreement between the classifier and a modified nearest-neighbor classifier on the testing example. We show empirically that high (low) trust scores produce surprisingly high precision at identifying correctly (incorrectly) classified examples, consistently outperforming the classifier's confidence score as well as many other baselines. Further, under some mild distributional assumptions, we show that if the trust score for an example is high (low), the classifier will likely agree (disagree) with the Bayes-optimal classifier. Our guarantees consist of non-asymptotic rates of statistical consistency under various nonparametric settings and build on recent developments in topological data analysis.
This paper proposes a "trust" score that is supposed to reliably identify whether a prediction is correct or not. The main intuition is that the prediction is more reliable if the instance is closer to the predicted class than other classes. By defining alpha-high-density-set, this paper is able to provide theoretical guarantees of the proposed algorithm, which is the main strength of this paper. This paper proceeds to evaluate the "trust" score using a variety of datasets. One cool experiment is to show that the "trust" score estimated from deeper layers than lower layers.  My major concern about this paper lies in the motivation behind the proposed score. Trust is certainly an important problem for our research community to work on. But I find this paper abuses this term, "trust". In general, trust should involve some notion of humans/users/agents. One definition in Lee and See, 2004 [1] of trust is "the attitude that an agent will help achieve an individual’s goals in a situation characterized by uncertainty and vulnerability". It is concerning that the "trust" score in this paper does not involve any consideration of the end user. For instance, the score will fall in any range and it is unclear that users or experts can determine whether they should trust the classifier based on this score.   A percentile adaptation could help alleviate this issue, which is how this paper conducts the evaluation. However, although the percentile adaptation seems to make the proposed score more interpretable, it carries very different information from calibration: the proposed score only guarantees that instances with high "trust" scores should be more likely to be correct, but the 90 percentile in "trust" score does not necessarily lead to 90% accuracy. This further shows that it can be challenging for any agent to make sense of the proposed "trust" score, even though the precision-percentile curves look nice.  Another difference from calibration that this paper emphasizes is that the ordering preserving property, but this property seems to be a limitation of the current calibration approach. The goal of calibration is that instances with a certain confidence score lead to the same accuracy. So a calibrated classifier may not optimize this precision-percentile curve. Again, I do not find the notion of "trust" score in this paper well motivated. This paper would have been stronger if it can articulate the notion of "trust" and its motivation and difference with confidence with the subject of trusting in mind.  Some relatively minor concerns about the experiment that requires clarification/explanation:  1) How is the percentile computed? Based on the description, it seems that the percentile is determined based on the test set, which seems rather strange. It would be natural to determine the percentile based on the training set.  2) The interpretation of percentile seems to be different from the normal interpretation. Does the highest trust score fall in the 100 percentile or the 0 percentile?  3) What are the chosen alphas? It seems that how well this metric behaves is highly dependent on the geometry space. One can also build a modified nearest-neighbor classifier based on the proposed score. How is the performance of the proposed score in the experiments related to the performance of that modified nearest-neighbor classifier?  4) Interpretable machine learning seems to be an area that is clearly related, but not really cited in the references. For instance, Why should I trust you?: Explaining the Predictions of Any Classifier by Ribeiro, Singh, and Guestrin; Examples are not Enough, Learn to Criticize! Criticism for Interpretability by Kim, Khanna, and Koyejo.  Also, references are not sorted alphabetically.  [1] John D. Lee and Katrina A. See, "Trust in Automation: Designing for Appropriate Reliance", in Human Factors.  I have read the rebuttal. I think that this paper is onto something interesting, but it fails to articular what the proposed ranking is for and how it is superior/complementary to calibration from a human perspective. I would caution against calling this trust.