A plethora of methods attempting to explain predictions of black-box models have been proposed by the Explainable Artificial Intelligence (XAI) community. Yet, measuring the quality of the generated explanations is largely unexplored, making quantitative comparisons non-trivial. In this work, we propose a suite of multifaceted metrics that enables us to objectively compare explainers based on the correctness, consistency, as well as the confidence of the generated explanations. These metrics are computationally inexpensive, do not require model-retraining and can be used across different data modalities. We evaluate them on common explainers such as Grad-CAM, SmoothGrad, LIME and Integrated Gradients. Our experiments show that the proposed metrics reflect qualitative observations reported in earlier works.
See post-rebuttal updates below!
Summary
---
(motivation)
There are lots of heat map/saliency/visual explanation approaches that try to deep image classifiers more interpretable.
It's hard to tell which ones are good, so we need better ways of evaluating explanations.
This paper proposes 3 such explanation evaluation metrics, correctness, consistency, and confidence.
(approach - correctness)
An explanation is correct if it highlights enough of an image for a classifier to tell the correct class with only the highlight parts of the image.
The default way to evaluate on only highlighted portions is to set the non-highlighted bckground to black/grey.
Instead, this method finds images with the same ground truth class which the classifier scored the lowest of all such images, forming a low-confidence baseline.
It copies the background from one of these images instead of using a black/grey background
to try and put the masked image back into the distribution of images from the ground truth class.
This style of masking is used to compute correctness.
(approach - consistency)
An explanation is consistent if it is invariance w.r.t. a number of mostly semantically invariant transformations.
These include small affine transformations, horizontal flips, vertical flips, and adding noise.
(approach - confidence)
An explanation is confident if the masked images it produces still have high condidence under the classifier.
Masked images are produced as for correctness, by copying a distractor from the same class into the background.
(experiments)
The experiments compare existing explanations (LIME, Grad-CAM, Integrated Gradients, SmoothGrad) using the proposed metrics.
1. Correctness: Classifiers have higher accuracy on explanation-masked images than on images they were least confident on (the ones used to fill in the background).
2. Grad-CAM is most correct, followed by SmoothGrad, Integrated Gradients, and LIME.
3. Consistency: Grad-CAM explanations are most resilient to the proposed transformations with Integrated Gradients, SmoothGrad, and LIME being successively less invariant.
4. Confidence: Explanation-masked images have higher scores for their ground truth class than the low-confidence baseline images.
5. Hyperparameter variations in the correcness/confidence metrics mostly preserve the ranking of methods, though the absolute values of performance do change substantially.
(conclusion)
The paper concludes that Grad-CAM is usually the best of the methods tested according to the new metrics and that LIME is the worst.
Strengths
---
I really like the related work section. It could be a valuable resource going forward.
I like the research direction of this paper very much. I think that enumerating a suite of complementary benchmarks is a good way to measure explanation quality because we can only come up with benchmarks that capture a small part of what we want so far.
Weaknesses
---
I see some major conceptual flaws with these metrics:
* In section 3.1 it seems like the first reasons that normal masking failed is not solved by the proposed approach. The generated images are still out of distribution because the "foreground" and the "background" don't match.
* I'm concerned about the low-confidence distractor images used in the background. They are from the same ground truth class as the high confidence images they are pasted into the background of, correct? The correctness metric is supposed to capture whether or not an explanation highlights all the class-relevant content in an image and no more. However, information that the explanation did not highlight (the background) can inform the classifier of the ground truth class because the background came from an image of that class (even if a low confidence one). This is especially true because the relevant objects might be in differrent positions in the two images. Thus it could be that the explanation did not highlight informative content but the classifier still gets the corresponding masked image correct because of the background. How often does this happen?
* Consistency is supposed to measure "the ability of the explainer to capture the relevant components" under semantically invariant transformations.
The reported metric is mimized when the explanation is the same before and after a variety of transformations.
If this were the case then at least one of them must be wrong in the sense that it would not have captured some relevant components
(unless perhaps it just highlighted everything and was thus useless).
Because of the transformation (e.g. 15 degree rotation) the relevant components would have been at a different position, but the best explanation according
to this metric would have been at the same position. Thus this metric seems to reward explanations for not capturing relevant components.
Parts I Didn't Understand:
* In section 3.1, I don't understand the second reason that masking failed. In what sense is masking made meaningless? How is that sense different from the out of distribution concern from the first point?
Missing Details / Presentation Weaknesses:
* Missing reference to [1] which provides more metrics.
* The meaning of confidence is different than it normally is and this may be confusing.
Neural networks should be well calibrated, not necessarily confident (in the commonly used sense of [3]).
Minor flaws:
* Masking by replacing the background with grey (i.e., the bias of the first conv layer) rather than black is more common (e.g., [2] and Grad-CAM). A grey background negates the bias. It's not clear that the background should cancel the bias, but it would be nice to compare to both grey and black masking in Table 7.
[1]: Adebayo, Julius et al. “Sanity Checks for Saliency Maps.” NeurIPS (2018).
[2]: Zeiler, Matthew D. and Rob Fergus. “Visualizing and Understanding Convolutional Networks.” ECCV (2013).
[3]: Guo, Chuan et al. “On Calibration of Modern Neural Networks.” ICML (2017).
Final Evaluation
---
This paper relies solely on theoretical arguments to show its metrics capture meaningful information. Empirically, it only shows that the proposed metrics can differentiate between some popular explanations. It does not empirically show that the differentiation is meaningful (e.g., by measuring agreement with human judgement). This by itself isn't a problem. However, above I detailed significant flaws in the theoretical justification for the metrics, so I can't recommend these metrics (this paper) on either a theoretical or an empirical basis.
Quality: Per above, I do not think the arguments/evidence in the paper support its conclusions.
Clarity: The paper could be clearer, but can be understood without too much effort.
Originality: These metrics are new enough, being novel variations on prior approaches.
Significance: If I was convinced the metrics made sense then I would guess this paper would be very impactful. As is, I don't think it will have much impact.
The quality of the paper is my reason for the low rating. I'm interested to see whether what others think to make sure I've understood the paper correctly and analyzed it accurately. If my understanding is incorrect I could definitely raise my rating.
Post-Rebuttal Evaluation
---
After reading the other reviews and the author responses and taking a brief look at the updated paper I still think this paper should be rejected.
The authors' response to my comments clarified my understanding of the consistency metric. Now I understand it and think it is a useful metric.
However, I did not find clarification about the confidence or correctness metrics, though I agree they are not redundant. They still don't really quite make sense to me. This puts me in about the same position as R3, who also doubts those metrics. In the end, this leaves my initial evaluation essentially unchanged. I still recommend rejection because the paper relies on a theoretical understanding of what makes confidence and correctness metrics useful and that understanding is not provided.