Sanity Checks for Saliency Maps
Saliency methods have emerged as a popular tool to highlight features in an input deemed relevant for the prediction of a learned model. Several saliency methods have been proposed, often guided by visual appeal on image data. In this work, we propose an actionable methodology to evaluate what kinds of explanations a given method can and cannot provide. We find that reliance, solely, on visual assessment can be misleading. Through extensive experiments we show that some existing saliency methods are independent both of the model and of the data generating process. Consequently, methods that fail the proposed tests are inadequate for tasks that are sensitive to either data or model, such as, finding outliers in the data, explaining the relationship between inputs and outputs that the model learned, and debugging the model. We interpret our findings through an analogy with edge detection in images, a technique that requires neither training data nor model. Theory in the case of a linear model and a single-layer convolutional neural network supports our experimental findings 2 .
Summary ---  This paper introduces a new set of criteria for evaluating saliency/heatmap visual explanations of neural net predictions. It evaluates some existing explanation approaches and shows that some of them do not pass these simple tests.  The first test compares explanations of a base model to explanations of the same model with its weights randomly re-initialized. If the same visualization results from both sets of weights then it does not reflect anything the neural net has learned. Similarly, the second test compares a base model to the same model trained with randomly permuted labels. In this case the weights are not randomly initialized, but still cannot contain much information specific to the task (image classification) because the labels were meaningless.  For both tests pairs of visualizations are compared by manual inspection and by computing rank correlation between the two visualizations. Layers are re-initialized starting from the output and moving toward the input, generating an explanation for each additional layer. Techniques including Integrated Gradients, Guided BackProp, and Guided GradCAM produce very similar explanations whether all parameters are random or not. Other techniques (Gradient and GradCAM) drastically change the explanations produced after the first layer is randomized and continue to produce these different and appropriately noisy explanations as more layers are randomized.  The surprising conclusion this work offers is that some methods which attempt to explain neural nets do not significantly change their explanations for some significantly different neural nets. To explain this phenomena it analyzes explanations of a linear model and of a single conv layer, claiming that the conv layer is similar to an edge detector. It also acknowledges that architectural priors play a significant role in the performance of conv nets and that random features are not trivial.  Strengths ---  I buy the basic idea behind the approach. I think this is a good principle to use for evaluating saliency style visual explanations, although it doesn't act as an ultimate form of quality, just potentially useful initial sanity checks. The criteria has clarity that other approaches to saliency evaluation lack.  Overall, the set of experiments is fairly large.  Weaknesses ---       None of these weaknesses stand out as major and they are not ordered by importance.  * Role of and relation to human judgement: Visual explanations are useless if humans do not interpret them correctly (see framework in [1]). This point is largely ignored by other saliency papers, but I would like to see it addressed (at least in brief) more often. What conclusions are humans supposed to make using these explanations? How can we be confident that users will draw correct conclusions and not incorrect ones? Do the proposed sanity checks help identify explanation methods which are more human friendly? Even if the answer to the last question is no, it would be useful to discuss.  * Role of architectures: Section 5.3 addresses the concern that architectural priors could lead to meaningful explanations. I suggest toning down some of the bolder claims in the rest of the paper to allude to this section (e.g. "properties of the model" -> "model parameters"; l103). Hint at the nature of the independence when it is first introduced.  Incomplete or incorrect claims:  * l84: The explanation of GBP seems incorrect. Gradients are set to 0, not activations. Was the implementation correct?  * l86-87: GradCAM uses the gradient of classification output w.r.t. feature map, not gradient of feature map w.r.t. input. Furthermore, the Guided GradCAM maps in figure 1 and throughout the paper appear incorrect. They look exactly (pixel for pixel) equivalent to the GBP maps directly to their left. This should not be the case (e.g., in the first column of figure 2 the GradCAM map assigns 0 weight to the top left corner, but somehow that corner is still non-0 for Guided GradCAM). The GradCAM maps look like they're correct.  l194-196: These methods are only equivalent gradient * input in the case of piecewise linear activations.  l125: Which rank correlation is used?  Theoretical analysis and similarity to edge detector:  * l33-34: The explanations are only somewhat similar to an edge detector, and differences could reflect model differences. Even if the same, they might result from a model which is more complex than an edge detector. This presentation should be a bit more careful.  * The analysis of a conv layer is rather hand wavy. It is not clear to me that edges should appear in the produced saliency mask as claimed at l241. The evidence in figure 6 helps, but it is not completely convincing and the visualizations do not (strictly speaking) immitate an edge detector (e.g., look at the vegitation in front of the lighthouse). It would be useful to include a conv layer initialized with sobel filter and a canny edge detector in figure 6. Also, quantitative experimental results comparing an edge detector to the other visual explanations would help. Figure 14 makes me doubt this analysis more because many non-edge parts of the bird are included in the explanations.  Although this work already provides a fairly large set of experiments there are some highly relevant experiments which weren't considered:  * How much does this result rely on the particular (re)intialization method? Which initialization method was used? If it was different than the one used to train the model then what justifies the choice?  * How do these explanations change with hyperparameters like choice of activation function (e.g., for non piecewise linear choices). How do LRP/DeepLIFT (for non piecewise linear activations) perform?  * What if the layers are randomized in the other direction (from input to output)? Is it still the classifier layer that matters most?  * The difference between gradient * input in Fig3C/Fig2 and Fig3A/E is striking. Point that out.  * A figure and/or quantitative results for section 3.2 would be helpful. Just how similar are the results?   Quality ---  There are a lot of weaknesses above and some of them apply to the scientific quality of the work but I do not think any of them fundamentally undercut the main result.  Clarity ---  The paper was clear enough, though I point out some minor problems below.  Minor presentation details:  * l17: Incomplete citation: "[cite several saliency methods]"  * l122/126: At first it says only the weights of a specific layer are randomized, next it says that weights from input to specific layer are randomized, and finally (from the figures and their captions) it says reinitialization occurs between logits and the indicated layer.  * Are GBP and IG hiding under the input * gradient curve in Fig3A/E?  * The presentation would be better if it presented the proposed approach as one metric (e.g., with a name), something other papers could cite and optimize for.  * GradCAM is removed from some figures in the supplement and Gradient-VG is added without explanation.   Originality ---  A number of papers evaluate visual explanations but none have used this approach to my knowledge.  Significance ---  This paper could lead to better visual explanations. It's a good metric, but it only provides sanity checks and can't identify really good explanations, only bad ones. Optimizing for this metric would not get the community a lot farther than it is today, though it would probably help.  In summary, this paper is a 7 because of novelty and potential impact. I wouldn't argue too strongly against rejection because of the experimental and presentation flaws pointed out above. If those were fixed I would argue strongly against rejection.  [1]: Doshi-Velez, Finale and Been Kim. “A Roadmap for a Rigorous Science of Interpretability.” CoRR abs/1702.08608 (2017): n. pag.