Attribution methods provide insights into the decision-making of machine learning models like artificial neural networks. For a given input sample, they assign a relevance score to each individual input variable, such as the pixels of an image. In this work we adapt the information bottleneck concept for attribution. By adding noise to intermediate feature maps we restrict the flow of information and can quantify (in bits) how much information image regions provide. We compare our method against ten baselines using three different metrics on VGG-16 and ResNet-50, and find that our methods outperform all baselines in five out of six settings. The method's information-theoretic foundation provides an absolute frame of reference for attribution values (bits) and a guarantee that regions scored close to zero are not necessary for the network's decision. Deep neural networks have become state of the art in many realworld applications. However, their increasing complexity makes it difficult to explain the model's output. For some applications such as in medical decision making or autonomous driving, model interpretability is an important requirement with legal implications. Attribution methods (Selvaraju et al., 2017; Zeiler & Fergus, 2014;Smilkov et al., 2017) aim to explain the model behaviour by assigning a relevance score to each individual input variable. When applied to images, the relevance scores can be visualised as heatmaps ovä er the input pixel space, thus highlighting salient areas relevant for the network's decision.
Summary
---
(motivation)
Lots of methods produce attribution maps (heat maps, saliency maps, visual explantions) that aim to highlight input regions with respect to a given CNN.
These methods produce scores that highlight regions that are in a vague sense "important."
While that's useful (relative importance is interesting), the scores don't mean anything by themselves.
This paper introduces another new attribution method that measures the amount of information (in bits!) each input region contains, calibrating this score by providing a reference point at 0 bits.
Non-highlighted regions contribute 0 bits of information to the task, so they are clearly irrelevant in the common sense that they have 0 mutual information with the correct output.
(approach - attribution methods)
An information bottleneck is introduced by replacing a layer's (e.g., conv2) output X with a noisy version Z of that output.
In particular, Z is a convex combination of the feature map (e.g., conv2) with Gaussian noise with the same mean and variance as that feature map.
The weights of the combination are found so they minimize the information shared between the input and Z and maxmimize information shared between Z and the task output Y.
These weights are either optimized on
1) a per-image basis (Per-Sample) or
2) predicted by a model trained on the entire dataset (Readout).
(approach - evaluation)
The paper uses 3 metrics with differing degrees of novelty:
1) The bbox metric rewards attribution methods that put a lot of mass in ground truth bounding boxes.
2) The original Sensitivity-n metric from (Ancona et al. 2017) is reported with a version that uses 8x8 occlusions.
3) Least relevant image degredation is compared to most relevant image degredation (e.g., from (Ancona et al. 2017)) to form a new occlusion style metric.
(experiments)
Experiments consider many of the most popular baselines, including Occlusion, Gradients, SmoothGrad, Integrated Gradients, GuidedBP, LRP, Grad-CAM, and Pattern Attribution. They show:
1) Qualitatively, the visualizations highlight only regions that seem relevant.
2) Both Per-Sample and Readout approaches put higher confidence into ground truth bounding boxes than all other baselines.
3) Both Per-Sample and Readout approaches outperform all baselines almost all the time according to the new image degredation metric.
Strengths
---
The idea makes a lot of sense. I think heat maps are often thought of in terms of the colloquial sense of information, so it makes sense to formalize that intuition.
The related work section is very well done. The first paragraph is particularly good because it gives not just a fairly comprehensive view of attribution methods, but also because it efficiently describes how they all work.
The results show that proposed approaches clearly outperform many strong baselines across different metrics most of the time.
Weaknesses
---
* I'm not sure why the new degredation metric is a useful addition. What does it add that MoRF and LeRF don't capture on their own independently?
* I think [1] would be a nice addition to the evaluation section as it tests for something qualitatively different than the various metrics from section 4. It would also be a good addition to the related work.
Missing Details / Points of Confusion
---
* I think there's an extra p(x) in eq. 11 in appendix D.
* I think the variable X is overloaded. In eq. 1 it refers to the input (e.g., the pixels of an image) while in eq. 2 it refers to an intermediate feature map (e.g., conv2) even though it later seems to refer to the input again (e.g., eq. 3). Different notation should be used for intermediate feature maps and inputs.
Presentation Weaknesses
---
* In section 3.1 is lambda meant to be constrained in the range [0, 1]? This is only mentioned later (section 3.2) and should probably be mentioned when lambda is introduced.
* "indicating that all negative evidence was removed." I think this should read "indicating that only negative evidence was removed."
Suggestions
---
"The bottleneck is inserted into an early layer to ensure that the information in the network is still local"
I'd like this to be explored a bit more. Though deeper feature maps are certainly more spatially coarse they still might be somewhat "local". To what degree to they loose localization information? My equally vague alternative intuition goes a bit differently: The amount of relevant information flowing through any spatial location seems like it shouldn't change that much, only the way its represented should change. If the proposed visualizations were the same for every choice of layer then it would confirm this intuition. That would also be an interesting result because most if not all of the cited baseline approaches (where applicable) produce qualitatively different attributions at different layers (e.g., see Grad-CAM).
[1]: Adebayo, Julius et al. “Sanity Checks for Saliency Maps.” NeurIPS (2018).
Preliminary Evaluation
---
Clarity: The paper is clearly written.
Originality: The idea of using the formal notion of information in attribution maps is novel, as is the bbox metric.
Significance: This method could be quite significant. I can see it becoming an important method to compare to.
Quality: The idea is sound and the evaluation is strong.
This is a very nice paper in all the ways listed above and it should be accepted!
Post-rebuttal comments
---
The author responses and other reviews have only increased my confidence that this paper should be accepted.