Saliency map, or the gradient of the score function with respect to the input, is the most basic means of interpreting deep neural network decisions. However, saliency maps are often visually noisy. Although several hypotheses were proposed to account for this phenomenon, there is no work that provides a rigorous analysis of noisy saliency maps. This may be a problem as numerous advanced attribution methods were proposed under the assumption that the existing hypotheses are true. In this paper, we identify the cause of noisy saliency maps. Then, we propose Rectified Gradient, a simple method that significantly improves saliency maps by alleviating that cause. Experiments showed effectiveness of our method and its superiority to other attribution methods. Codes and examples for the experiments will be released in public.
In the paper, the authors proposed a new saliency map method, based on some empirical observations about the cause of noisy gradients.
Specifically, through experiments, the authors clarified that the noisy gradients are due to irrelevant information propagated in the forward pass in DNN. Because the backpropagation follows the same pass, irrelevant feature are conveyed back to the input, which results in noisy gradients.
To avoid noisy gradients, the authors proposed a new backpropagation named Rectified Gradient (RectGrad). In RectGrad, the backward pass is filtered out if the product of the forward signal and the backward signal are smaller than a threshold. The authors claim that, with this modification in backpropagation, the gradients get less noisy.
In some experiments, the authors presented that RectGrad can produce clear saliency maps.
I liked the first half of the paper: the observations that irrelevant forward passes are causing noisy gradients seem to be convincing. The experiments are designed well to support the claim.
Here, I would like to point out, that noisy gradients in occluded images may be because of the convolutional structures. Each filter in convolution layer is trained to respond to certain patterns. Because the same filter is used for each of subimages, some filters can be activated occasionally on occluded parts. I think this does not happen if the network is densely connected without convolutional structures. The trained dense connection will be optimized to remove the effects of occluded parts. Hence, for such networks, the gradient will be zeros for occluded parts.
The second half of the paper (Sec.4 and 5) are not very much convincing to me.
Below, I raise several concerns.
1. There is no justification on the definition of RectGrad: Why Rl = I(al * Rl > t) R(l+1)?
The authors presented Rl = I(al * Rl > t) R(l+1) as RectGrad, that can filter out irrelevant passes. However, there is no clear derivation of this formula: the definition suddenly appears. If the irrelevant forward passes are causes of noisy gradients, the modification Rl = I(al > t) R(l+1) seems to be more natural to me. It is also a natural extension to the ReLU backward pass Rl = I(al > 0) R(l+1). Why we need to filter out negative signals in backward pass?
2. The experimental results are less convincing: Is RectGrad truly good?
In Sec.5.2, the authors presented saliency maps only on a few images, and claimed that they look nicely. However, it is not clear that those "nicely looking" saliency map are truly good ones. I expect the authors to put much efforts on quantitative comparisons rather than qualitative comparisons, so that we can understand that those "nicely looking" saliency maps are truly good ones.
Sec.5.3 presents some quantitative comparisons, however, the reported Sensitivity and ROAR/KAR on RectGrad are not significant. The authors mentioned that this may be because of the sparsity of RectGrad. However, if the sparsity is the harm, the underlying observations of RectGrad may have some errors. I think the current manuscript has an inconsistency between the fundamental idea (based on empirical observations) and the performance of RectGrad.
[Minor Concern]
In Sec.5, the authors frequently refer to the figures in appendix. I think the main body of the paper should be self-contatined. I therefore think that some of the figures related to main results should appear in the main part.