Saliency methods aim to explain the predictions of deep neural networks. These methods lack reliability when the explanation is sensitive to factors that do not contribute to the model prediction. We use a simple and common pre-processing step -adding a mean shift to the input data-to show that a transformation with no effect on the model can cause numerous methods to incorrectly attribute. We define input invariance as the requirement that a saliency method mirror the sensitivity of the model with respect to transformations of the input. We show, through several examples, that saliency methods that do not satisfy a input invariance property are unreliable and can lead to misleading and inaccurate attribution.
Saliency methods are effective tools for interpreting the computation performed by DNNs, but evaluating the quality of interpretations given by saliency methods are often largely heuristic. Previous work has tried to address this shortcoming by proposing that saliency methods should satisfy "implementation invariance", which says that models that compute the same function should be assigned the same interpretation. This paper builds on this work by proposing and studying "input invariance", a specific kind of implementation invariance between two DNNs that compute identical functions but where the input is preprocessed in different ways. Then, they examine whether a number of existing saliency methods satisfy this property.
The property of "implementation invariance" proposed in prior work seems poorly motivated, since the entire point of interpretations is that they should explain the computation performed by a specific network. Even if two DNNs compute the same function, they may do so using very different computations, in which case it seems natural that their interpretations should be different. Nevertheless, I can believe that the narrower property of input invariance should hold for saliency methods.
A much more important concern I have is that the proposed input invariance property is not well motivated. A standard preprocessing step for DNNs is to normalize the training data, for example, by subtracting the mean and dividing by the standard deviation. Similarly, for image data, pixel values are typically normalized to [0,1]. Assuming inputs are transformed in this way, the input invariance property (for mean shift) is always trivially satisfied. The paper does not justify why we should consider networks where the training data is not normalized is such a way.
Even if the input is not normalized, the failures they find in existing saliency methods are typically rather trivial. For example, for the gradient times input method, they are simply noting that the interpretation is translated by the gradient times the mean shift. The paper does not discuss why this shift matters. It is not at all clear to me that the quality of the interpretation is adversely affected by these shifts.
I believe the notion that saliency methods should be invariant to input transformations may be promising, but more interesting transformations must be considered -- as far as I can tell, the property of invariance to linear transformations to the input does not provide any interesting insight into the correctness of saliency methods.