We introduce switched linear projections for expressing the activity of a neuron in a ReLU-based deep neural network in terms of a single linear projection in the input space. The method works by isolating the active subnetwork, a series of linear transformations, that completely determine the entire computation of the deep network for a given input instance. We also propose that for interpretability it is instructive and meaningful to focus on the patterns that deactive the neurons in the network, which are ignored by the exisiting methods that implicitly track only the active aspect of the network's computation. We introduce a novel interpretability method for the inactive state sensitivity (Insens). Comparison against existing methods shows that Insens is robust (in the presence of noise), complete (in terms of patterns that affect the computation) and a very effective interpretability method for deep neural networks.
The work basically introduced a new way of looking at interpretability; instead of focusing on the source of activations in the network for a given input image, focus on the source of stability (non-active) neurons (in a ReLU network). The work starts by proving (although it is trivial) that in a ReLU (more generally any piece-wise linear) network, for a given input image, there is a locally linear relationship between a given neuron's activation and the image: v= w^T x + b. As the authors correctly mention, focusing on 'w' as the sensitivity analysis is basically the vanilla gradient method. The contribution, however, is focusing on the projection of bias and the introduced notion of 'centre'. With this provided notion, one can focus on the deactivated neurons in the network and how each input pixel is responsible for it. In other words, unlike previous work that focuses on the activation map, the authors correctly refer to the deactivated neurons as another source of the network's prediction.
I'm have reasons for both accepting and rejecting this work.  The work provides a new perspective and asks a very interesting question. The introduced method, although quite simple and trivial, is useful and the authors do a very good job of making valid and reasonable claims about their work's contribution and how it connects to the existing literature. The main drawback of the paper, however, is whether the contributions are enough for this venue. The paper does not convince me that the introduced method would result in better interpretability of deep networks compared to what is already there. Another minor (or for some people in the field major) issue is the experimental setup.  All of the experiments are focused on subjective examples and no objective measure of the introduced method is provided (and the field has many of those objective measures). Providing a few examples of the method in comparison with other methods is not sufficient. Anyhow, the experiment where they prove the usefulness of the method by adding background noise is interesting. I would personally suggest the authors to expand this experiment to testing the method's sanity using the sanity measures provided in previous work: https://arxiv.org/abs/1810.03292 The claims made about the results on smallNORB can be controversial as the authors interpret their method's flipping of importance to be the reality of what's happening in the network and the other method's focus on the edges as false; this is not clear to be true. My score would be subject to change if better experimental results are provided (and the other way round).
A few suggestions and questions:
* One very important issue with the method is that it considers all of the inactive neurons. We know that a substantial percentage of inactive neurons are just dead neurons the stability of which does not matter. How would the method address the issue?
* There definitely needs to be an objective measure of the introduced method's performance compared to previous work.
* The work seems very related to DeepLIFT while there is no mentioning of it.
* I'm not a fan, but adding results on a SOTA ImageNET paper always helps with making the experiments section crisper.
* The authors claim that even small perturbations will change the activation pattern. This is not a small claim and is definitely in need of more evidence.