HOW IMPORTANT IS A NEURON?
The problem of attributing a deep network's prediction to its input/base features is well-studied (cf. Simonyan et al. (2013)). We introduce the notion of conductance to extend the notion of attribution to understanding the importance of hidden units. Informally, the conductance of a hidden unit of a deep network is the flow of attribution via this hidden unit. We can use conductance to understand the importance of a hidden unit to the prediction for a specific input, or over a set of inputs. We justify conductance in multiple ways via a qualitative comparison with other methods, via some axiomatic results, and via an empirical evaluation based on a feature selection task. The empirical evaluations are done using the Inception network over ImageNet data, and a convolutional network over text data. In both cases, we demonstrate the effectiveness of conductance in identifying interesting insights about the internal workings of these networks.
This paper presents a new method to measure the importance of hidden neurons in deep neural networks. The method integrates notions of activation value, input influence to a neuron and neuron influence to the network's output. They provide results confirming that this measure is able to identify neurons that are important for specific tasks.
Quality
The experiments are well designed to verify their hypothesis, although there could be more to make sure those results are not particular to the few selected problems. Nevertheless, the results are consistent across those experiments.
Clarity
The text is well written in general, but the structure could be improved. The introduction contains too much related work, which should be divided in another section. Section 2 contains mostly high level explanations of the work, which should be integrated in the introduction, and thus before the related work section, to improve readability. See minor comments for more specific suggestions.
It is difficult to understand the goal of Section 4.2. Section 2 states that section 4.2 proves that a "path method" must be used in order to satisfy the axioms, but why such axioms are important is not stressed enough. Also, it is not clear why those are called axioms since they are not use to build anything else. It seems to me that those are rather "desirable properties" than axioms.
Originality
A important number of related works are cited and compared with the current work. Although the proposed measure is close to what is proposed by Datta et al., this paper makes the distinction clear and benchmarks its results properly against it.
Significance
There is an increasing need to interpretability of deep neural networks as they get more and more applied to real-world problems. Measures as the one proposed in this paper are a very important building block towards this.
Conclusion
For its original importance measure and the proper experiment benchmarks, I believe this paper should be accepted. There is however many minor issues that should be fixed for the camera-ready version. Although the recommended length is 8 pages, the strict limit is 10, so I would recommended to use a bit of the remaining extra space to conclude the paper properly with a discussion on the results and their consequences, as well as a conclusion to wrap up the paper.
***
Minor Comments
Introduction:
- The term flow is never defined precisely, we need to infer it based on the definition of conductance and attribution.
- First paragraph would be more clear with simple word explanation rather than maths. Also, second sentence is not a complete sentence
- Work on image indicators of importance could be compared better with current work. Indicators can be seen as a measure of importance.
- This sentence is not clear: "[...]; the nature of correlations in the two models may differ".
Section 2:
- Last paragraph of section 2 can be true for any well-performing importance measure. The statements should be put in perspective with others.
Section 3:
- Section 3 should be introduced by explaining the goal of the section otherwise it breaks the flow of reading.
- The role of the baseline x' should be better explained when it is presented (first paragraph of section 3).
- The interchangeable use of the term "conductance of neuron j" for equations 2 and 3 is confusing. Different terms should be use, even if the context makes it possible to infer which one is being referred to.
- Remark 1 seems trivial, but the selection of baseline x' seems less trivial. Some explanations should be devoted to it.
- Second paragraph of remark 1 is not clear. Why couldn't we take another layer's neuron as the neuron of interest, bounding the conductance measure on one layer as the input and the output of the model? If we make the input to be a neuron y rather than the true input x, we could take another neuron z in a subsequent layer to be the neuron of interest, resulting in conductance measure Cond^z_i(y).
Section 4:
- List of importance measure at beginning of Section 4 should probably have citations.
- Backward reference to section 3 seems to be a mistake, should it be subsection 4.2?
- Each of the justifications to get around the issue of distinguishing strange model behavior from bad feature importance technique should be explained briefly in paragraph before section 4.1.
Subsection 4.1:
- I do not understand the problem explained in fourth paragraph of section 4.1. g(f(1 - epsilon)) = 0, why would it be 1- epsilon?
- Problem explained in fifth Paragraph of section is not clear unless what the influence of the unit is clearly stated. Is it simply dg/df? 
- A short explanation of what is tested in section 6 should be given at last paragraph of section 4.1. Although the results are favorable to the conductance metric, it is not clear how they precisely confirm the problem of incorrect signs presented in the caricature examples.
Subsection 4.2:
- As said in the my main comments, I am not convinced by the use of the term Axiom. They are not use as building blocks, and are rather used as desirable properties for which the authors prove that only "path methods" can satisfy.
- Footnote 2 on page 5 it difficult to read.
- Although the proof does not seem to use the axioms as a building block, which is fortunate since it would make it a circular argument otherwise, the text suggests so: "Given these three axioms, we can show that:".
- The importance of section 4.2 should be clarified. More emphasis on the importance of the axioms (desirable properties) should be made.
Section 5:
- Choices for experiments should be explained. Why choosing layers mixed** rather than others? Why choosing filters?
- Figures 1-4 are difficult to interpreted on a printed version. Since this is qualitative, I suggest to change the saturation of the images to make them easier to interpret. The absolute values are not important for a qualitative interpretation
- Figure 4 could be more interesting if compared with other classes, like other animal faces. Anyhow, I understand that those were chosen based on the subset of classes used for the experiments.
- Space should be added between figures to better divide the captions
Section 6:
- The difference between experiments of Figure 5 and 6 should be made more clear.
Section 7-8:
- Where are they? No discussion? No conclusion?