LEARNING HOW TO EXPLAIN NEURAL NETWORKS: PATTERNNET AND PATTERNATTRIBUTION
DeConvNet, Guided BackProp, LRP, were invented to better understand deep neural networks. We show that these methods do not produce the theoretically correct explanation for a linear model. Yet they are used on multi-layer networks with millions of parameters. This is a cause for concern since linear models are simple neural networks. We argue that explanation methods for neural nets should work reliably in the limit of simplicity, the linear models. Based on our analysis of linear models we propose a generalization that yields two explanation techniques (PatternNet and PatternAttribution) that are theoretically sound for linear models and produce improved explanations for deep networks.
summary of article: 
This paper organizes existing methods for understanding and explaining deep neural networks into three categories based on what they reveal about a network: functions, signals, or attribution. “The function extracts the signal from the data by removing the distractor. The attribution of output values to input dimensions shows how much an individual component of the signal contributes to the output…” (p. 5). The authors propose a novel quality criterion for signal estimators, inspired by the analysis of linear models. They also propose two new explanatory methods, PatternNet (for signal estimation) and PatternAttribution (for relevance attribution), based on optimizing their new quality criterion. They present quantitative and qualitative analyses comparing PatternNet and PatternAttribution to several existing explanation methods on VGG-19.
* Quality: The claims of the paper are well supported by quantitative results and qualitative visualizations. 
* Clarity: Overall the paper is clear and well organized. There are a few points that could benefit from clarification.
* Originality: The paper puts forth an original framing of the problem of explaining deep neural networks. Related work is appropriately cited and compared. The authors's quality criterion for signal estimators allows them to do a quantitative analysis for a problem that is often hard to quantify.
* Significance: This paper justifies PatternNet and PatternAttribution as good methods to explain predictions made by neural networks. These methods may now serve as an important tool for future work which may lead to new insights about how neural networks work. 
Pros:
* Helps to organize existing methods for understanding neural networks in terms of the types of descriptions they provide: functions, signals or attribution.
* Creative quantitative analyses that evaluate their signal estimator at the level of single units and entire networks.
Cons:
* Experiments consider only the pre-trained VGG-19 model trained on ImageNet. Results may not generalize to other architectures/datasets.
* Limited visualizations are provided. 
Comments:
* Most of the paper is dedicated to explaining these signal estimators and quality criterion in case of a linear model. Only one paragraph is given to explain how they are used to estimate the signal at each layer in VGG-19. On first reading, there are some ambiguities about how the estimators scale up to deep networks. It would help to clarify if you included the expression for the two-component estimator and maybe your quality criterion for an arbitrary hidden unit. 
* The concept of signal is somewhat unclear. Is the signal 
    * (a) the part of the input image that led to a particular classification, as described in the introduction and suggested by the visualizations, in which case there is one signal per image for a given trained network?
    *  (b) the part of the input that led to activation of a particular unit, as your unit wise signal estimators are applied, in which case there is one signal for every unit of a trained network? You might benefit from two terms to separate the unit-level signal (what caused the activation of a particular unit?) from the total signal (what caused all activations in this network?).
* Assuming definition (b) I think the visualizations would be more convincing if you showed the signal for several output units. One would like to see that the signal estimation is doing more than separating foreground from background but is actually semantically specific. For instance, for the mailbox image, what does the signal look like if you propagate back from only the output unit for umbrella compared to the output unit for mailbox? 
* Do you have any intuition about why your two-component estimator doesn’t seem to be working as well in the convolutional layers? Do you think it is related to the fact that you are averaging within feature maps? Is it strictly necessary to do this averaging? Can you imagine a signal estimator more specifically designed for convolutional layers?
Minor issues: 
* The label "Figure 4" is missing. Only subcaptions (a) and (b) are present.
* Color scheme of figures: Why two oranges? It’s hard to see the difference.