Effective Path: Know the Unknowns of Neural Network
Despite their enormous success, there is still no solid understanding of deep neural network’s working mechanism. As such, researchers have demonstrated DNNs are vulnerable to small input perturbation, i.e., adversarial attacks. This work proposes the effective path as a new approach to exploring DNNs* internal organization. The effective path is an ensemble of synapses and neurons, which is reconstructed from a trained DNN using our activation-based backward algorithm. The per-image effective path can be aggregated to the class-level effective path, through which we observe that adversarial images activate effective path different from normal images. We propose an effective path similarity-based method to detect adversarial images and demonstrate its high accuracy and broad applicability.
 This paper proposes a measure (“effective path”) of which units and weights were most important for classification of a particular input or input class. Using the effective path, the authors analyze the overlap between paths across classes for CNNs and between adversarially modified and unmodified images. Finally, the paper proposes an adversarial defense method based on effective path which detects adversarially manipulated images with high accuracy and generality to a variety of settings. Overall, this paper is interesting and provides several novel observations. The clarity of the exposition is generally good, but can be improved in several places (mentioned below). As for significance, effective path is likely to inform future analyses of neural networks, and the adversarial defense may prove impactful, though ultimately, its impact will depend on if and when the defense is broken. However, there are several important controls missing from the analysis, several claims which are unsubstantiated, and experimental details are lacking in a few places. As such, in its current form, I can only weakly recommend this paper for acceptance. If in the revision the controls requested below are included, additional evidence is provided for the unsubstantiated claims (or if those claims are toned down), and exposition of missing experimental details is included, I’d be happy to raise my score. Major points: 1) While the observation regarding path specialization is very interesting, one cannot gauge whether or not the degree of overlap observed between class-specific paths signals path specialization or simply high input-to-input path variance (which is similar both within and across classes). In order to distinguish between these possibilities, a measure of intra-class path similarity is necessary. In addition, an experiment similar to that in Figure 2 with CIFAR-10 would be quite helpful in evaluating whether this phenomenon exists in more natural datasets (the ImageNet results are difficult to interpret due to the large number of classes). 2) Several claims in the path specialization section are unsubstantiated. 2a) In particular, the claim that ‘1’ has the highest degree of specialization “because of its unique shape” is made without evidence as is the similarity between ‘5’ and ‘8’. ‘6’ is also similar to ‘8’ and yet does not show the same similarity in the path specialization. These differences may very well simply be due to chance. 2b) The claim that the path specialization in ImageNet matches the class hierarchy is made only based on the rough non-linearity of Figure 3. Please either measure the overlap within and across class categories or soften this claim. 3) The similarity analysis for adversarial images is also very interesting, but a comparison between unmodified and randomly perturbed images with matched norms to the adversarially perturbed images is necessary to establish whether this effect is due to noise generally or adversarial noise. It’s unclear how the effective path is calculated when negative weights are involved. Further exposition of this aspect would be helpful. Minor points/typos: 1) There are several places where confusing concepts are introduced in one paragraph but explained several paragraphs later. In particular, the distinction between synapses and weights is introduced halfway through page 2 but explained on page 3 and the fact that the coefficients for the defense metric are learned is unclear until page 4 even though they’re introduced on page 3. 2) Typos: 2a) Section 1, fourth paragraph: “...and adversarial images, we uncover...” should be “...and adversarial images, and we uncover...” 2b) Section 1, fourth paragraph: “...by small perturbation, the network…” should be “...by small perturbations, the network…” 2c) Section 2, first paragraph: “...the black-boxed neural…” should be “...the black-box neural…” 2d) Section 2, first paragraph: “In the high level…” should be “At a high level…” 2e) Section 4, first paragraph: “...as it does no modify…” should be “...as it does not modify…” 2f) Title, should be "Neural Network"?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Effective Path: Know the Unknowns of Neural Network
Despite their enormous success, there is still no solid understanding of deep neural network’s working mechanism. As such, researchers have demonstrated DNNs are vulnerable to small input perturbation, i.e., adversarial attacks. This work proposes the effective path as a new approach to exploring DNNs* internal organization. The effective path is an ensemble of synapses and neurons, which is reconstructed from a trained DNN using our activation-based backward algorithm. The per-image effective path can be aggregated to the class-level effective path, through which we observe that adversarial images activate effective path different from normal images. We propose an effective path similarity-based method to detect adversarial images and demonstrate its high accuracy and broad applicability.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposes a measure (“effective path”) of which units and weights were most important for classification of a particular input or input class. Using the effective path, the authors analyze the overlap between paths across classes for CNNs and between adversarially modified and unmodified images. Finally, the paper proposes an adversarial defense method based on effective path which detects adversarially manipulated images with high accuracy and generality to a variety of settings. Overall, this paper is interesting and provides several novel observations. The clarity of the exposition is generally good, but can be improved in several places (mentioned below). As for significance, effective path is likely to inform future analyses of neural networks, and the adversarial defense may prove impactful, though ultimately, its impact will depend on if and when the defense is broken. However, there are several important controls missing from the analysis, several claims which are unsubstantiated, and experimental details are lacking in a few places. As such, in its current form, I can only weakly recommend this paper for acceptance. If in the revision the controls requested below are included, additional evidence is provided for the unsubstantiated claims (or if those claims are toned down), and exposition of missing experimental details is included, I’d be happy to raise my score. Major points: 1) While the observation regarding path specialization is very interesting, one cannot gauge whether or not the degree of overlap observed between class-specific paths signals path specialization or simply high input-to-input path variance (which is similar both within and across classes). In order to distinguish between these possibilities, a measure of intra-class path similarity is necessary. In addition, an experiment similar to that in Figure 2 with CIFAR-10 would be quite helpful in evaluating whether this phenomenon exists in more natural datasets (the ImageNet results are difficult to interpret due to the large number of classes). 2) Several claims in the path specialization section are unsubstantiated. 2a) In particular, the claim that ‘1’ has the highest degree of specialization “because of its unique shape” is made without evidence as is the similarity between ‘5’ and ‘8’. ‘6’ is also similar to ‘8’ and yet does not show the same similarity in the path specialization. These differences may very well simply be due to chance. 2b) The claim that the path specialization in ImageNet matches the class hierarchy is made only based on the rough non-linearity of Figure 3. Please either measure the overlap within and across class categories or soften this claim. 3) The similarity analysis for adversarial images is also very interesting, but a comparison between unmodified and randomly perturbed images with matched norms to the adversarially perturbed images is necessary to establish whether this effect is due to noise generally or adversarial noise. It’s unclear how the effective path is calculated when negative weights are involved. Further exposition of this aspect would be helpful. Minor points/typos: 1) There are several places where confusing concepts are introduced in one paragraph but explained several paragraphs later. In particular, the distinction between synapses and weights is introduced halfway through page 2 but explained on page 3 and the fact that the coefficients for the defense metric are learned is unclear until page 4 even though they’re introduced on page 3. 2) Typos: 2a) Section 1, fourth paragraph: “...and adversarial images, we uncover...” should be “...and adversarial images, and we uncover...” 2b) Section 1, fourth paragraph: “...by small perturbation, the network…” should be “...by small perturbations, the network…” 2c) Section 2, first paragraph: “...the black-boxed neural…” should be “...the black-box neural…” 2d) Section 2, first paragraph: “In the high level…” should be “At a high level…” 2e) Section 4, first paragraph: “...as it does no modify…” should be “...as it does not modify…” 2f) Title, should be "Neural Network"?
assistant
 This paper proposes a measure (“effective path”) of which units and weights were most important for classification of a particular input or input class. Using the effective path, the authors analyze the overlap between paths across classes for CNNs and between adversarially modified and unmodified images. Finally, the paper proposes an adversarial defense method based on effective path which detects adversarially manipulated images with high accuracy and generality to a variety of settings. Overall, this paper is interesting and provides several novel observations. The clarity of the exposition is generally good, but can be improved in several places (mentioned below). As for significance, effective path is likely to inform future analyses of neural networks, and the adversarial defense may prove impactful, though ultimately, its impact will depend on if and when the defense is broken. However, there are several important controls missing from the analysis, several claims which are unsubstantiated, and experimental details are lacking in a few places. As such, in its current form, I can only weakly recommend this paper for acceptance. If in the revision the controls requested below are included, additional evidence is provided for the unsubstantiated claims (or if those claims are toned down), and exposition of missing experimental details is included, I’d be happy to raise my score. Major points: 1) While the observation regarding path specialization is very interesting, one cannot gauge whether or not the degree of overlap observed between class-specific paths signals path specialization or simply high input-to-input path variance (which is similar both within and across classes). In order to distinguish between these possibilities, a measure of intra-class path similarity is necessary. In addition, an experiment similar to that in Figure 2 with CIFAR-10 would be quite helpful in evaluating whether this phenomenon exists in more natural datasets (the ImageNet results are difficult to interpret due to the large number of classes). 2) Several claims in the path specialization section are unsubstantiated. 2a) In particular, the claim that ‘1’ has the highest degree of specialization “because of its unique shape” is made without evidence as is the similarity between ‘5’ and ‘8’. ‘6’ is also similar to ‘8’ and yet does not show the same similarity in the path specialization. These differences may very well simply be due to chance. 2b) The claim that the path specialization in ImageNet matches the class hierarchy is made only based on the rough non-linearity of Figure 3. Please either measure the overlap within and across class categories or soften this claim. 3) The similarity analysis for adversarial images is also very interesting, but a comparison between unmodified and randomly perturbed images with matched norms to the adversarially perturbed images is necessary to establish whether this effect is due to noise generally or adversarial noise. It’s unclear how the effective path is calculated when negative weights are involved. Further exposition of this aspect would be helpful. Minor points/typos: 1) There are several places where confusing concepts are introduced in one paragraph but explained several paragraphs later. In particular, the distinction between synapses and weights is introduced halfway through page 2 but explained on page 3 and the fact that the coefficients for the defense metric are learned is unclear until page 4 even though they’re introduced on page 3. 2) Typos: 2a) Section 1, fourth paragraph: “...and adversarial images, we uncover...” should be “...and adversarial images, and we uncover...” 2b) Section 1, fourth paragraph: “...by small perturbation, the network…” should be “...by small perturbations, the network…” 2c) Section 2, first paragraph: “...the black-boxed neural…” should be “...the black-box neural…” 2d) Section 2, first paragraph: “In the high level…” should be “At a high level…” 2e) Section 4, first paragraph: “...as it does no modify…” should be “...as it does not modify…” 2f) Title, should be "Neural Network"?