Effective Path: Know the Unknowns of Neural Network
Despite their enormous success, there is still no solid understanding of deep neural network’s working mechanism. As such, researchers have demonstrated DNNs are vulnerable to small input perturbation, i.e., adversarial attacks. This work proposes the effective path as a new approach to exploring DNNs* internal organization. The effective path is an ensemble of synapses and neurons, which is reconstructed from a trained DNN using our activation-based backward algorithm. The per-image effective path can be aggregated to the class-level effective path, through which we observe that adversarial images activate effective path different from normal images. We propose an effective path similarity-based method to detect adversarial images and demonstrate its high accuracy and broad applicability.
 The authors propose the notion of effective path, for the purpose of identifying neurons that contributes to the predictions and being able to detect adversarial images in the context of image classification. Overall the paper is well written except that the authors are mixing two highly related but still different topics: explanation and adversary detection so that the motivation is confusing. The experimental results indeed show promises that effective path can help understand class similarities and network efficiencies but doesn’t really show how the proposed work is adding value to the field. It lacks the experimental comparison with previous methods but only include discussion in texts. This paper could turn out to be a stronger paper but it is not ready yet. Below are some more detailed comments. 1) The authors motivate by stating that the vulnerability of NN to input perturbations is due to the lack of interpretability (Section Introduction & Abstract). I can understand that we want more interpretability, and we want less vulnerability, but I can’t agree that vulnerability is caused by lack of interpretability. Also, the authors are trying to accomplish both tasks, interpretability and adversary detection, by showing data analysis of how the findings coincide with prior knowledge (eg. Class of digit 1 is the most different from other classes in MNIST task), and by showing detecting adversary images. However, neither has valid quantitative comparison with previous work; actually for the interpretability topic, the authors didn’t really provide a tool or a generalizable method. Thus, I would suggest to choose one of the two topics (ie. adversarial image detection) and focus on it by adding thorough comparison with other methods; in the discussion and result section, include the interpretability analysis to justify why the proposed adversary detection method is behaving in certain ways. 2) One topic that is missing from the paper is the time complexity of the proposed method. At a naïve estimate, it would require tracking and finding the minimum set of effective neurons with threshold .theta and thus per instance, at least O(m log m) is required at prediction phase, where m is the number of features; for n instances, the asymptotic complexity is O(nm log m) How does it compare to the other adversary detection methods? 3) Page 3 mentions that the work for critical routing path (Wang et al. 2018) requires re-training for every image; this statement is not really true without more context. Also authors discuss this work again very briefly in Page 8 due to the high similarity in methods and motivation with the proposed method, but the authors don’t show any quantitative comparison. After all, both methods are trying to identify neurons that contribute the most to the prediction, some more concrete comparison would be nice. 4) Page 3 mentions that the derived overall effective path is highly sparse compared to the original network and the effective path density for five trained models ranges from 13% to 42% which conforms with the “80%” claim from another paper. Together with the other similar statements, it would be really nice to note what .theta is used for such statements; how does such statement change with different .theta. Also some discussion would be nice about what such sparsity implies. Specifically, does the sparsity suggest the opportunity for feature selection, or does it suggest a way for detecting overfitting? 5) Page 5 shows the path similarity between the normal and the adversary examples; from the figure 5a and 5b, we can see the on the first layer, the mean deviate between normal and others but why the last layer they almost reach to the same point? It seems it is the middle layer that distinguish the normal from the adversary examples the most. Some more discussion would be good. 6) Some justification of why .theta=0.5 is chosen would be good on Page 6. 7) On Page 7, the authors are discussing the performance of the proposed method, however, there is no really comparison with other methods. But rather, the authors stated “better accuracy”, “AUC… is better…” by comparing different evaluation scenarios. I don’t find such discussion helpful in showing the contribution of the proposed method. Also in the parameter sensitivity, it would be nice to add the analysis for the effective path density and see if it still conforms with the “80%” claim with different .theta. 8) Page 1, need to add citations for the statement “… and even outperformed human beings.” 9) Minor issue: Page 1 “such computer vision…” should be “such as computer vision…”.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Effective Path: Know the Unknowns of Neural Network
Despite their enormous success, there is still no solid understanding of deep neural network’s working mechanism. As such, researchers have demonstrated DNNs are vulnerable to small input perturbation, i.e., adversarial attacks. This work proposes the effective path as a new approach to exploring DNNs* internal organization. The effective path is an ensemble of synapses and neurons, which is reconstructed from a trained DNN using our activation-based backward algorithm. The per-image effective path can be aggregated to the class-level effective path, through which we observe that adversarial images activate effective path different from normal images. We propose an effective path similarity-based method to detect adversarial images and demonstrate its high accuracy and broad applicability.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The authors propose the notion of effective path, for the purpose of identifying neurons that contributes to the predictions and being able to detect adversarial images in the context of image classification. Overall the paper is well written except that the authors are mixing two highly related but still different topics: explanation and adversary detection so that the motivation is confusing. The experimental results indeed show promises that effective path can help understand class similarities and network efficiencies but doesn’t really show how the proposed work is adding value to the field. It lacks the experimental comparison with previous methods but only include discussion in texts. This paper could turn out to be a stronger paper but it is not ready yet. Below are some more detailed comments. 1) The authors motivate by stating that the vulnerability of NN to input perturbations is due to the lack of interpretability (Section Introduction & Abstract). I can understand that we want more interpretability, and we want less vulnerability, but I can’t agree that vulnerability is caused by lack of interpretability. Also, the authors are trying to accomplish both tasks, interpretability and adversary detection, by showing data analysis of how the findings coincide with prior knowledge (eg. Class of digit 1 is the most different from other classes in MNIST task), and by showing detecting adversary images. However, neither has valid quantitative comparison with previous work; actually for the interpretability topic, the authors didn’t really provide a tool or a generalizable method. Thus, I would suggest to choose one of the two topics (ie. adversarial image detection) and focus on it by adding thorough comparison with other methods; in the discussion and result section, include the interpretability analysis to justify why the proposed adversary detection method is behaving in certain ways. 2) One topic that is missing from the paper is the time complexity of the proposed method. At a naïve estimate, it would require tracking and finding the minimum set of effective neurons with threshold .theta and thus per instance, at least O(m log m) is required at prediction phase, where m is the number of features; for n instances, the asymptotic complexity is O(nm log m) How does it compare to the other adversary detection methods? 3) Page 3 mentions that the work for critical routing path (Wang et al. 2018) requires re-training for every image; this statement is not really true without more context. Also authors discuss this work again very briefly in Page 8 due to the high similarity in methods and motivation with the proposed method, but the authors don’t show any quantitative comparison. After all, both methods are trying to identify neurons that contribute the most to the prediction, some more concrete comparison would be nice. 4) Page 3 mentions that the derived overall effective path is highly sparse compared to the original network and the effective path density for five trained models ranges from 13% to 42% which conforms with the “80%” claim from another paper. Together with the other similar statements, it would be really nice to note what .theta is used for such statements; how does such statement change with different .theta. Also some discussion would be nice about what such sparsity implies. Specifically, does the sparsity suggest the opportunity for feature selection, or does it suggest a way for detecting overfitting? 5) Page 5 shows the path similarity between the normal and the adversary examples; from the figure 5a and 5b, we can see the on the first layer, the mean deviate between normal and others but why the last layer they almost reach to the same point? It seems it is the middle layer that distinguish the normal from the adversary examples the most. Some more discussion would be good. 6) Some justification of why .theta=0.5 is chosen would be good on Page 6. 7) On Page 7, the authors are discussing the performance of the proposed method, however, there is no really comparison with other methods. But rather, the authors stated “better accuracy”, “AUC… is better…” by comparing different evaluation scenarios. I don’t find such discussion helpful in showing the contribution of the proposed method. Also in the parameter sensitivity, it would be nice to add the analysis for the effective path density and see if it still conforms with the “80%” claim with different .theta. 8) Page 1, need to add citations for the statement “… and even outperformed human beings.” 9) Minor issue: Page 1 “such computer vision…” should be “such as computer vision…”.
assistant
 The authors propose the notion of effective path, for the purpose of identifying neurons that contributes to the predictions and being able to detect adversarial images in the context of image classification. Overall the paper is well written except that the authors are mixing two highly related but still different topics: explanation and adversary detection so that the motivation is confusing. The experimental results indeed show promises that effective path can help understand class similarities and network efficiencies but doesn’t really show how the proposed work is adding value to the field. It lacks the experimental comparison with previous methods but only include discussion in texts. This paper could turn out to be a stronger paper but it is not ready yet. Below are some more detailed comments. 1) The authors motivate by stating that the vulnerability of NN to input perturbations is due to the lack of interpretability (Section Introduction & Abstract). I can understand that we want more interpretability, and we want less vulnerability, but I can’t agree that vulnerability is caused by lack of interpretability. Also, the authors are trying to accomplish both tasks, interpretability and adversary detection, by showing data analysis of how the findings coincide with prior knowledge (eg. Class of digit 1 is the most different from other classes in MNIST task), and by showing detecting adversary images. However, neither has valid quantitative comparison with previous work; actually for the interpretability topic, the authors didn’t really provide a tool or a generalizable method. Thus, I would suggest to choose one of the two topics (ie. adversarial image detection) and focus on it by adding thorough comparison with other methods; in the discussion and result section, include the interpretability analysis to justify why the proposed adversary detection method is behaving in certain ways. 2) One topic that is missing from the paper is the time complexity of the proposed method. At a naïve estimate, it would require tracking and finding the minimum set of effective neurons with threshold .theta and thus per instance, at least O(m log m) is required at prediction phase, where m is the number of features; for n instances, the asymptotic complexity is O(nm log m) How does it compare to the other adversary detection methods? 3) Page 3 mentions that the work for critical routing path (Wang et al. 2018) requires re-training for every image; this statement is not really true without more context. Also authors discuss this work again very briefly in Page 8 due to the high similarity in methods and motivation with the proposed method, but the authors don’t show any quantitative comparison. After all, both methods are trying to identify neurons that contribute the most to the prediction, some more concrete comparison would be nice. 4) Page 3 mentions that the derived overall effective path is highly sparse compared to the original network and the effective path density for five trained models ranges from 13% to 42% which conforms with the “80%” claim from another paper. Together with the other similar statements, it would be really nice to note what .theta is used for such statements; how does such statement change with different .theta. Also some discussion would be nice about what such sparsity implies. Specifically, does the sparsity suggest the opportunity for feature selection, or does it suggest a way for detecting overfitting? 5) Page 5 shows the path similarity between the normal and the adversary examples; from the figure 5a and 5b, we can see the on the first layer, the mean deviate between normal and others but why the last layer they almost reach to the same point? It seems it is the middle layer that distinguish the normal from the adversary examples the most. Some more discussion would be good. 6) Some justification of why .theta=0.5 is chosen would be good on Page 6. 7) On Page 7, the authors are discussing the performance of the proposed method, however, there is no really comparison with other methods. But rather, the authors stated “better accuracy”, “AUC… is better…” by comparing different evaluation scenarios. I don’t find such discussion helpful in showing the contribution of the proposed method. Also in the parameter sensitivity, it would be nice to add the analysis for the effective path density and see if it still conforms with the “80%” claim with different .theta. 8) Page 1, need to add citations for the statement “… and even outperformed human beings.” 9) Minor issue: Page 1 “such computer vision…” should be “such as computer vision…”.