Despite their enormous success, there is still no solid understanding of deep neural network's working mechanism. As such, researchers have demonstrated DNNs are vulnerable to small input perturbation, i.e., adversarial attacks. This work proposes the effective path as a new approach to exploring DNNs' internal organization. The effective path is an ensemble of synapses and neurons, which is reconstructed from a trained DNN using our activation-based backward algorithm. The per-image effective path can be aggregated to the class-level effective path, through which we observe that adversarial images activate effective path different from normal images. We propose an effective path similarity-based method to detect adversarial images and demonstrate its high accuracy and broad applicability.
This paper proposes a method for the detection of adversarial examples based on identification of critical paths (called "effective paths") in DNN classifiers. Borrowing from the analysis of execution paths of control-flow programs, the authors use back-propagation from the neuron associated from the final class decision to identify a minimal subset of input synapses accounting for more than a threshold proportion ("theta") of the total input weight. The identification process is then recursively applied at the preceding layer for those neurons associated with the selected minimal subset of synapses, forming a tree of synapses (the "effective path"). The authors then propose to compare the effective paths (actually, unions of paths) of different examples using simple structural dissimilarity measures, which they extend to allow comparison to a typical (aggregated) path for multiple examples drawn from a common class.
In their experimentation with their measure, they noted that examples generated by a number of adversarial attacks tend to be less similar to their first-ranked estimated class than normal examples are to their own first-ranked classes. Similarly, they note that these same adversarial attacks tend to be *more* similar to their second-ranked classes than normal examples are to their own second-ranked classes (as the authors point out, this is likely due to the increased likelihood of the second-ranked class of adversarial examples being the true class for the original example from which it was perturbed). The authors then propose the difference between these two similarities (that is, first-ranked dissimilarity minus second-ranked dissimilarity) as a characterization of adversarial examples.
The idea of using critical paths in the DNN to detect adversarial examples is interesting, and the authors deserve credit for showing that these critical paths (as defined in this paper) do show differences from those of normal examples. However, the originality of the approach is undercut by the recent work of Wang et al. (CVPR, 2018), which the authors acknowledge only in the discussion of experimental results. Although the details are different as to how critical paths are identified, and how adversarial examples can be detected using them, the strategies are definitely related - a more detailed explanation of this should have been given in the introduction of the paper. More troubling is the fact that a head-to-head experimental comparison is not provided, neither with Wang et al. nor with other state of the art detectors, other than a qualitative assessment of the capabilities of some detectors in Table 1. Note that even this qualitative discussion does not include some of the recent detection approaches, such as BPDA (Athalye et al., ICML 2018) or LID (Ma et al., ICLR 2018).
The question of how best to define critical paths and their similarities is still very much open - the authors' approach is rather simplistic and straightforward. For example, is their similarity measure biased towards the contributions from early layers? Can a layer-by-layer weighting of contributions improve the performance?
The authors do not always interpret their own experimental results correctly. For example, their results in Figures 7i and 7j don't really support their conclusion that performance "remains almost unchanged" when theta is in the range 0.5 - 1.0. Also, Figure 4 does not show that their effective path similarity is not *directly* "a great metric to distinguish between normal and adversarial" examples, because a large proportion of adversarial examples have scores that fall in the typical range for normal examples (however, there are differences in tendency which can be exploited, as the authors do show).
The organization of the paper is in some need of improvement. For example, the discussion of densities of "effective paths" (Section 2) comes well before the details of the choice of threshold value theta used to generate them (Section 4.1).
To summarize:
Pros:
* A good case is made for the use of critical paths as a way of differentiating adversarial examples from normal examples.
* The reported improvement in similarity of adversarial examples with respect to their second-ranked classes is particularly intriguing.
* The paper is generally well written and easy to follow.
Cons:
* The experimental treatment is insufficient; in particular, a more carefully considered experimental justification is needed with respect to other detection strategies.
* The question of how best to define critical paths and their similarities is still very much open.
* The authors do not always interpret their own experimental results correctly.
* The organization of the paper is in some need of improvement.