Almost all current adversarial attacks of CNN classifiers rely on information derived from the output layer of the network. This work presents a new adversarial attack based on the modeling and exploitation of class-wise and layer-wise deep feature distributions. We achieve state-of-the-art targeted blackbox transfer-based attack results for undefended ImageNet models. Further, we place a priority on explainability and interpretability of the attacking process. Our methodology affords an analysis of how adversarial attacks change the intermediate feature distributions of CNNs, as well as a measure of layer-wise and class-wise feature distributional separability/entanglement. We also conceptualize a transition from task/data-specific to model-specific features within a CNN architecture that directly impacts the transferability of adversarial examples.
The paper proposes a new adversarial attack for the targeted blackbox model Unlike previous approaches which use the output layer possibly with some additional terms and regularization, the proposed approaches only rely on intermediate features. In fact, the adversarial example is based on a single intermediate layer. The adversarial examples are built by training, for each target class, a binary classifier for the class based only on the features of that layer.
- There are multiple similar prior works that use intermediate layers in some way, and the paper does a good job to explain the differences between the proposed approach and these.
The paper proposes three variants. The simplest one appears rather weak in numerical experiments. The two other methods seem to achieve a different tradeoff, between focusing on the general error rate or on the targeted success rate.
In particular, FDA+fd is similar in some ways to Zhou et al. (2018) which uses an additional loss term to maximize the distance of the features at various layers. A comparison with this method (which uses the output layer) would be interesting, and may constitute a relevant additional baseline, at least for non-targeted metrics like error (which is where FDA+ms shines).
Similarly, a comparison to AA is provided in Figure 2 in the case of 10 classes, but not in Table 1 for the 1000 classes experiment. This casts some doubt on the significance of the result in the 1000 classes setting. A major issue with the proposed method is that training binary classifiers in large multi-class settings is quite costly, which is bound to hinder one's ability to identify 'high performing attack settings' In particular, the paper mentions that the 10 classes are used for that, but no evidence is provided that these settings are actually good for all classes.
The conclusions are rather consistent accross different pairs of source/target network.
The paper does not mention adversarial training. As this has become a common defence practice in the literature, it would be interesting to understand to what extent the proposed methods are thwarted by adversarial training. This could increase the significance of the paper if working with intermediate feature representation makes the attack more robust to this type of defense.
In summary, I find the experiments satisfying, although they can definitely be made more convincing by adding additional strong baselines and demonstrating how much performance can really be attained in the 1000 classes scenario.
- Generally, not using the output layer at all is an interesting approach that deserves in my opinion to be discussed and investigated further.
- The paper is clearly written, and notation is rigorous.