With Friends Like These, Who Needs Adversaries?
The vulnerability of deep image classification networks to adversarial attack is now well known, but less well understood. Via a novel experimental analysis, we illustrate some facts about deep convolutional networks for image classification that shed new light on their behaviour and how it connects to the problem of adversaries. In short, the celebrated performance of these networks and their vulnerability to adversarial attack are simply two sides of the same coin: the input image-space directions along which the networks are most vulnerable to attack are the same directions which they use to achieve their classification performance in the first place. We develop this result in two main steps. The first uncovers the fact that classes tend to be associated with specific image-space directions. This is shown by an examination of the class-score outputs of nets as functions of 1D movements along these directions. This provides a novel perspective on the existence of universal adversarial perturbations. The second is a clear demonstration of the tight coupling between classification performance and vulnerability to adversarial attack within the spaces spanned by these directions. Thus, our analysis resolves the apparent contradiction between accuracy and vulnerability. It provides a new perspective on much of the prior art and reveals profound implications for efforts to construct neural nets that are both accurate and robust to adversarial attack.
The reason behind adversarial vulnerability of high dimensional classifier is still an open question. There are a few works trying to address this question by providing various and sometimes conflicting hypotheses. This work attempts to address the problem from a geometric perspective. The main hypothesis is that the “adversarial” directions are the ones which the classifier uses to distinguish between different classes. They provide some empirical evidence to support this idea.  The paper raises some interesting points regarding adversarial robustness. It shows that the subspace spanned by the “curved” directions of the classifier in the image space captures the most relevant features for the classification task. Hence, projecting images on this subspace does not degrade the classification performance, while by projecting on the complement (flat subspace), the classifier accuracy drops significantly (Figure 4). Then, the authors argue that various adversarial perturbations are captured in the subspace of curved directions. Therefore, they conclude that adversarial directions are nothing but the discriminative features of different classes. This is indeed what happens in linear classifiers.  I have to admit that I have not understood completely all the experiments. Honestly, I had hard time to follow some of the arguments as they were not rigorous enough. I feel that most of the arguments could be conveyed more clearly by equipping them with suitable math equations. Though this work has a lot of potential, it lacks the sufficient clarity and quality. For example,  - The quality of plots are quite low, axes are missing, some numbers are even not rendered correctly. E.g., the numbers of the y-axes for CIFAR10 networks are missing. - Many definitions are missing and it is basically relied on the reader to guess them, e.g. N_f, N_p, N_n, etc. - Where does QR decomposition come from? I can guess but please be explicit. - What is Q_d? - What are the axes in Figure 1? x-axis? y-axis? To what does each image correspond? - How Alg 1 differs from that in [3]?  why? - How do you compute the subspaces for ImageNet?  One of the main drawbacks of the paper is that half of it is dedicated to prior works while it skips many important details regarding the contributions. Also, it is sometimes difficult to recognize their contribution from prior works.  Though I really like the approach they take to study the adversarial vulnerability of deep classifiers, I guess the paper can heavily benefit from a complete rewriting. The most important change could be to shorten the prior works and to elaborate the contributions by taking the time to explain things using math equations.  Anyway, the paper ends with the following sentence: “…nets’ favourite features are their worst adversaries”. This would be a great discovery if the authors could take a bit more time to polish their work.