Classifiers such as deep neural networks have been shown to be vulnerable against adversarial perturbations on problems with high-dimensional input space. While adversarial training improves the robustness of classifiers against such adversarial perturbations, it leaves classifiers sensitive to them on a non-negligible fraction of the inputs. We argue that there are two different kinds of adversarial perturbations: shared perturbations which fool a classifier on many inputs and singular perturbations which only fool the classifier on a small fraction of the data. We find that adversarial training increases the robustness of classifiers against shared perturbations. Moreover, it is particularly effective in removing universal perturbations, which can be seen as an extreme form of shared perturbations. Unfortunately, adversarial training does not consistently increase the robustness against singular perturbations on unseen inputs. However, we find that adversarial training decreases robustness of the remaining perturbations against image transformations such as changes to contrast and brightness or Gaussian blurring. It thus makes successful attacks on the classifier in the physical world less likely. Finally, we show that even singular perturbations can be easily detected and must thus exhibit generalizable patterns even though the perturbations are specific for certain inputs.
Summary:
This paper empirically studies adversarial perturbations dx and what the effects are of adversarial training (AT) with respect to shared (dx fools for many x) and singular (only for a single x) perturbations. Experiments use a (previously published) iterative fast-gradient-sign-method and use a Resnet on CIFAR.
The authors conclude that in this experimental setting:
- AT seems to defend models against shared dx's.
- This is visible on universal perturbations, which become less effective as more AT is applied.
- AT decreases the effectiveness of adversarial perturbations, e.g. AT decreases the number of adversarial perturbations that fool both an input x and x with e.g. a contrast change.
- Singular perturbations are easily detected by a detector model, as such perturbations don't change much when applying AT.
Pro:
- Paper addresses an important problem: qualitative / quantitative understanding of the behavior of adversarial perturbations is still lacking.
- The visualizations of universal perturbations as they change during AT are nice.
- The basic observation wrt the behavior of AT is clearly communicated.
Con:
- The experiments performed are interesting directions, although unfocused and rather limited in scope. For instance, does the same phenomenon happen for different datasets? Different models?
- What happens when we use adversarial attacks different from FGSM? Do we get similar results?
- The papers lacks a more in-depth theoretical analysis. Is there a principled reason AT+FGSM defends against universal perturbations?
Overall:
- As is, it seems to me the paper lacks a significant central message (due to limited and unfocused experiments) or significant new theoretical insight into the effect of AT. A number of questions addressed are interesting starting points towards a deeper understanding of *how* the observations can be explained and more rigorous empirical investigations.
Detailed:
-