ON THE SENSITIVITY OF ADVERSARIAL ROBUSTNESS TO INPUT DATA DISTRIBUTIONS
Neural networks are vulnerable to small adversarial perturbations. Existing literature largely focused on understanding and mitigating the vulnerability of learned models. In this paper, we demonstrate an intriguing phenomenon about the most popular robust training method in the literature, adversarial training: Adversarial robustness, unlike clean accuracy, is sensitive to the input data distribution. Even a semantics-preserving transformations on the input data distribution can cause a significantly different robustness for the adversarial trained model that is both trained and evaluated on the new distribution. Our discovery of such sensitivity on data distribution is based on a study which disentangles the behaviors of clean accuracy and robust accuracy of the Bayes classifier. Empirical investigations further confirm our finding. We construct semantically-identical variants for MNIST and CIFAR10 respectively, and show that standardly trained models achieve comparable clean accuracies on them, but adversarially trained models achieve significantly different robustness accuracies. This counter-intuitive phenomenon indicates that input data distribution alone can affect the adversarial robustness of trained neural networks, not necessarily the tasks themselves. Lastly, we discuss the practical implications on evaluating adversarial robustness, and make initial attempts to understand this complex phenomenon.
The paper is interesting and topical: robustness to adversarial input presentation (or shifts in training data itself, even those of the nature described by the authors 'semantic-lossless' shifts). Adversarial inputs are investigated under l-inf bounded perturbations, while multiclass classification on images is the target problem considered. The theoretical parts of the paper, assigning lack of adversarial robustness to the shape of the input distribution (Section 2) is the strongest part of the paper, adding some simple and important insights. Unfortunately, the empirical part of the paper is weakened by an over-reliance of (custom perturbations of ) the popular MNIST and CIFAR10 datasets (which are themselves based on larger sets). Furthermore, the basic conclusion as to causes and remedies of lack of robustness is not evident, and it is not evident that it has been sufficiently investigated. Shape yes, differences in perturbable volume not (how does that concur with Section 2?), and inter-class distance also not. Are we to base these conclusions on 2 perturbed datasets? How are readers to synthesize the final conclusion that robustness is a 'complex interaction of tasks and data', other than what they would already expect? In short, a valiant effort, and a good direction, but one that needs more work.