Over the last few years, the phenomenon of adversarial examples -maliciously constructed inputs that fool trained machine learning models -has captured the attention of the research community, especially when the adversary is restricted to making small modifications of a correctly handled input. At the same time, less surprisingly, image classifiers lack human-level performance on randomly corrupted images, such as images with additive Gaussian noise. In this work, we show that these are two manifestations of the same underlying phenomenon. We establish this connection in several ways. First, we find that adversarial examples exist at the same distance scales we would expect from a linear model with the same performance on corrupted images. Next, we show that Gaussian data augmentation during training improves robustness to small adversarial perturbations and that adversarial training improves robustness to several types of image corruptions. Finally, we present a model-independent upper bound on the distance from a corrupted image to its nearest error given test performance and show that in practice we already come close to achieving the bound, so that improving robustness further for the corrupted image distribution requires significantly reducing test error. All of this suggests that improving adversarial robustness should go hand in hand with improving performance in the presence of more general and realistic image corruptions. This yields a computationally tractable evaluation metric for defenses to consider: test error in noisy image distributions.
This paper propose an alternative view for adversarial examples in high dimension spaces by considering the "error rate" in a Gaussian distribution centered at each test point. However, as mentioned in the related work, adversarial examples through the lens of isoperimetric inequality is not new to this paper; the implication of adversarial sensitivity by error rate in the test-sample-centered-Gaussian in general non-linear case is rather weak; and the empirical results does not show advantage over simple adversarial training against lp constrained adversarial attacks.
Here are some more detailed comments and feedbacks:
The clarity could be improved by making clear use of notations and define some key terms explicitly:
1. For example, the error rate sometimes refer to the test error, sometimes refer to the error rate under a special distribution centered at a particular test example.
2. Similarly, the distribution q sometimes refer to the original (unknown) input data distribution, but the same notation is also used to refer to this Gaussian distribution centered on each test example. Although the paper says that "q need not be restricted to the distribution from which the training set was sampled", it could potentially confuse the reader less if there is a symbol for the "usual" test error and a different one for this test-error-under-Gaussian-centered-at-a-particular-test-example.
3. It would be good if the paper could make a formal definition of the problem being studied and explicitly specify the assumptions on the existence of a deterministic target function (concept) and explicitly define the error set E.
4. It seems to assume the input distribution is continuous everywhere but not stated. If for example, the original data is supported on disconnected manifolds separated with low density or even zero-density margins, then the Gaussian distribution centered on test example argument will need to be modified to talk about the intersection of the Gaussian with the data manifold instead. If the paper does decide to make this kind of assumption, some empirical study on the data to verify the fidelity of the assumptions would be great.
5. The paper does not mention how measurement against the error set E. Under the original data distribution, it is natural to measure the error rate with the provided training or test data with labels. However, under each newly formed Gaussian distribution centered at each test point, the labels for the newly sampled examples from this Gaussian is unknown, and since there is no "ground truth classifier" for MNIST or CIFAR10 available, it seems impossible to "calculate" the true label for those samples, which are needed to calculate the error rate. It is not very clear from the paper how this issue is solved. I'm guess it uses the label from the Gaussian center for all the samples from the Gaussian. While this might be reasonable assumption for Gaussian with tiny variances, it is less clear how reasonable it is for the large variance Gaussian distributions considered in the paper. If this assumption is made, please state it explicitly and empirically or theoretically study how reasonable this assumption is in the regime of variances considered in this paper. If my guessing is wrong, please also explicitly what approach is used to get around this issue.
The followings are some feedbacks on the contents and ideas of the paper:
6. I think one sentence in the text summarize a large part of the paper very well: "to measure adversarial robustness is to ask whether or not there are any errors in the linf ball, ... and to measure test error in noise is to measure the volume of the error set in the defined noise distribution". However, this looks like a rather roundabout approach to attack another problem (measuring volume of an unknown set in a very high dimension space) in order to solve the original problem, while the implication is rather weak (a precise implication can only be obtained for linear separating hyper-plane, while for non-linear classifiers it is much less clear).
Note while exact adversarial robustness is NP-hard, volume estimation in high dimension is not easy (if not harder). For cifar-10, the inputs are in dimension 32x32x3 = 3072, the 1,000 samples used in the paper to estimate the volume of a set in this high dimension seem to be quite inaccurate. I would appreciate if variances could be reported in those studies to show the confidence of the estimations. For imagenets, the inputs are in even larger spaces.
Given the difficulty (in terms of sample complexity) to estimate the "error-in-noise", it might not be very surprising that the noise augmentation does not show advantages to lp constrained adversarial attacks (comparing to adversarial training).
7. In the conclusion, the paper states "we proved a fundamental relationship between generalization in noisy image distributions and the existence of small adversarial perturbations". I believe a formal proof is only given to the case of existence of small adversarial perturbations to examples to the "noisy examples" from the Gaussian distribution, and in this case, it is a rather direct corollary from the Gaussian Isoperimetric Inequality. For the more "practical case" (in the sense that is more related to the usual notion of adversarial examples) of existence of small adversarial examples, it seems only the case of the linear classifier is formally discussed.
Moreover, I'm a little bit worried some important pieces might be lost and create potentially misleading or seemly strong conclusion. Maybe it would be helpful if a concrete example could be given in the paper that shows the full path from the error-in-noise to existence of adversarial example, by showing all the constants involved. I'm a bit confused here because (in order for the isoperimetric inequality to have favorable bounds?) the Gaussian distributions used in error-in-noise seem to have rather large variance. As mentioned in the paper, the majority of the mass in the Gaussian distribution considered will be in a thin sphere of radius sigma * sqrt(n) centered at the test example x. If sigma = 0.1 and dimension n = 3072 (cifar-10), then the radius is around 5.5 (in l2 distance) which is probably quite far from the test point x (is it?). It is then less clear how "a large majority of this thin sphere far away from x is epsilon close to the error set" could tightly imply properties of adversarial robustness of x itself in its close vicinity. Maybe a specific example with all the numerical constants spelled out would help illustrate this.
In summary, I think this paper takes an interesting but roundabout perspective to adversarial robustness, and the implication is weak in the non-linear case. (Potentially because of the weak implication), the suggested approach for defenses by augmenting with noises does not show advantage over adversarial training.