Robustness of classifiers: from adversarial to random noise
Several recent works have shown that state-of-the-art classifiers are vulnerable to worst-case (i.e., adversarial) perturbations of the datapoints. On the other hand, it has been empirically observed that these same classifiers are relatively robust to random noise. In this paper, we propose to study a semi-random noise regime that generalizes both the random and worst-case noise regimes. We propose the first quantitative analysis of the robustness of nonlinear classifiers in this general noise regime. We establish precise theoretical bounds on the robustness of classifiers in this general regime, which depend on the curvature of the classifier's decision boundary. Our bounds confirm and quantify the empirical observations that classifiers satisfying curvature constraints are robust to random noise. Moreover, we quantify the robustness of classifiers in terms of the subspace dimension in the semi-random noise regime, and show that our bounds remarkably interpolate between the worst-case and random noise regimes. We perform experiments and show that the derived bounds provide very accurate estimates when applied to various state-of-the-art deep neural networks and datasets. This result suggests bounds on the curvature of the classifiers' decision boundaries that we support experimentally, and more generally offers important insights onto the geometry of high dimensional classification problems.
This paper offers a thorough analysis of the effect of both worse-case (adversarial) and random noise in machine learning classifiers. It derives bounds that precisely describe the robustness of classifiers in function of the curvature of the decision boundary. This leads to some surprisingly (at least to me) general conclusions: * For random noise, the robustness of classifiers behaves as sqrt(d) times the distance from the datapoint to the classification boundary (where d denotes the dimension of the data) provided the curvature of the decision boundary is sufficiently small. This corroborates the intuition that random noise is less of an issue for high-dimensional data. On the other hand, how do we know the curvature of decision boundaries for general classifiers? * For semi-random noise, defined as the worst possible perturbation possible a random subspace of dimension m, the robustness behaves as sqrt(d/m) times the distance to the boundary. Hence, it is quite possible to find small perturbations that cause data misclassification, even within strong constraints.  These theoretical estimates are evaluated empirically on image recognition tasks using deep neural nets. Vice versa, it leads to some insight on the curvature of the decision boundary in deep neural nets.  Overall, this is very solid work that will likely appeal to the NIPS community. If I have one comments, it is that this work should be extended to many other types of algorithms to demonstrate its claimed generality.
This is very solid work and the paper is overall very pleasant to read. My one comment would be that this work seems to open up a lot of interesting future work when applied to other learners besides deep neural nets (and in some cases, we know something about the curvature of the decision boundaries (e.g. decision trees) that would make for interesting analysis. Are the authors interested in pursuing this?  Minor point: There is no reference in the paper to the proofs (in the supplementary materials). Make sure that they are easy to find in the final version.