ROBUSTNESS MAY BE AT ODDS WITH ACCURACY
We show that there exists an inherent tension between the goal of adversarial robustness and that of standard generalization. Specifically, training robust models may not only be more resource-consuming, but also lead to a reduction of standard accuracy. We demonstrate that this trade-off between the standard accuracy of a model and its robustness to adversarial perturbations provably exists even in a fairly simple and natural setting. These findings also corroborate a similar phenomenon observed in practice. Further, we argue that this phenomenon is a consequence of robust classifiers learning fundamentally different feature representations than standard classifiers. These differences, in particular, seem to result in unexpected benefits: the features learned by robust models tend to align better with salient data characteristics and human perception.
This paper presents a study of tradeoffs between adversarial and standard accuracy of classifiers. Though it might be expected that training for adversarial robustness always leads to improvement in standard accuracy, however the authors claim that the actual situation is quite subtle. Though adversarial training might help towards increasing standard accuracy in certain data regimes such as data scarcity, but when sufficient data is available there exists a trade-off between the two goals. The tradeoff is demonstrated in a fairly simple setting in which case data consists of two kinds of features - those which are weakly correlated with the output, and those which are strongly correlated. It is shown that adversarial accuracy depends on the feature which exhibit strong correlation, while standard accuracy depends on weakly correlated features.
Though the paper presents some interesting insights. Some of the concerns  are :
 - The paper falls short in answering the tradeoff question under a more general setup. The toy example is very specific with a clear separation between weak and strongly correlated features. It would be interesting to see how similar results can be derived when under more complicated setup with many features with varying extent of correlation.
 - The tradeoff between standard and robustness under linear classification has also been demonstrated in a recent work [1]. In [1], it is also argued that for datasets consisting of large number of labels, when some of the labels are under data-scarce regimes, an adversarial robustness view-point (via l1-regularization) helps in accuracy improvement for those labels. However, for other set of labels for which there is sufficient data available,  l2-regularization is more suited, and adversarial robustness perspective decreases standard accuracy. From this view-point, one could argue that some of the main contributions in the current paper, could be seen as empirical extensions for deep learning setup. It would be instructive to contrast and explore connections between this paper, and the observations in [1].
[1] Adversarial Extreme Multi-label Classification, https://arxiv.org/abs/1803.01570
==============post-rebuttal======
thanks for the feedback, I update my rating of the paper