The trade-off between robustness and standard accuracy has been consistently reported in the machine learning literature. Although the problem has been widely studied to understand and explain this trade-off, the problem seems to remain as an open problem. In this paper, motivated by the fact that the high dimensional distribution is poorly represented by limited data samples, we introduce sensible adversarial learning and demonstrate the synergistic effect between pursuits of natural accuracy and robustness. Specifically, we define a sensible adversary which is useful for learning a defense model and keeping a high natural accuracy simultaneously. We theoretically establish that the Bayes rule is the most robust multi-class classifier with the 0-1 loss under sensible adversarial learning. We propose a novel and efficient algorithm that trains a robust model with sensible adversarial examples, without a significant drop in natural accuracy. Our model on CIFAR10 yields state-of-the-art results against various attacks with perturbations restricted to ∞ with = 8/255, e.g., the robust accuracy 57.23% against PGD attacks as well as the natural accuracy 91.51%.
Summary:
The paper studies the phenomenon of trade-off between robust and standard accuracies that is usually observed in adversarial training. Many existing studies try to understand this trade-off and show that it is unavoidable. In contrast, this work shows that under a sensible definition of adversarial risk, there is no trade-off between standard accuracy and sensible adversarial accuracy. It is shown that Bayes optimal classifier has optimal standard and sensible adversarial accuracies. The authors then go on to propose a new adversarial training algorithm which tries to minimize the sensible adversarial risk. Experimental results show that models learned through the proposed technique have high adversarial and standard accuracies. 
Comments:
1) Sensible Adversarial Risk: The first contribution of the work is to define a new notion of adversarial risk which the authors call ''sensible adversarial risk'' and study its properties.  There is a recent work[1] which also proposes a new definition of adversarial risk and which is similar in spirit to what the current work tries to achieve.  In [1], the authors define a perturbation as adversarial only if it doesn't change the label of the Bayes optimal classifier, which is similar to what the current paper does. Owing to this similarity, the properties of sensible adversarial risk obtained in Theorems 1,2 in the current work look similar to [1]. So the authors should discuss/compare their results with [1].
2) Sensible Adversarial Training: I believe the major contribution of the paper is to propose an algorithm for minimizing sensible adversarial risk. However, I have some concerns with the proposed algorithm. The authors say that since the Bayes optimal classifier is unknown, they use a reference model f_r (which can be naturally trained). Consider the following scenario. Suppose the true data is given by (x, f^B(x)), for some unknown f^B; that is, the Bayes optimal classifier has perfect standard accuracy. Suppose f_r has perfect standard accuracy, but very bad adversarial accuracy on train and test sets. Suppose f_r is substituted for f^B in the sensible adversarial risk (with 0-1 loss). Then it is easy to see that f_r is a minimizer of the resulting objective. So the proposed algorithm will just output f_r in this scenario. This is clearly not desirable. Given this, I believe a more thorough understanding of the proposed algorithm is needed. When will the algorithm converge to non-robust classifiers? How should one initialize the algorithm to avoid such undesirable behavior?
While the notion of sensible adversarial risk is sensible, it is not clear why it should result in such high adversarial accuracies as reported in Tables 1,5. The adversarial perturbation of epsilon=8/255 on cifar10 is considered so small that sensible adversarial risk (with reference model f^B) at any point will almost always be equal to the existing notion of adversarial risk at that point. So, minimizing sensible adversarial risk (assuming you are given f^B) is exactly equivalent to minimizing adversarial risk. But it is known that minimizing adversarial risk results in models with low standard accuracies. So I feel sensible adversarial risk is not the reason behind such high accuracies. Could the authors explain what is the reason for such good adversarial accuracies reported in the paper?
3) Experiments:  While the experimental results on cifar10 look impressive, I have some concerns about the way the PGD attacks are run. I downloaded the model provided by the authors and ran PGD attack on it.  I ran L_infty attacks with epsilon=8/255, step size=2/255. The results I obtained seem to differ from the results presented in the paper:
PGD Steps | Adversarial accuracy of SENSE
20            62.09
50            60.34
100           59.99
I believe PGD attack with multiple random restarts will reduce the adversarial accuracy even further. Given this, I'd appreciate if the authors perform more careful attacks (with appropriate hyper-parameters) on their model. It'd also be great if the authors report the performance of PGD trained model using the same attacks used to report the performance on their model. 
4) Other comments:  I'm not sure if the toy example (cheese hole distribution) in Section 2 is helpful. What the authors seem to conclude from it is that adversarial training can improve standard accuracy. But I do not agree with these conclusions. What if Figure 1b is the true distribution? Will the same conclusions hold? In general, these toy examples need not be illustrative of the behavior on real datasets. So instead of having these toy examples, I'd suggest the authors have a thorough discussion on theoretical and experimental results.   
[1] Suggala, A. S., Prasad, A., Nagarajan, V., & Ravikumar, P. (2018). Revisiting Adversarial Risk. arXiv preprint arXiv:1806.02924.