Published as a conference paper at ICLR 2018 TOWARDS DEEP LEARNING MODELS RESISTANT TO ADVERSARIAL ATTACKS
Recent work has demonstrated that neural networks are vulnerable to adversarial examples, i.e., inputs that are almost indistinguishable from natural data and yet classified incorrectly by the network. To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization. This approach provides us with a broad and unifying view on much prior work on this topic. Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal. In particular, they specify a concrete security guarantee that would protect against a well-defined class of adversaries. These methods let us train networks with significantly improved resistance to a wide range of adversarial attacks. They also suggest robustness against a first-order adversary as a natural security guarantee. We believe that robustness against such well-defined classes of adversaries is an important stepping stone towards fully resistant deep learning models.
This paper proposes to look at making neural networks resistant to adversarial loss through the framework of saddle-point problems. They show that, on MNIST, a PGD adversary fits this framework and allows the authors to train very robust models. They also show encouraging results for robust CIFAR-10 models, but with still much room for improvement. Finally, they suggest that PGD is an optimal first order adversary, and leads to optimal robustness against any first order attack.
This paper is well written, brings new ideas and perfoms interesting experiments, but its claims are somewhat bothering me, considering that e.g. your CIFAR-10 results are somewhat underwhelming. All you've really proven is that PGD on MNIST seems to be the ultimate adversary. You contrast this to the fact that the optimization is non-convex, but we know for a fact that MNIST is fairly simple in that regime; iirc a linear classifier gets something like 91% accuracy on MNIST. So my guess is that the optimization problem on MNIST is in fact pretty convex and mostly respects the assumptions of Danskin's theorem, but not so much for CIFAR-10 (maybe even less so for e.g. ImageNet, which is what Kurakin et al. seem to find).
Considering your CIFAR-10 results, I don't think anyone should "suggest that secure neural networks are within reach", because 1) there is still room for improvement 2) it's a safe bet that someone will always just come up with a better attack than whatever defense we have now. It has been this way in many disciplines (crypto, security) for centuries, I don't see why deep learning should be exempt. Simply saying "we believe that our robust models are significant progress on the defense side" was enough, because afaik you did improve on CIFAR-10's SOTA; don't overclaim. 
You make these kinds of claims in a few other places in this paper, please be careful with that.
The contributions in your appendix are interesting. 
Appendix A somewhat confirms one of the postulates in Goodfellow et al. (2014): "The direction of perturbation, rather than the specific point in space, matters most. Space is not full of pockets of adversarial examples that finely tile the reals like the rational numbers".
Appendix B and C are not extremely novel in my mind, but definitely add more evidence. 
Appendix E is quite nice since it gives an insight into what actually makes the model resistant to adversarial examples.
Remarks:
- The update for PGD should be using \nabla_{x_t} L(\theta,x_t,y), (rather than only \nabla_x)?
- In table 2, attacking a with 20-step PGD is doing better than 7-step.  When you say "other hyperparameter choices didn’t offer a significant decrease in accuracy", does that include the number of steps? If not why stop there? What happens for more steps? (or is it too computationally intensive?)
- You only seem to consider adversarial examples created from your dataset + adv. noise. What about rubbish class examples? (e.g. rgb noise)