It is becoming increasingly clear that many machine learning classifiers are vulnerable to adversarial examples. In attempting to explain the origin of adversarial examples, previous studies have typically focused on the fact that neural networks operate on high dimensional data, they overfit, or they are too linear. Here we show that distributions of logit differences have a universal functional form. This functional form is independent of architecture, dataset, and training protocol; nor does it change during training. This leads to adversarial error having a universal scaling, as a power-law, with respect to the size of the adversarial perturbation. We show that this universality holds for a broad range of datasets (MNIST, CIFAR10, ImageNet, and random data), models (including state-of-the-art deep networks, linear models, adversarially trained networks, and networks trained on randomly shuffled labels), and attacks (FGSM, step l.l., PGD). Motivated by these results, we study the effects of reducing prediction entropy on adversarial robustness. Finally, we study the effect of network architectures on adversarial sensitivity. To do this, we use neural architecture search with reinforcement learning to find adversarially robust architectures on CIFAR10. Our resulting architecture is more robust to white and black box attacks compared to previous attempts.
This work presents an empirical study aiming at improving the understanding of the vulnerability of neural networks to adversarial examples. Paraphrasing the authors, the main observation of the study is that the vulnerability is due to an inherent uncertainty that neural networks have about their predictions ( the difference between the logits). This is consistent across architectures, datasets. Further, the authors note that "the universality is not a result of the specific content of these datasets nor the ability of the model to generalize."
While this empirical study contains valuable information, its above conclusions are factually wrong. It can be theoretically proven at least using two routes. They are also in contradiction with other empirical observations consistent across several previous studies. 
1-Constructive counter-argument: Consider a neural network that always outputs a constant prediction. It (1) is by definition independent of any dataset (2) generalizes perfectly (3) has zero adversarial error, hence contradicting the central statement of the paper. 
2- Analysis-based counter-argument: Consider a neural network with one hidden layer and two classes. It is easy to show that the difference between the scores (logits) of the two classes is linear in the operator norm of the hidden weight matrix and linear in the L2-norm of the last weight vector. Therefore, the robustness of the model indeed depends on its capability to generalize because the latter is essentially governed by the geometric margin of the linear separator and the spectral norm of the weight matrix (see [1,2,3]). QED.
3- Further, the lack of calibration of neural networks and its causes are well known. Among other things, it is due to the use of building blocks (such as batch-norm [4]), regularization (e.g., weight decay) or the use of softmax+cross-entropy during training. While this is convenient for optimization reasons, it indeed hurts the calibration. The authors should try to train a neural network with a large margin criteria and see if the same phenomenon still holds when they measure the geometric margin. Another alternative is to use a temperature with the softmax[4]. Therefore, the observations of the empirical study cannot be generalized to neural networks and should be explicitly restricted to neural networks using softmax with cross-entropy as criteria. 
I believe the conclusions of this study are misleading, hence I recommend to reject the paper. 
[1] Spectrally Normalized Margin-bounds Margin bounds for neural networks (Bartlett et al., 2017)
[2] Parseval Networks: Improving Robustness to Adversarial Examples (Cisse et al., 2017) 
[3] Formal Guarantees on the Robustness of a classifier against adversarial examples (Hein et al., 2017)
[4] On the Calibration of Modern Neural Networks (Guo et al., 2017)