L 2 -NONEXPANSIVE NEURAL NETWORKS
This paper proposes a class of well-conditioned neural networks in which a unit amount of change in the inputs causes at most a unit amount of change in the outputs or any of the internal layers. We develop the known methodology of controlling Lipschitz constants to realize its full potential in maximizing robustness, with a new regularization scheme for linear layers, new ways to adapt nonlinearities and a new loss function. With MNIST and CIFAR-10 classifiers, we demonstrate a number of advantages. Without needing any adversarial training, the proposed classifiers exceed the state of the art in robustness against white-box L 2 -bounded adversarial attacks. They generalize better than ordinary networks from noisy data with partially random labels. Their outputs are quantitatively meaningful and indicate levels of confidence and generalization, among other desirable properties.
Summary:
The paper presents techniques for training a non expansive network, which keeps the Lipchitz constant of all layers lower than 1. While being non-expansive, means are taken to preserve distance information better than standard networks. The architectural changes required w.r.t standard networks are minor, and the most interesting changes are made to the loss minimized. The main claim of the paper is that the method is robust against adversarial attacks of a certain kind. However, the results presented show that a) such robustness comes at a high cost of accuracy for standard examples, and b) even though the network is preferable to a previous alternative in combating adversarial examples, the accuracy obtained in the face of adversarial attacks is too low to be of practical value. Other properties of the networks, explored empirically, are that the confidence of the prediction is indicative of robustness (to adversarial attacks) and that the networks learn better in the presence of high label noise. 
In short, this paper may be of interest to a sub-community interested in defense against certain types of adversarial attacks, even when the defense level is much too low to be practical. I am not part of this community, hence did not find this part very interesting. I believe the regularization results are of wider interest. However, to present this as the main contribution of L2NNN more work is required to find configuration which are resilient to overfit yet enable high training accuracy, and more diverse experiments are required.
Pros:
+ the idea of non expansive network is interesting and important
+ results indicate some advantages in fighting adversarial examples and label noise
Cons:
- the results for fighting adversarial examples are not significant from a practical perspective
- the results for copying with label noise are preliminary and require expansion with more experiments.
- the method has costs in accuracy, which is lower than standard networks and this issue is not faced with enough attention
- presentation clarity is medium: proofs for claims are missing, as well as relevant background on the relevant adversarial attacks. The choice to place the related work at the end also reduces presentation clarity.
More detailed comments:
Pages 1-3: In many places, small proofs are left to the reader as ‘straightforward’. Examples are: the claim in the introduction, in eq. 2, in section 2.2, section 2.3’ last line of page 3, etc.. While the claim are true (in the cases I tried to verify them long enough), this makes reading difficult and not fluent. For some of these claims I do not see the argument behind them. In general, I think proofs should be brought for claims, and short proofs (preferably) should be brought for small claims. Leaving every proof to the reader as an exercise is not a convenient strategy. 
Page 4: The loss is complex and its terms utility require empirical evidence. The third term is shown to be clearly useful, enabling a trade off between train accuracy and margin. However, the utility of terms 4) and 5) is not verified. Do we really need both these terms? Cannot we just stay with one?
The main claim is robustness w.r.t “white-box non targeted L2-bounded attacks”. This seems to be a very specific attack type, and it is not explained at all in the text. Hence it is hard to judge the value of this robustness. Explanation of adversarial attack kinds, and specifically of “white-box non targeted
L2-bounded attacks” is required for this paper to be a stand alone readable paper. Similarly ‘L_\infty’-bounded attacks, for which results are shown, should be explained.
Table 1,2: First, the model architecture used in these experiments is not stated. Second, the accuracy of the ‘natural’ baseline classifier, at least in the MNist case, is somewhat low – much better results can be obtained with CNN on MNist. Third, the accuracies of the suggested robust models are very low compared to what can be obtained on these datatsets. Forth, while the accuracies under attack of the proposed method are better than those of Madri et al., both are quite poor and indicate that the classifier is not useful under attack (from a practical perspective).
Page 6: The classifiers which share the work between an L2NNN network and a regular more accurate network may be interesting, as the accuracies reported for them are significantly higher than the L2NNN networks. However, the robustness scores are not reported for these classifiers, so it is not possible to judge if they lead to a practical and effective strategy.
Page 7: For me, the results with partially random labels are the most interesting in the paper. The resistance of L2NNN to overfit and its ability to learn with very noisy data are considerably better than the suggested alternatives.
Relevant work not mentioned “Spectral Norm Regularization for Improving the Generalizability of Deep Learning” - Yuichi Yoshida and Takeru Miyato, Arxiv, 2017.
I have read the rebuttal.
The discussion was interesting, but I do not see a need to change my assessment.
The example of ad-blocking in indeed a case (the first I encounter) where l2- perturbated adversarial examples can be useful for cyber attack. The other ones are less relevant (the attacks are not based on adversarial attacks in the sense used in the paper: images crated with small gradient-direction perturbations). Anyway talking about 'attacks on a self-driving car' are still not neaningful to me: I do not understand what adversarial examples have to do with this.
I do not find the analogy of 'rocket improvements and moon landing' convincing: in 69 rocket improvements were of high interest in multiple applications, and moon landing was visible over the corner.