In this work, we study the connection between regularization and robustness of deep neural networks by viewing them as elements of a reproducing kernel Hilbert space (RKHS) of functions and by regularizing them using the RKHS norm. Even though this norm cannot be computed, we consider various approximations based on upper and lower bounds. These approximations lead to new strategies for regularization, but also to existing ones such as spectral norm penalties or constraints, gradient penalties, or adversarial training. Besides, the kernel framework allows us to obtain margin-based bounds on adversarial generalization. We show that our new algorithms lead to empirical benefits for learning on small datasets and learning adversarially robust models. We also discuss implications of our regularization framework for learning implicit generative models.
This paper looks at adversarial examples from the context of RKHS norms for neural networks.  The work builds conceptually on the work of Bietti and Mairal (2018), who investigate approximate RKHS norms for neural networks (including computation via a specialized convolutional kernel), and Xu et al., (2009) which looks at robustness properties of kernel classifiers.  The authors discuss how the RKHS norm of neural network functions provide robustness guarantees for the resulting classifier, both in terms of a straightforward robustness property for a given example, as well as in terms of generalization guarantees about robustness.
Overall, I think there are some interesting ideas in this work, but ultimately not enough to make a compelling independent paper.  The core issue here is that the RKHS properties are used only in a very minimal manner to actually provide much analysis or insight into the robustness properties of the network.  For example, the upper bound in (8) seems to be central here to illustrating how a bound on the RKHS norm can be upper bounded as a function of the operator l2 norm of the inner weight matrices (though the actual form of the bound isn't mentioned), and the latter term could thus provide a certified bound on the robustness loss of a classifier.  However, there are two big issues here: 1) it's trivial to directly bound the l2 robustness of a classifier by the product of the weight spectral norms and 2) the actual regularization term the authors proposed to use (the sum of spectral norms) is notably _not_ an upper bound on either the robust loss or the RKHS norm; naturally, this penalty along with the constrained version will still provide some degree of control over the actual robustness, but the authors don't connect this to any real bound.  I also think the authors aren't properly acknowledging just how similar this is to past work: the Parseval networks paper (Cisse et al., 2017), for instance, presents a lot of similar discussion of how to bound generalization error based based upon terms involving operator norms of the matrices, and the actual spectral normalization penalty that the authors advocate for has been studied by Miyato et al. (2018).  To be clear, both of these past works (and several similar ones) are of course cited by the current paper, but from a practical standpoint it's just not clear to me what the takeaways should be here above and beyond this past work, other than the fact that these quantities _also_ bound the relevant RKHS norms.  Likewise the generalization bound in the paper is a fairly straightforward application of existing bounds given the mechanics of the RKHS norm defined by previous work.
To be clear, I think the RKHS perspective that the authors advocate for here is actually quite interesting.  I wasn't particularly familiar with the Bietti and Mairal (2018) work, and going through it in some detail for reviewing this paper, I think it's an important directly for analysis of deep networks, including from a perspective of robustness.  But the results here seem more like a brief follow-on note to the past work, not a complete set of results in and of themselves.  Indeed, because the robustness perspective here can largely be derived completely independently of the RKHS framework, and because the resulting training procedures seem to be essentially identical to previously-proposed approaches, the mere contribution of connecting these works to the RKHS norm doesn't seem independently to be enough of a contribution in my mind.
One final, though more minor, point: It's worth pointing out that (globally) bounding the Lipschitz constant seems top stringent a condition for most networks, and most papers on certifiable robustness seem to instead focus on some kind of local Lipschitz bound around the training or test examples.  Thus, it's debatable whether even the lower bound on the RKHS norm is really reasonable to consider for the purposes of adversarial robustness.