Recent research has shown that CNNs are often overly sensitive to high-frequency textural patterns. Inspired by the intuition that humans tend to be more sensitive to lower-frequency (larger-scale) patterns, we design a regularization scheme that penalizes large differences between adjacent components within each convolutional kernel. We apply our regularization onto several popular training methods, demonstrating that the models with the proposed smooth kernels enjoy improved adversarial robustness. Further, building on recent work establishing connections between adversarial robustness and interpretability, we show that our method appears to give more perceptually-aligned gradients.
Paper summary:  This paper argues that reducing the reliance of neural networks on high-frequency components of images could help robustness against adversarial examples. To attain this goal, the authors propose a new regularization scheme that encourages convolutional kernels to be smoother. The authors augment standard loss functions with the proposed regularization scheme and study the effect on adversarial robustness, as well as perceptual-alignment of model gradients.
Comments: I will first discuss some high-level concerns I have with the paper, followed by more specific comments.
Motivation and general idea:
The idea of improving model robustness by eliminating high-frequency components in the data is not new. This has in fact been the motivation behind several (later shown to be unsuccessful) defenses such as JPEG compression. In recent work, Yin et al. explore this phenomenon in depth, and importantly discuss how adversarial examples are by no means entirely a high-frequency phenomenon. Specifically, they show that while adversarial examples for natural models do tend to be biased towards higher-frequency components, this is not true for robust models. They argue that it is in general always possible to find adversarial examples in the lower end of the frequency spectrum (for instance, transfer attacks from robust models). Thus, based on evidence from prior work, it is unlikely that a defense based entirely on removing high-frequency components from the input data can be successful.
Empirical evaluation:
There are significant issues with the empirical evaluation of the proposed defense. In particular, in Table 1 (also Figures 7-9 in the Appendix):
1. The numbers suggest that FGSM is more successful as an attack than PGD. The later is a multi-step version of the former, and hence should be strictly better (more successful in lowering model accuracy). This clearly highlights that the PGD attack is not being used correctly/not run with enough steps, and thus the numbers are not an accurate reflection of model robustness.
2. At a higher-level, the numbers that the authors highlight in the table are the best performance over attacks. This is not the correct way to evaluate robustness, which must always be reported as the *worst-case performance* of the model and hence the lowest accuracy over attacks. If one takes this into consideration, it is clear that the proposed regularization does not really improve robustness.
3. Furthermore, the authors state they use default parameters from Foolbox to evaluate their models. The issue with this is evident in the MNIST/FashionMNIST results where even with arbitrary eps, strong attacks like PGD are not able to break the model. This does not mean that the model is robust, it just means that the default hyperparameters from Foolbox cannot break the model. The authors need to re-run the attacks with different steps sizes/step counts for various attacks. The goal in evaluating model robustness should not be finding one attack (or one set of hyperparameters) which is not able to fool the model, but to show that *no attack* (for the given perturbation set) can lower model accuracy.
4. The eps that are used for the CIFAR-10 and ImageNet experiments are extremely large, and not standard in the literature. In fact, based on my experience, it is probably not possible to be (too) robust to eps as large as 0.1 on CIFAR as this is large enough to visibly change the class even for a human (cf. Tsipras et al.). 
Thus, I think the robustness evaluation in Table 1 is incorrect and violates some basic sanity checks (such as PGD being stronger than FGSM), and thus does not accurately reflect the model’s true performance. 
Other comments:
i. It is unclear why the sensitivity of the proposed regularization scheme to the scale of w can be fixed by subtracting the norm of w (the exp loss will just be raised to the power alpha if you scale weights by alpha). Dividing by the norm would be the more correct way to fix the scale invariance and the authors should include results with this regularization, at least in the Appendix.
ii. In Figure 1, the kernels and activations that are visualized are after the first convolutional layer. The authors should perform a similar visualization after the second layer (or later layers in general) because it is somewhat obvious that you will get these properties at the first layer with the proposed regularization scheme (if the lambda is correctly tuned). The part that is unclear is how effective the regularization is over repeated applications of convolution coupled with non-linearities such as ReLU/pooling. 
iii. While the visualizations in Figures 2-6 are interesting, it seems that many of the cases in which +R looks semantically meaningful, it also looks meaningful with the base loss itself. It thus is unclear whether the perceptual alignment is coming from the base loss, or from the added regularization. Moreover, the perceptual alignment of models should improve when their dependance on “human-meaningless” features reduces. Since high-frequency components are likely “human-meaningless” features, it is plausible that the proposed regularization scheme makes activation maximization/gradients more perceptually alignment. However, I think this is somewhat orthogonal to (and is not enough to tell us anything about) the robustness of the model itself, which is affected by its reliance on many kinds of “non-robust” features (Ilyas et al., 2019) of which high-frequency components might just be one example.
iv. In recent work by Wang et al., the authors conduct an interesting experiment to see how dependent the prediction of a given classifier is to high-frequency components in the data (vs low frequency components) (cf. Figure 1 in their paper). It would be interesting to see this experiment replicated with the proposed regularization scheme.
Overall, I think there are significant issues with the paper, especially in the empirical evaluation section. The authors need to re-evaluate their models with the strongest possible form of the attack and demonstrate an improvement in worst-case performance to actually establish the merits of the proposed regularization scheme. Thus, I recommend rejection.
References:
Yin, Dong, et al. "A fourier perspective on model robustness in computer vision." arXiv preprint arXiv:1906.08988 (2019).
Tsipras, Dimitris, et al. "Robustness may be at odds with accuracy." arXiv preprint arXiv:1805.12152 (2018).
Ilyas, Andrew, et al. "Adversarial examples are not bugs, they are features." arXiv preprint arXiv:1905.02175 (2019).
Wang, Haohan, et al. "High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks." arXiv preprint arXiv:1905.13545 (2019).