Adversarial attacks on neural networks perturb the input at test time in order to fool trained and deployed neural network models. Most attacks such as gradientbased Fast Gradient Sign Method (FGSM) by Goodfellow et al. (2015) and DeepFool by Moosavi-Dezfooli et al. (2016) are input-dependent, small, pixel-wise perturbations, and they give different attack directions for different inputs. On the other hand, universal adversarial attacks are input-agnostic and the same attack works for most inputs. Translation or rotation-equivariant neural network models provide one approach to prevent universal attacks based on simple geometric transformations. In this paper, we observe an interesting spectral property shared by all of the above input-dependent, pixel-wise adversarial attacks on translation and rotation-equivariant networks. We exploit this property to get a single universal attack direction that fools the model on most inputs. Moreover, we show how to compute this universal attack direction using principal components of the existing input-dependent attacks on a very small sample of test inputs. We complement our empirical results by a theoretical justification, using matrix concentration inequalities and spectral perturbation bounds. We also empirically observe that the top few principal adversarial attack directions are nearly orthogonal to the top few principal invariant directions.
This paper studies the problem of computing non-data specific perturbations, also known as universal perturbations, to attack neural networks and take profit of their inherent vulnerability. Compared to previous works in the domain, the authors look specifically at equivariant networks, and derive geometric insights and methods to compute universal perturbations for these networks. 
The paper starts by analysing the main/principal directions of set of perturbations that are able to change the decisions in different forms of equivariant neural networks. With this heuristic study, a few main directions are shown to be shared by most adversarial perturbations. The authors then propose to construct universal perturbations built on the insights given by the principle directions of perturbations, which is an interesting an effective method. In addition, it is shown that a few adversarial samples are sufficient to identify pretty  accurately the principle directions. The fooling rates achieved by this method is pretty good, which demonstrates that the proposed strategy is reasonable.
The key idea in this paper (using principal shared directions of perturbations, computed on a small subset of data points) has unfortunately already been proposed and tested in classical (non-equivariant) neural networks - see for example Fig 9 in Moosavi-Dezfooli, 2017, cited in the paper, and published in CVPR 2017. The present paper proposes however a few additional bits of information with a nice theoretical analysis, while the previous works were mostly based on heuristics. It is probably not sufficient however to pass the cut in ICLR. 
The interesting additional novelty here is the study of equivariant networks. However, this ends up falling sort of initial expectations - there seems to be nothing specific to equivariant networks in the proposed study, and the solution and algorithm is actually applicable to any neural network architectures (?). Also, no specific insights are derived for equivariant networks, which could be potentially very interesting to make progress in understanding better equivariant representations, which still consist in a widely open research problem. 
In general, the paper has a non-classical organisation, with a lot of heuristics that are not discussed in depth - that gives a sort of high-level impression that the proposed idea is potentially nice, but that but superficially addressed. It should probably be improved in the next versions of this work.