Deep neural network image classifiers are reported to be susceptible to adversarial evasion attacks, which use carefully crafted images created to mislead a classifier. Recently, various kinds of adversarial attack methods have been proposed, most of which focus on adding small perturbations to input images. Despite the success of existing approaches, the way to generate realistic adversarial images with small perturbations remains a challenging problem. In this paper, we aim to address this problem by proposing a novel adversarial method, which generates adversarial examples by imposing not only perturbations but also spatial distortions on input images, including scaling, rotation, shear, and translation. As humans are less susceptible to small spatial distortions, the proposed approach can produce visually more realistic attacks with smaller perturbations, able to deceive classifiers without affecting human predictions. We learn our method by amortized techniques with neural networks and generate adversarial examples efficiently by a forward pass of the networks. Extensive experiments on attacking different types of nonrobustified classifiers and robust classifiers with defence show that our method has state-of-the-art performance in comparison with advanced attack parallels.
The paper introduces a new approach to generate adversarial examples for deep classifiers. As opposed to the majority of work on adversarial attack models, which generally limit the attacker on pixel-space distortions measured with respect to an Lp norm, the authors here consider a slightly more general attack model that is a combination of an affine transformation and additive L2 perturbation of the input example. 
Finding optimal attacks for this model can be non-trivial (standard due to the highly nonlinear coupling between the affine parameters and the additive perturbation), so the authors instead propose training a surrogate neural network that generates the attack affine-transformation and distortion- parameters sequentially. This can, in principle, be done in a traditional supervised training setup; however, to force the adversarial images to look perceptually close to natural looking images, the authors throw a discriminator loss on top, and train the attack generator network adversarially.
The paper is well-written in general, the idea is intuitive, and the experiments are well-described. However, I have a few concerns that lead to me to give a low score (at least in the first round of reviews).
- Novelty. 
Leveraging spatial distortions (or other visually meaningful transformations) to generate adversarial attacks is not a new idea, but the authors seem to have been unaware of this very large body of work. See, for example:
** Engstrom et al, "Exploring the landscape of spatial robustness", ICML 2019
** Poovendran et al, "Semantic adversarial examples", CVPR 2018
** Ho et al, "Catastrophic Child's Play", CVPR 2019
** Joshi et al, "Semantic adversarial attacks", ICCV 2019
among many others.
Using GAN-like transformation models to generate attacks is also not a new idea. A few of the above papers use this approach, and the authors refer to a few other such papers as well.
So as such, the conceptual novelty of the contribution seems to be low (beyond the specific choice of combining affine and L2 perturbations).
- Experimental evaluation.
The authors do a commendable job thoroughly laying out the experimental setup. However, a couple of red flags emerge in the experiments. First, why not look at L-infty perturbations (as opposed to L2)? Second, why not test on more challenging datasets (CIFAR, CelebA, etc) as opposed to simple black/white datasets such as MNIST/Fashion-MNIST? One would imagine that the smaller, simpler datasets are easier to optimize for, and therefore the "amortized" attack generator networks are not necessary here.
- Weakness of theoretical part.
I am not sure the theorem is saying anything strong or useful (since the underlying transformer neural network is assumed to possess infinite capacity). I would suggest just removing it.