Data augmentation (DA) has been widely utilized to improve generalization in training deep neural networks. Recently, human-designed data augmentation has been gradually replaced by automatically learned augmentation policy. Through finding the best policy in well-designed search space of data augmentation, AutoAugment (Cubuk et al., 2018) can significantly improve validation accuracy on image classification tasks. However, this approach is not computationally practical for large-scale problems. In this paper, we develop an adversarial method to arrive at a computationally-affordable solution called Adversarial AutoAugment, which can simultaneously optimize target related object and augmentation policy search loss. The augmentation policy network attempts to increase the training loss of a target network through generating adversarial augmentation policies, while the target network can learn more robust features from harder examples to improve the generalization. In contrast to prior work, we reuse the computation in target network training for policy evaluation, and dispense with the retraining of the target network. Compared to AutoAugment, this leads to about 12× reduction in computing cost and 11× shortening in time overhead on ImageNet. We show experimental results of our approach on CIFAR-10/CIFAR-100, ImageNet, and demonstrate significant performance improvements over state-of-the-art. On CIFAR-10, we achieve a top-1 test error of 1.36%, which is the currently best performing single model. On ImageNet, we achieve a leading performance of top-1 accuracy 79.40% on ResNet-50 and 80.00% on ResNet-50-D without extra data.
This paper proposes a technique called Adversarial AutoAugment which dynamically learns good data augmentation policies during training. An adversarial approach is used: a target network tries to achieve good classification performance on a training set, while a policy network attempts to foil the target network by developing data augmentation policies that will produce images that are difficult to classify. To train the policy network, each mini-batch is augmented multiple times with different policies sampled from the policy network. Each augmented mini-batch is passed through the target network to produce a corresponding training loss, which is used as the training signal for the policy network. Experimental results are shown for CIFAR-10, CIFAR-100, and ImageNet datasets, where Adversarial AutoAugment outperforms competing methods (AutoAugment and Population Based Augmentation) on a variety of model architectures.
In its current state, I would tend towards rejecting this paper. The overall structure, the figures, and the experimental results are very nice, but there are two major issues that are holding it back. First, I am skeptical that the policy network is actually learning useful policies. Secondly, there are many grammatical errors in the paper which hamper readability. Upon reading the first paragraph of the paper, my initial impression was already quite negative, simply due to the number of grammatical errors. Fixing these would strengthen the paper considerably, and I would increase my score accordingly if properly addressed.
Primary Concerns:
1) One of my main concerns with this paper is this line here: "To guarantee the convergence during adversarial learning, the magnitude of all the operations are set in a moderate range". Since the policy network has no incentive to select transformations that the target network can still learn from, I assume that the ranges are required so that the policy network cannot choose to apply extreme transformations which always fool the target network, such as setting brightness to 0 to make the entire image black. How are acceptable ranges determined? If cross validation is required, then this becomes very much like the original hand-tuning of data augmentations that we wanted to avoid in the first place. 
Additionally, I think it would be useful to to see a plot of the magnitude of each transformation versus training epoch, similar to Figure 4a in [1]. If the policy network simply learns to use the most extreme augmentations available in order to fool the target network, then this may indicate that the gain in performance is from tuning the magnitude ranges, and not from the policy network selecting good policies.
2) The paper could benefit greatly from some revision of the grammar. I would recommend either having a friend or colleague read it over, or even using an automated grammar checking program, such as Grammarly. For example, in the first paragraph alone there are several sentences that could be improved:
"Massive amount of data promotes the great success" -> "Massive amounts of data have promoted the great success"
"when more supervised data available" -> "when more supervised data is available"
"or better data augmentation method adapted" -> "or a better data augmentation method is adopted"
"which can automated learn" -> "which can automatically learn"
"there still requires tens of thousands of GPU-hours consumption." -> "tens of thousands of GPU-hours of computation are still required"
Things to improve the paper that did not impact the score:
3) In the first paragraph it is claimed that data augmentation policies have weak transferability across different tasks and datasets. I do not fully agree with this claim, since papers such as AutoAugment [2] have shown that learned policies are highly transferable to new datasets, and augmentation strategies such as CutMix have been shown to be highly effective for a variety of tasks. 
4) No citation for the original GAN paper, despite multiple mentions of adversarial learning, and GANs themselves.
5) There is no computation time comparison with PBA, which is about 1000x faster than Autoaugment, and therefore roughly 100x faster than Adversarial AutoAugment.
6) CIFAR-10 results are not necessarily state-of-the-art. The 1.36% error rate claimed by the paper is surpassed by work in [3], which achieved 1.33% using a combination of AutoAugment and mixup. 
7) Citation for the CIFAR-10 dataset incorrectly refers to the Adam optimizer paper [4].
References: 
[1] Daniel Ho, Eric Liang, Ion Stoica, Pieter Abbeel, and Xi Chen. Population based augmentation: Efficient learning of augmentation policy schedules. ICML, 2019.
[2] Cubuk, Ekin D., Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. "Autoaugment: Learning augmentation policies from data." CVPR (2019).
[3] Wistuba, Martin, Ambrish Rawat, and Tejaswini Pedapati. "A Survey on Neural Architecture Search." arXiv preprint arXiv:1905.01392 (2019).
[4] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015
EDIT: The authors have addressed the majority of my concerns, and as such I have increased my score from a 3 to a 6.