summaries: This paper target the task of automatically determining the best augmentation method to obtain improved accuracy. While the previous related studies focus on the image-level augmentation and ignore the semantic information of the augmented images, the proposed algorithm augments the grid-wise patches of the given input with the preserved semantic information. To overcome the enlarged number of combinations to consider all the patches, the algorithm utilizes the MARL algorithm with the unified reward function. By MARL algorithm, the number of parameters can be reduced and the training speed can be much improved, compared to the previous auto-augmentation methods. Through the image classification and fine-grained image recognition tasks, the proposed algorithm was validated, and it shows the state-of-the-art performance among the compared methods. This paper proposes an automatic data augmentation approach. Different from existing works, they proposed to augment patches in the image rather than the whole image. The approach is formulated as a multi-agent reinforcement learning problem. They empirically show the effectiveness of their approach across several image classification datasets. This paper proposed a fine-grained automated data augmentation approach, Patch AutoAugment (PAA), which tries to increase diversity in local regions by divide an image into a grid of patches and search for the joint optimal augmentation policies for the patches. The proposed PAA considers the task as a multi-agent reinforcement learning problem,  and adopt a multi-agent reinforcement learning algorithm to automatically search for the optimal augmentation policies by considering the contextual relationship between the patches. They verify the proposed method on many classification and fine-grained recognition dataset(CIFAR10, CIFAR-100, ImageNet, CUB-200-2011, Stanford Cars and FGVC-Aircraft). The experiments show a good result and visualization results provide some insights that the PAA  help the target network to localize more class-related cues.    The paper proposes an evolution of the traditional   pipeline of image data augmentation used to reduce ML   model overfitting.    Instead of applying transformations such as shear,   rotate, CutOut, etc. at the image level, the proposed   technique divides the images into a fixed grid and   applies a potentially different transformation to each   cell. The problem of selecting a transform for each cell   is cast as a multi-agent RL (MARL) task, and the agents   learn as the main network trains within a (multi-agent)   Advantage Actor Critic framework.    The agents use a shared reward mechanism, with the reward   defined as the difference between the loss on the   augmented sample and the original loss.    The regular grid is fixed for a dataset and   hyperparameters like the magnitude of the augmentations   follow a fixed schedule; the agents only pick _which_   augmentation to apply on a patch.    Experiments performed on CIFAR-{10,100}, ImageNet,   CUB-200-2011, Stanford Cars, and FGVC-Aircraft show   relatively small but very consistent improvements in   terms of image classification accuracy. Different design   choices (MARL vs. single-agent vs. random, grid size,   etc.) are ablated and discussed in detail. 
abstractText: Data augmentation (DA) plays a critical role in improving the generalization of deep learning models. Recent works on automatically searching for DA policies from data have achieved great success. However, existing automated DA methods generally perform the search at the image level, which limits the exploration of diversity in local regions. In this paper, we propose a more fine-grained automated DA approach, dubbed Patch AutoAugment, to divide an image into a grid of patches and search for the joint optimal augmentation policies for the patches. We formulate it as a multi-agent reinforcement learning (MARL) problem, where each agent learns an augmentation policy for each patch based on its content together with the semantics of the whole image. The agents cooperate with each other to achieve the optimal augmentation effect of the entire image by sharing a team reward. We show the effectiveness of our method on multiple benchmark datasets of image classification, fine-grained image recognition and object detection (e.g., CIFAR-10, CIFAR-100, ImageNet, CUB-200-2011, Stanford Cars, FGVC-Aircraft and Pascal VOC 2007). Extensive experiments demonstrate that our method outperforms the state-of-the-art DA methods while requiring fewer computational resources.
references.authors: Kai Arulkumaran Marc Peter Deisenroth Miles Brundage Anil Anthony Bharath
references.title: Deep reinforcement learning: A brief survey
references.venue: IEEE Signal Processing Magazine,
references.authors: Craig Boutilier
references.title: Planning, learning and coordination in multiagent decision processes
references.venue: In TARK,
references.authors: Lucian Busoniu Robert Babuska Bart De Schutter
references.title: A comprehensive survey of multiagent reinforcement learning
references.venue: IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews),
references.authors: Yue Chen Yalong Bai Wei Zhang Tao Mei
references.title: Destruction and construction learning for finegrained image recognition
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Tianshu Chu Shuhui Qu Jie Wang
references.title: Large-scale multi-agent reinforcement learning using image-based state representation
references.venue: In 2016 IEEE 55th Conference on Decision and Control (CDC),
references.authors: Ekin D Cubuk Barret Zoph Dandelion Mane Vijay Vasudevan Quoc V Le
references.title: Autoaugment: Learning augmentation policies from data
references.venue: arXiv preprint arXiv:1805.09501,
references.authors: Ekin D Cubuk Barret Zoph Jonathon Shlens Quoc V Le
references.title: Randaugment: Practical automated data augmentation with a reduced search space
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,
references.authors: Ali Dabouei Sobhan Soleymani Fariborz Taherkhani Nasser M Nasrabadi
references.title: Supermix: Supervising the mixing data augmentation
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Jia Deng Wei Dong Richard Socher Li-Jia Li Kai Li Li Fei-Fei
references.title: Imagenet: A large-scale hierarchical image database
references.venue: IEEE conference on computer vision and pattern recognition,
references.authors: Terrance DeVries Graham W Taylor
references.title: Improved regularization of convolutional neural networks with cutout
references.venue: arXiv preprint arXiv:1708.04552,
references.authors: Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly
references.title: An image is worth 16x16 words: Transformers for image recognition at scale
references.venue: arXiv preprint arXiv:2010.11929,
references.authors: Ruoyi Du Dongliang Chang Ayan Kumar Bhunia Jiyang Xie Zhanyu Ma Yi-Zhe Song Jun Guo
references.title: Fine-grained visual classification via progressive multi-granularity training of jigsaw patches
references.venue: In European Conference on Computer Vision,
references.authors: Mark Everingham John Winn
references.title: The pascal visual object classes challenge 2007 (voc2007) development kit
references.authors: Jakob Foerster Gregory Farquhar Triantafyllos Afouras Nantas Nardelli Shimon Whiteson
references.title: Counterfactual multi-agent policy gradients
references.venue: In Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Jakob N Foerster Richard Y Chen Maruan Al-Shedivat Shimon Whiteson Pieter Abbeel Igor Mordatch
references.title: Learning with opponent-learning awareness
references.venue: arXiv preprint arXiv:1709.04326,
references.authors: Ryosuke Furuta Naoto Inoue Toshihiko Yamasaki
references.title: Pixelrl: Fully convolutional network with reinforcement learning for image processing
references.venue: IEEE Transactions on Multimedia,
references.authors: Xavier Gastaldi
references.title: Shake-shake regularization
references.venue: arXiv preprint arXiv:1705.07485,
references.authors: Chengyue Gong Dilin Wang Meng Li Vikas Chandra Qiang Liu
references.title: Keepaugment: A simple information-preserving data augmentation approach
references.venue: arXiv preprint arXiv:2011.11778,
references.authors: Raphael Gontijo-Lopes Sylvia J Smullin Ekin D Cubuk Ethan Dyer
references.title: Affinity and diversity: Quantifying mechanisms of data augmentation
references.venue: arXiv preprint arXiv:2002.08973,
references.authors: Dongyoon Han Jiwhan Kim Junmo Kim
references.title: Deep pyramidal residual networks
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Lei Han Peng Sun Yali Du Jiechao Xiong Qing Wang Xinghai Sun Han Liu Tong Zhang
references.title: Grid-wise control for multi-agent reinforcement learning in video game ai
references.venue: In International Conference on Machine Learning,
references.authors: Ryuichiro Hataya Jan Zdenek Kazuki Yoshizoe Hideki Nakayama
references.title: Faster autoaugment: Learning augmentation strategies using backpropagation
references.venue: arXiv preprint arXiv:1911.06987,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Dan Hendrycks Norman Mu Ekin D Cubuk Barret Zoph Justin Gilmer Balaji Lakshminarayanan
references.title: Augmix: A simple data processing method to improve robustness and uncertainty
references.authors: Daniel Ho Eric Liang Xi Chen Ion Stoica Pieter Abbeel
references.title: Population based augmentation: Efficient learning of augmentation policy schedules
references.venue: In International Conference on Machine Learning,
references.authors: Sungeun Hong Sungil Kang Donghyeon Cho
references.title: Patch-level augmentation for object detection in aerial images
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops,
references.authors: Shaoli Huang Xinchao Wang Dacheng Tao
references.title: Snapmix: Semantically proportional mixing for augmenting fine-grained data
references.venue: arXiv preprint arXiv:2012.04846,
references.authors: Jang-Hyun Kim Wonho Choo Hosan Jeong Hyun Oh Song
references.title: Co-mixup: Saliency guided joint mixup with supermodular diversity
references.venue: arXiv preprint arXiv:2102.03065,
references.authors: Jonathan Krause Michael Stark Jia Deng Li Fei-Fei
references.title: 3d object representations for fine-grained categorization
references.venue: In Proceedings of the IEEE international conference on computer vision workshops,
references.authors: Alex Krizhevsky Geoffrey Hinton
references.title: Learning multiple layers of features from tiny images
references.authors: Yonggang Li Guosheng Hu Yongtao Wang Timothy Hospedales Neil M Robertson Yongxin Yang
references.title: Dada: Differentiable automatic data augmentation
references.venue: arXiv preprint arXiv:2003.03780,
references.authors: Xuan Liao Wenhao Li Qisen Xu Xiangfeng Wang Bo Jin Xiaoyun Zhang Yanfeng Wang Ya Zhang
references.title: Iteratively-refined interactive 3d medical image segmentation with multi-agent reinforcement learning
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Sungbin Lim Ildoo Kim Taesup Kim Chiheon Kim Sungwoong Kim
references.title: Fast autoaugment
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Chen Lin Minghao Guo Chuming Li Xin Yuan Wei Wu Junjie Yan Dahua Lin Wanli Ouyang
references.title: Online hyper-parameter learning for auto-augmentation strategy
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Aoming Liu Zehao Huang Zhiwu Huang Naiyan Wang
references.title: Direct differentiable augmentation search
references.venue: arXiv preprint arXiv:2104.04282,
references.authors: Ze Liu Yutong Lin Yue Cao Han Hu Yixuan Wei Zheng Zhang Stephen Lin Baining Guo
references.title: Swin transformer: Hierarchical vision transformer using shifted windows
references.venue: arXiv preprint arXiv:2103.14030,
references.authors: Ryan Lowe Yi Wu Aviv Tamar Jean Harb Pieter Abbeel Igor Mordatch
references.title: Multi-agent actorcritic for mixed cooperative-competitive environments
references.venue: arXiv preprint arXiv:1706.02275,
references.authors: Chaofan Ma Qisen Xu Xiangfeng Wang Bo Jin Xiaoyun Zhang Yanfeng Wang Ya Zhang
references.title: Boundary-aware supervoxel-level iteratively refined interactive 3d image segmentation with multi-agent reinforcement learning
references.venue: IEEE Transactions on Medical Imaging,
references.authors: Jie Ma Stephen Cameron
references.title: Combining policy search with planning in multi-agent cooperation
references.venue: In Robot Soccer World Cup,
references.authors: Subhransu Maji Esa Rahtu Juho Kannala Matthew Blaschko Andrea Vedaldi
references.title: Fine-grained visual classification of aircraft
references.venue: arXiv preprint arXiv:1306.5151,
references.authors: Volodymyr Mnih Adria Puigdomenech Badia Mehdi Mirza Alex Graves Timothy Lillicrap Tim Harley David Silver Koray Kavukcuoglu
references.title: Asynchronous methods for deep reinforcement learning
references.venue: In International conference on machine learning,
references.authors: Samuel G Müller Frank Hutter
references.title: Trivialaugment: Tuning-free yet state-of-the-art data augmentation
references.venue: arXiv preprint arXiv:2103.10158,
references.authors: Gonçalo Neto
references.title: From single-agent to multi-agent reinforcement learning: Foundational concepts and methods
references.venue: Learning theory course,
references.authors: Tabish Rashid Mikayel Samvelyan Christian Schroeder Gregory Farquhar Jakob Foerster Shimon Whiteson
references.title: Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning
references.venue: In International Conference on Machine Learning,
references.authors: Shaoqing Ren Kaiming He Ross Girshick Jian Sun
references.title: Faster r-cnn: Towards real-time object detection with region proposal networks
references.venue: Advances in neural information processing systems,
references.authors: Edgar Riba Dmytro Mishkin Daniel Ponsa Ethan Rublee Gary Bradski
references.title: Kornia: an open source differentiable computer vision library for pytorch
references.venue: In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,
references.authors: Ramprasaath R Selvaraju Michael Cogswell Abhishek Das Ramakrishna Vedantam Devi Parikh Dhruv Batra
references.title: Grad-cam: Visual explanations from deep networks via gradient-based localization
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Connor Shorten Taghi M Khoshgoftaar
references.title: A survey on image data augmentation for deep learning
references.venue: Journal of Big Data,
references.authors: Ashish Shrivastava Tomas Pfister Oncel Tuzel Joshua Susskind Wenda Wang Russell Webb
references.title: Learning from simulated and unsupervised images through adversarial training
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Satinder Singh Tommi Jaakkola Michael L Littman Csaba Szepesvári
references.title: Convergence results for single-step on-policy reinforcement-learning algorithms
references.venue: Machine learning,
references.authors: Sahil Singla Besmira Nushi Shital Shah Ece Kamar Eric Horvitz
references.title: Understanding failures of deep networks via robust feature extraction
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Richard S Sutton Andrew G Barto
references.title: Reinforcement learning: An introduction
references.venue: MIT press,
references.authors: Ardi Tampuu Tambet Matiisen Dorian Kodelja Ilya Kuzovkin Kristjan Korjus Juhan Aru Jaan Aru Raul Vicente
references.title: Multiagent cooperation and competition with deep reinforcement learning
references.venue: PloS one,
references.authors: Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles Hervé Jégou
references.title: Training data-efficient image transformers & distillation through attention
references.venue: In International Conference on Machine Learning,
references.authors: Toan Tran Trung Pham Gustavo Carneiro Lyle Palmer Ian Reid
references.title: A bayesian data augmentation approach for learning deep models
references.venue: arXiv preprint arXiv:1710.10564,
references.authors: AFM Uddin Mst Monira Wheemyung Shin TaeChoong Chung Sung-Ho Bae
references.title: Saliencymix: A saliency guided data augmentation strategy for better regularization
references.venue: arXiv preprint arXiv:2006.01791,
references.authors: Sangdoo Yun Dongyoon Han Seong Joon Oh Sanghyuk Chun Junsuk Choe Youngjoon Yoo
references.title: Cutmix: Regularization strategy to train strong classifiers with localizable features
references.venue: In Proceedings of the IEEE International Conference on Computer Vision,
references.authors: Sergey Zagoruyko Nikos Komodakis
references.title: Wide residual networks
references.venue: arXiv preprint arXiv:1605.07146,
references.authors: Hongyi Zhang Moustapha Cisse Yann N Dauphin David Lopez-Paz
references.title: mixup: Beyond empirical risk minimization
references.venue: arXiv preprint arXiv:1710.09412,
references.authors: Kaiqing Zhang Zhuoran Yang Tamer Başar
references.title: Multi-agent reinforcement learning: A selective overview of theories and algorithms
references.venue: arXiv preprint arXiv:1911.10635,
references.authors: Xinyu Zhang Qiang Wang Jian Zhang Zhao Zhong
references.title: Adversarial autoaugment
references.venue: arXiv preprint arXiv:1912.11188,
references.authors: Zhun Zhong Liang Zheng Guoliang Kang Shaozi Li Yi Yang
references.title: Random erasing data augmentation
references.venue: In AAAI,
references.authors: Bolei Zhou Aditya Khosla Agata Lapedriza Aude Oliva Antonio Torralba
references.title: Learning deep features for discriminative localization
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Li Zhou Zhaohui Yang Qing Yuan Zongtan Zhou Dewen Hu
references.title: Salient region detection via integrating diffusion-based compactness and local contrast
references.venue: IEEE Transactions on Image Processing,
references.authors: Barret Zoph Vijay Vasudevan Jonathon Shlens Quoc V Le
references.title: Learning transferable architectures for scalable image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
sections.text: 1 INTRODUCTION
  Data Augmentation (DA) has been widely used to alleviate the overfitting risk in training deep neural networks by appropriately enriching the diversity of training data (Shorten & Khoshgoftaar, 2019; Gontijo-Lopes et al., 2020). Notable DA methods improve the performance and robustness of the neural networks, such as rotation, Mixup (Zhang et al., 2017) and Cutmix (Yun et al., 2019). But these approaches are typically handcraft and require human prior knowledge, which causes weak transferability of DA across different datasets. To relieve the dependence on manual design and further explore more adaptive augmentation, AutoAugment (AA) (Cubuk et al., 2018), as a new DA paradigm, is proposed to automate the search of the optimal DA policies (i.e., DA operation, probability and magnitude) from the training dataset. To be specific, AA trains a proxy model with the augmentation policy generated by a controller, which is updated through reinforcement
  learning using validation accuracy as the reward signal. In spite of the superior performance of AA, its optimization procedure is computationally intensive due to the need to evaluate thousands of policies. Therefore, to reduce the search costs, multiple automated DA approaches (Lim et al., 2019; Ho et al., 2019; Hataya et al., 2019; Zhang et al., 2019b; Lin et al., 2019) have been proposed. For example, (Lim et al., 2019) employs density matching as a search method to accelerate the policy search, and (Zhang et al., 2019b) introduces adversarial learning to organize the target network training and augmentation policies search in an online manner.
  Yet the aforementioned automated DA methods all search for policies at the image level. They ignore the exploration of diversity in local regions, which may result in insufficient diversity of dataset and limit the benefits of DA (Gontijo-Lopes et al., 2020). In addition, due to this coarse-grained augmentation, they may damage key semantic information and introduce ambiguity into the training process (see a simple example in Figure 1 row 2, column 2). With those in mind, it is necessary to automatically search for the optimal augmentation policies for different regions by taking regional diversity into account. One straightforward idea is to directly apply image-wise automated DA methods in different regions. But such an intuitive solution ignores the contextual relationship between regions, which may lead to the non-globally optimal effectiveness of DA policies across the entire image. Besides, it may encounter an extremely high computational cost due to the need of optimizing multiple policies for regions respectively.
  In this paper, we propose a new approach, named Patch AutoAugment (PAA) (see the last row of Figure 1), to address the above-mentioned problems. We first divide an image into a grid of patches to increase the flexibility of representations for different regions and take “patch” as the basic control unit. Then, we model the search for the augmentation policies of patches as a fully cooperative multiagent task, and we leverage a multi-agent reinforcement learning (MARL) algorithm where agents cooperatively learn the policies. Specifically, based on the content of each patch and the semantics of the entire image, the agent searches for a policy in terms of choosing which transformation to apply out of the pre-defined DA operations. To encourage our policy networks to adaptively learn the beneficial augmentation policies for the target network, we use the feedback of the target network as the team reward signal to guide the policy networks to learn on the fly. All agents wind up benefiting from two mechanisms (i.e., parameter sharing and centralized training with decentralized execution) in MARL. In this way, all agents collaboratively and parallelly learn policies to further achieve the joint optimal DA policy across the whole image and alleviate the computational cost.
  In summary, our contributions can be summarized as follows: 1) We pinpoint that exploring diversity in local regions is important for automated learned DA approaches. To our best knowledge, we are the first to propose a more fine-grained automated DA approach, that searches for the optimal policies for the patches according to the content of the patch and the semantics of the entire image. 2) To further achieve the joint optimal policy across the image, we model the DA policy search of patches as a fully cooperative multi-agent task, and adopt a multi-agent reinforcement learning algorithm for Patch AutoAugment by considering the contextual relationship between the patches. 3) Our visualization results provide some insights to the DA community on which augmentation operation should be chosen for patches with different content during the whole training process.
sections.heading: 2 RELATED WORK
sections.heading: 2.1 DATA AUGMENTATION
sections.text: Despite the remarkable performance of deep learning models in computer vision tasks, they often suffer from overfitting. Data augmentation (DA) as an effective technique has been proved to im-
  prove the generalization ability of deep learning models (Shorten & Khoshgoftaar, 2019). Previous works (Cubuk et al., 2020; Gontijo-Lopes et al., 2020) indicate that the main benefit of DA arises from increasing the diversity of images. Popular techniques, such as rotation, flipping, color transformation, have been performed as commonly used augmentation methods. Recently, thanks to the skillful design of human experts, many DA methods (e.g., Cutout (DeVries & Taylor, 2017), Mixup (Zhang et al., 2017), CutMix (Yun et al., 2019)) have been proposed and show significant performance. However, these manually designed methods require additional human prior knowledge on the dataset and sometimes they are limited to certain datasets and target tasks. Naturally, automatically finding DA methods from data have emerged to overcome the limitations of dependence on manual cumbersome exploration. Some works use generative adversarial networks (GANs) to directly generate training data (Shrivastava et al., 2017; Tran et al., 2017).
  Furthermore, recent studies aim to automate the search for augmentation policies that choose the optimal transformations for training images. AutoAugment (AA) (Cubuk et al., 2018) adopts a controller to generate an augmentation policy which is used to train a proxy network, then gets the validation accuracy as the reward signal to update the controller using reinforcement learning. Unfortunately, the evaluation of thousands of policies makes AA computationally expensive. Therefore, multiple automated DA approaches focus on reducing the huge complexity and have achieved great progress. For example, PBA (Ho et al., 2019) employs hyperparameter optimization, Fast AA (Lim et al., 2019) uses a density matching algorithm, Adv AA (Zhang et al., 2019b) proposes an adversarial framework to jointly optimize the target network and the augmentation network, and DADA (Li et al., 2020) and DDAS (Liu et al., 2021a) target relaxing the discrete DA policy selection to a differentiable problem. Besides, RandAugment (Cubuk et al., 2020) removes the separate search on a proxy task in the offline automated DA methods (e.g., AA) while also outperforming them. However, these automated DA methods perform the search at the image level, i.e., they use the same policy on the whole image. It inevitably ignores the diversity of different regions in an image, which limits the diversity of data increased by data augmentation, and sometimes causes the damage of critical semantic information. In contrast, our method takes diversity in local regions and contextual relationships into account to search for augmentation policies for multiple regions.
  Our proposed method is conceptually orthogonal to most region-based DA methods where DA transformations are performed in a non-automated way. For example, RandomErasing (Zhong et al., 2020), CutMix (Yun et al., 2019) perform cropping or replacement operation on a randomly selected rectangle region, Hard chips (Hong et al., 2019) extracts the object patches from an object pool and pastes them to an image to train the detector, and SECOND (Yan et al., 2018) performs DA by pasting the objects into the 3D point cloud. Some works further exploit class activation map (Zhou et al., 2016) or saliency map (Zhou et al., 2015) to select representative regions which are augmented by a randomly selected DA operation (e.g., SaliencyMix (Uddin et al., 2020), SnapMix (Huang et al., 2020)), AugMix (Hendrycks et al., 2019), KeepAugment (Gong et al., 2020) and TrivialAugment (Müller & Hutter, 2021)). Yet our proposed method automatically searches for the augmentation transformations based on the given datasets and tasks.
sections.heading: 2.2 MULTI-AGENT REINFORCEMENT LEARNING
sections.text: The most significant characteristic of MARL is the cooperation between agents (Tampuu et al., 2017; Ma & Cameron, 2008; Foerster et al., 2017) which is distinct from directly applying reinforcement learning (RL) algorithms to multi-agent systems. Due to the limited observation and action of a single agent, cooperation is necessary in the reinforced multi-agent system to achieve the common goal. Compared with independent agents, cooperative agents can improve the efficiency and robustness of the model (Neto, 2005; Zhang et al., 2019a; Busoniu et al., 2008). Many vision tasks use MARL to interact with the public environment to make decisions, with the goal of maximizing the expected total return of all agents, such as image segmentation (Liao et al., 2020; Han et al., 2019; Ma et al., 2020), image processing (Furuta et al., 2019).
sections.heading: 3 PROPOSED METHOD
sections.text: As above mentioned, we aim to search augmentation policies for the patches to explore more augmentation diversity. Learning the optimal DA policies often needs to interact with the target network, which aligns with the nature of RL (i.e., RL agents learn to make decisions through interacting with the environment (Sutton & Barto, 2018; Arulkumaran et al., 2017)). In addition, it is of great
  necessity to take the patch content and the contextual relationship between patches into account. Therefore, we use multi-agent reinforcement learning (MARL) algorithm. Specifically, the search for patch policies is based on the content of the patch together with the semantics of the image, and policies are encouraged to coordinate to achieve the joint optimal DA policy across the whole image. In this section, we first describe the preliminaries of MARL, then elaborate on our augmentation policy formulation and modelling. Furthermore, we summarize the framework of Patch AutoAugment.
sections.heading: 3.1 PRELIMINARIES OF MULTI-AGENT REINFORCEMENT LEARNING
sections.text: We first introduce the preliminaries of reinforcement learning (RL). RL models the decision-making problem as a Markov decision process (MDP) which is presented with a tuple (S,A, P,R, γ, T ). In RL framework, given the state s ∈ S, the agent takes an a ∈ A according to its policy π(a |s) : S × A → [0, 1] and then receives a reward r : S × A → R. The environment moves to the next state with a transition function denoted as P : S × A × S → [0, 1]. γ ∈ (0, 1] is a discount factor and T is a time horizon. The agent aims to maximize the long-term reward R over T steps to learn the optimal policy π∗.
  Furthermore, multi-agent reinforcement learning (MARL) considers a group of N agents, denoted as N , operating cooperatively in a shared environment towards a common goal. It can be formulated as a multi-agent MDP (MAMDP) (Boutilier, 1996) represented with a tuple (S, {Oi}Ni=1, {Ai}Ni=1, P,R, γ, T ). Here, S describes the shared state space and {O1, · · ·,ON} is a set of observations for agents. In the MARL configuration, each agent receives a private observation correlated with part of state, i.e., oi : S → Oi. According to the global state s and its observation oi, the agent takes its action ai ∈ Ai based on its policy πi(ai|oi, s), Ai is the action space for the i-th agent and A = {A1, · · ·,AN} denotes the joint action space. Then, the joint action a = a1× · · · ×aN produces next state according to transition function P : S ×A×S → [0, 1] and the environment gives the team reward r : S ×A → R to agents N . The objective of each agent is to maximize the total accumulative reward R to cooperatively learn the globally optimal policy π∗ = {π∗1, · · ·, π∗N} that consists of the optimal policy of each agent. In this paper, as mentioned before, we employ MARL to search optimal policy for each patch, and achieve the optimal effectiveness of DA across the entire image to further improve the performance of the target network as possible.
sections.heading: 3.2 PATCH AUTOAUGMENT
sections.text: In our proposed Patch AutoAugment (PAA), we formulate the task of policy search for the patches as a cooperative multi-agent decision-making problem and adopt multi-agent reinforcement learning (MARL) to solve it. In the following, We clarify the detailed formulation i) the state, observation and action modeling for the policy, ii) an effective team reward function design and iii) the detailed MARL algorithm for policy learning) of Patch AutoAugment.
  Policy Modeling. As illustrated in Figure 2, given the original input batch x and the corresponding label y (i.e., {x, y} = {xj , yj}bj=1 and b is the batch size), we divide an image xj into N equalsized and non-overlapping patches, denoted as xj = {Pij}Ni=1 whereP j j is the i-th patch of the image xj . In our proposed method, we aim to search the augmentation policy for each patch. Therefore, in MARL formulation, the augmentation policy of the patch Pi is controlled by an agent i ∈ N and we detail the state, observation and action for the augmentation policy as below.
  State. As aforementioned, the selection of augmentation operation for a patch is closely bound up with the contextual relationship between regions. Therefore, the augmentation policy needs to perceive the image semantics and we take the deep features of the whole image extracted by a backbone (e.g., ImageNet pre-trained ResNet-18 (He et al., 2016)) as the global state s which is visible to all agents. We analyze the impact of feature extractor initialization in Appendix F.
  Observation. Apart from capturing state (i.e., the global information), the agent only use their own observation (i.e., the local information) which is invisible to other agents. The augmentation policy seeks to choose augmentation operation based on the content of the patch and the observation is generally part of the state. Considering all these factors, we utilize the deep features of the i-th patch as observation oi, which are extracted by the same backbone as the state extractor.
  Action. The augmentation policy is responsible for choosing which transformations to apply from pre-defined operations. Following the previous automated DA methods (Cubuk et al., 2018; Lim et al., 2019), we define the fifteen operation functions (i.e., ShearX/Y, TranslateX/Y, Rotate, Invert, Equalize, Solarize, Posterize, Contrast, Color, Brightness, Sharpness, RandomErasing, Cutout, Mixup, Cutmix) to construct the action spaceA. Given the state s and the observation oi, according to policy πi(ai|oi, s), each agent i determines an action ai(·) ∈ A, which is the operation performed on the patch. Each operation is associated with two hyperparameters: probability p and magnitude m. In order to dramatically and effectively reduce the action space for augmentation policy, similar with (Cubuk et al., 2020; Ho et al., 2019; Lim et al., 2019), we take a fixed probability and magnitude schedule. Among them, the probability of applying the operation is sampled from the uniform distribution (i.e., pi ∼ U(0, 1). Following (Ho et al., 2019), we employ the same linear scale to be the magnitude schedule. In summary, the processed i-th patch in image xj is denoted as P̃ ij = ai(P i j) with the probability pi and the magnitude of ai(·) is mi, otherwise P̃ ij = P ij . Note that most operations are label-invariant, except Mixup (Zhang et al., 2017) and CutMix (Yun et al., 2019) are label-disturbing operations that combine different patches as well as their labels. We take Mixup as an example, then P̃ij = λPij + (1 − λ)Pit with probability pi, where Pit is a patch from another image xt, t 6= j, λ ∼ Beta(α, α), for α ∈ (0,∞), and the one-hot label is modified as ỹj = λ N yj+ (1−λ) N yt. Once all patches are processed by the corresponding operations chosen by the augmentation policies, we obtain the image augmented by our PAA, denoted as x̃j = {P̃1j , ···, P̃Nj }, and the final label ỹj . More operation details are shown in Appendix A.
  Reward Function. The reward function is of importance to guide the agents to learn so that they follow desired behaviors. The previous work, AdvAA (Zhang et al., 2019b), attempts to increase the training loss of the target network to generate harder augmentation policies and explore the weakness of the target network. Inspired by AdvAA, we reformulate the reward design appropriately under our configuration. In our proposed PAA, the common objective of all agents is to improve the performance of the mainstream target task through enhancing the benefits of DA. Therefore, we compare the feedback of target network φ(·) on the augmented data processed by our proposed PAA x̃ with the original data x and take their difference on the training losses as the reward for the policy in MARL, as in Eq. (1):
  r = l(φ(x̃), ỹ)− l(φ(x),y), (1)
  where x and y denote raw inputs and labels in supervision tasks. In our PAA model, all agents are encouraged to cooperate to achieve the common goal, thus we adopt the team reward function design (i.e., the shared reward mechanism in MARL) as Eq. (1) for all agents to make the joint augmentation policy achieve the optimal effectiveness.
  Policy Learning. Here, we introduce the training for the augmentation policies mentioned above. Considering that the action space is discrete, we adopt multi-agent Advantage Actor-Critic algorithm (Lowe et al., 2017) to learn the augmentation policies and encourage the coordination behaviors. In MARL, the framework of centralized training with decentralized execution (Foerster et al., 2018; Rashid et al., 2018) is generally adopted. More concretely, each agent i has an actor which is to learn discrete policy πi(ai|oi, s) and agents share a common critic which aims to estimate the value of global state V π(s). And we use the centralized critic to train decentralised actors. Here, we reformulate it appropriately for our task. We model the centralized action-value Q function that takes the actions of all agents in addition to state information s and outputs the Q-value for the team, formulated as:
  Qπ(s,a) = Eπ[Rt|s, a1, · · ·, aN ], (2)
  where a is the joint action of all agents a = {ai, · · ·, aN} and Rt = ∑T l=0 γ
  lrt+l is the long-term discounted reward. Then, the advantage function on the augmentation policy is given as follows:
  Aπ(s,a) = Qπ(s,a)− V π(s). (3)
  The critic network is a Multi-layer Perceptron (MLP). And we use ϕ to denote the its parameters. We take the square value of the advantage function Aπ as the loss function to update ϕ:
  L(ϕ) = (Aπ(s,a)) 2 . (4)
  The global policy network is a fully convolutional network (FCN), denoted as θ. Besides, to further achieve the ability to cooperate, similar to (Foerster et al., 2018; Rashid et al., 2018), the parameters of the actor networks of all agents are shared. The loss function for updating θ is defined as: L(θ) = − log πθ(a|s)Aπ(s,a). (5) Framework Summary. In this part, we summarize the overall framework of our proposed PAA. As shown in Figure 2, PAA first divides an image into a grid of patches. Then, we use a feature extractor to obtain the deep features of the whole image as the state. Each agent draws its individual observation, i.e., the deep features of the patch. According to the global state (i.e., the semantics of the entire image) and the local observation (i.e., the content of the patch), the actor networks output the augmentation operations of patches to further construct the joint operation map performed on the whole image. The augmented images are processed by our proposed PAA and then we input them to the target task network for parameters updating. Moreover, the feedback of the target network is used as the team reward signal to update the policy network.
sections.heading: 4 EXPERIMENTS
sections.heading: 4.1 EXPERIMENT OVERVIEW
sections.text: In this section, to study the effectiveness of Patch AutoAugment (PAA), we conduct experiment on image classification, fine-grained image recognition and local tasks where local information is of importance, e.g., object detection. Specifically, we focus on CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009) datasets as well as three fine-grained object recognition datasets, i.e., CUB-200-2011 (Wah et al., 2011), Stanford Cars (Krause et al., 2013), FGVCAircraft (Maji et al., 2013) and Pascal VOC 2007 (Everingham & Winn, 2009). We describe the datasets in detail in Appendix B. We compare PAA with baseline pre-processing, Cutout (DeVries & Taylor, 2017), Mixup (Zhang et al., 2017), Cutmix (Yun et al., 2019), Co-Mixup (Co-Mix) (Kim et al., 2021), AutoAugment (AA) (Cubuk et al., 2018), Fast AutoAugment (FastAA) (Lim et al., 2019), RandAugment (RA) (Cubuk et al., 2020), DADA (Li et al., 2020) and Adversarial AA (AdvAA). The baseline follows (Zoph et al., 2018; Yamada et al., 2018; Gastaldi, 2017): standardizing the data, horizontally flipping with 0.5 probability, zero-padding and random cropping. Our policy model architecture design mainly follows (Chu et al., 2016), and we make some adjustments according to our task scenario. More details about our policy network architectures and the hyperparameters of the target model and our policy model are supplied in Appendix C and D, respectively. Moreover, we default the number of patches N to 16 except for CIFAR tasks N = 4. To ensure the reliability of our experiments, we run each experiment four times using different random seeds.
sections.heading: 4.2 RESULTS AND ANALYSIS
sections.text: Classification Results on CIFAR-10 and CIFAR-100. For CIFAR-10 and CIFAR-100, we examine on Wide-ResNet-28-10 (WRN-28-10) (Zagoruyko & Komodakis, 2016), Shake-Shake (SS) (Gastaldi, 2017) and Pyramid-Net+ShakeDrop (Pyramid+SD) (Han et al., 2017; Yamada et al., 2018) models. The results are reported in Table 1, which shows the proposed approach consistently outperforms several state-of-the-art DA methods. We observe that the improvement of performance is relatively slight, due to the small image size of CIFAR which is 32 × 32. In the following, we further apply our proposed PAA on datasets with larger image sizes and other networks.
  Classification Results on ImageNet. As shown in Table 2, we evaluate our method on ResNet-50 and ResNet-200 (He et al., 2016) backbone on ImageNet, and our PAA significantly improves the performance of the target networks. The results further demonstrate that our proposed method is an effective DA technique for consistent and expressive benefits for datasets with larger image sizes.
  Effectiveness of Fine-grained Classification. Furthermore, we evaluate our proposed method on fine-grained image recognition tasks. According to previous work (Du et al., 2020; Chen et al., 2019), we take ResNet-50 and ResNet-101 as the backbones. The results are shown in Table 4, which illustrates that the performance of PAA is consistently better than other methods and PAA has achieved remarkable performance on these challenging fine-grained tasks.
  Effectiveness of Local Tasks. In order to further verify the effectiveness of our proposed method, we conduct experiments on the object detection task. We adopt the mainstream detector Faster RCNN (Ren et al., 2015) (ResNet-50) on the dataset Pascal VOC 2007 (Everingham & Winn, 2009) and show the mAP results on the Table 5. We observe that the proposed PAA achieves consistent improvements and PAA is indeed effective for the local tasks like object detection.
sections.heading: 4.3 COMPLEXITY ANALYSIS
sections.text: In this section, in order to further demonstrate the performance of PAA in terms of complexity, we compare the policy search time and training time of PAA with AA (Cubuk et al., 2018), FastAA (Lim et al., 2019) and AdvAA (Zhang et al., 2019b), as illustrated in Table 3. As shown in Table 3, compared to the previous works, PAA requires the fewest total computational resources, and the search time is almost negligible. As for the parameters, the total number of PAA model parameters (about 0.23M) is less than 1% of the target network (e.g., ResNet50: about 25.5M).
  In summary, the main methods to reduce the computational cost lie in three aspects: 1) The augmentation policy network is jointly optimized with the target network, similar to (Zhang et al., 2019b; Lin et al., 2019). Namely, our proposed method searches for policies in an online manner, obviating thousands of policies validation on a small proxy network and the requirement for retraining the target network. Besides, we use fixed schedules for the two corresponding hyperparameters (i.e., probability and magnitude) of each transformation to effectively reduce the search space, which makes it easier to search for effective policies. By these means, PAA compresses most of the policy search time 2) As mentioned before, a MARL algorithm is adopted, in which all agents parallelly learn the augmentation policies, to reduce the training time. 3) Compared with the previous DA methods using image-by-image sequential transformations, we perform parallel transformations on tensor. Specifically, we pick the patches in a batch performing the same operation to reconstruct
  a new tensor. And we use Kornia1 to realize tensor transformations on GPU to further reduce the processing time which is included in the training time.
sections.heading: 4.4 ABLATION STUDY
sections.text: In this section, we study the effectiveness of MARL and also discuss the design of patch numbers, through several ablation experiments.
  Performance of random policies. We randomize the augmentation policy for each patch, dubbed Patch RandAugment and compare it with our proposed PAA. To be specific, the augmentation policy is randomly selected from predefined transformations with uniform probability. As shown in Table 6, Patch RandAugment (PRA) leads to considerable appreciable improvements. However, our Patch AutoAugment with meaningful guidance significantly surpasses the random patch policies.
  Performance of the policies searched using single-agent RL. Furthermore, we directly use the same single-agent RL to search for each patch’s policy, termed as Patch SingleAutoAugment (PSAA) and compare it to our PAA, where the search task is a cooperative multi-agent problem. In short, PSAA ignores the contextual relationship between patches, where the agents are independent and patches are non-cooperative. The results (see Table 6) indicate that taking the contextual information into account with cooperative RL-agents has improved the joint effectiveness of DA.
  Impact of the number of patches. We explore the effect of the number of patches on the target model performance. As shown in Table 7, we respectively set the number of patches N = {4, 16, 64, 256, 1024}. The results show that on ImageNet / CUB-200-2011, when patch numbers N increase, the performance accuracy first increases and then decreases. In addition, when N = 16, PAA achieves the best performance. We analyze that the small number of patches may cause PAA to be unable to effectively explore local diversity, and its advantages would be limited. In particular, when N = 1, PAA degenerates into image-wise automated DA. In contrast, under too larger values (e.g., the extreme case is to search for DA policy for each pixel), the local semantic consistency is broken and the benefit brought by the consideration of contextual relationship between patches is gradually overtaken.
sections.heading: 4.5 VISUALIZATION
sections.text: Policy Visualization. Grad-CAM (Selvaraju et al., 2017) is used to localize the important regions in the image. Therefore we can calculate the importance score of each patch, and we divide the importance scores into four bins, i.e., very important, important, normal and not important. Then, we categorize patches into four groups and draw four stacked area charts showing the percentages of operations selected by PAA augmentation policies over time. We take ResNet-50 backbone trained on CUB-200-2011 as an example. Since our proposed PAA searches for patch policies in an online manner, the strategies change dynamically over time as shown in Figure 4. At the beginning of the training process, the selected actions are messy since the MARL network is in the exploratory stage. At the tail end of the training, the target network has converged, causing the percentage of all operations to be almost the same and the percentage to flatten out.
  In addition, we have some interesting findings which may provide some insights to the DA community. i) The optimal augmentation strategies of patches vary by their importance levels. Therefore, it is necessary to take the image content into account when performing data augmentation. ii) In
  1Kornia (Riba et al., 2020) is a differentiable computer vision library for PyTorch. We use it to accelerate augmentation operation on tensors.
  Table 7: Ablation study: performance comparisons (Top-1 accuracy (%)) under five different patch numbers N on CIFAR-10 (image size: 32 × 32) with WRN-28-10 and on ImageNet (image size: 224×224) together with CUB-200-2011 (CUB) (image size: 448×448) with ResNet-50 backbone.
  N 4 16 64 256 1024 CIFAR-10 97.5 97.3 97.2 97.1 -
  CUB 87.2 87.5 87.3 87.1 86.8
  Baseline AutoAugment Ours
  Figure 3: Grad-CAM visualization on the examples from CUB with ResNet-50 trained by baseline, AutoAugment and our PAA. PAA tends to help the target network focus on task-related features.
  O pe
  ra tio
  n Pe
  rc en
  ta ge
  (% )
  O pe
  ra tio
  n Pe
  rc en
  ta ge
  (% )
  brightness (c) Normal
  cutmix color
  invert
  Epochs (a) Very Important
  Epochs (b) Important
  Epochs Epochs
  100
  80
  60
  40
  20
  0
  100
  80
  60
  40
  20
  0
  100
  80
  60
  40
  20
  0
  100
  80
  60
  40
  20
  0 0 20 40 60 80 100 120 140 160 180 2000 20 40 60 80 100 120 140 160 180 200
  0 20 40 60 80 100 120 140 160 180 200 0 20 40 60 80 100 120 140 160 180 200
  contrast (d) Not Important
  equalizemixup shear randomerasing translate cutout sharpnessrotation posterize sloarize
  Figure 4: We categorize patches into four groups according to the patch importance calculated through Grad-CAM. We draw four stacked area chart of the operations’ percentages over time. The area represents the percentage of each operation.
  the middle of the training process, different types of patches prefer to select different augmentation operations. Concretely, as illustrated in Figure 4, for the important patches, color transformation is mostly picked. For the unimportant patches, RandomErasing and Cutout are usually chosen by PAA. Important patches commonly take along semantic information that is related to mainstream tasks. It is better to choose the mild transformations (e.g., color) for them, which can effectively protect the semantic information from being damaged. In contrast, unimportant patches typically carry unexpected features (Singla et al., 2021) which are causally unrelated to the desired class. Severe transformations (e.g., Cutout, RandomErasing) could be chosen for them, which introduce noise and disturbance to reduce the impact of unexpected features on the target network learning.
  Grad-CAM Visualization. Here, we adopt Grad-CAM (Selvaraju et al., 2017) to visualize the learned features to intuitively show the impact of PAA, as shown in Figure 3. We take the ResNet-50 as the backbone which is trained with dataset processed by 1) Baseline 2) AutoAugment and 3) Patch AutoAugment, respectively. We observe that the model trained with PAA focus on more task-related areas rather than spurious correlations (e.g., the branch where the bird stands) or overemphasized features (e.g., birds’ claws). More visualization results are provided in the Appendix E.
sections.heading: 5 CONCLUSION
sections.text: In this paper, we propose Patch AutoAugment (PAA), a more fine-grained automated data augmentation approach. Our method adopts multi-agent reinforcement learning to automatically search for the optimal augmentation policies for patches, and encourages agents to cooperate with each other to further achieve the joint optimal policy across the entire image. Extensive experiments demonstrate that PAA improves the target network performance with low computational cost in various tasks. Meanwhile, we use visualization to show that PAA is beneficial for the target network to localize more class-related cues. Furthermore, we hope that our visual observations of policies will be useful for future development. In future work, we will investigate different schemes on dividing different regions. Additionally, our method is naturally aligned with the patch token mechanism in the current vision transformers (Dosovitskiy et al., 2020; Touvron et al., 2021; Liu et al., 2021b) and the data augmentation specific to vision transformers has not been extensively studied. Therefore, we leave the automated data augmentation for vision transformers to future work, which may provide some interesting insights to the community.
sections.heading: A OPERATIONS DETAILS
sections.text: Following (Cubuk et al., 2018; Ho et al., 2019; Lim et al., 2019; Zhang et al., 2019b), we define the fifteen common augmentation operations to form the action space. Here, we detail the description of these operations, as illustrated in Table 8. In addition, we give magnitudes range of augmentation operations corresponding to hyperparameters of the functions in the Kornia PyTorch library. Some operations, such as Cutout and CutMix, have no parameters.
  Additionally, in our implementation, we pick out the patches that perform the same operation and put them into a new tensor. Then, we speed up the process by performing parallel transformations on the tensor. Tensor transformations on GPU can be realized by Kornia (Riba et al., 2020) to reduce the computational costs. In particular, when performing the label-disturbing operations (i.e., Mixup and CutMix), a patch needs to mix with another patch that is randomly selected from the new tensor. Namely, a patch is mixed with another patch that performs the same operation.
sections.heading: B DATASETS
sections.text: We evaluate Patch AutoAugment (PAA) on the following datasets: CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), ImageNet (Deng et al., 2009) and three fine-grained
  image recognition datasets (CUB-200-2011 (Wah et al., 2011), Stanford Cars (Krause et al., 2013), FGVC-Aircraft (Maji et al., 2013)) and Pascal VOC (Everingham & Winn, 2009).
  To be specific, both CIFAR-10 and CIFAR-100 have 50,000 training examples. Each image of size 32 × 32 belongs to one of 10 categories. ImageNet dataset has about 1.2 million training images and 50,000 validation images with 1000 classes. Original ImageNet data have different sizes and we resize them to 224 × 224. In addition, we evaluate the performance of our proposed PAA on three standard fine-grained object recognition datasets. CUB-200-2011 (Wah et al., 2011) consists of 6,000 train and 5,800 test bird images distributed in 200 categories. Stanford Cars (Krause et al., 2013) contains 16,185 images in 196 classes. The FGVC-Aircraft dataset contains 10,200 images of aircraft, with 100 images for each of 102 different aircraft model variants, most of which are airplanes. The image size in the above three datasets is 448× 448. The Pascal VOC dataset consists of a set of images and each image has an annotation file giving a bounding box and object class label for each object in one of the twenty classes present in the image. The data has been split into 50% for training and 50% for testing. In total there are 9,963 images, containing 24,640 annotated objects.
sections.heading: C MODEL ARCHITECTURE
sections.text: Here, we provide the detailed model architecture for each component in our PAA augmentation policy model, including feature extractor network, actor network and critic network. We use the pretrained on ImageNet ResNet-18 backbone (excluding the final avgpool and softmax layer) to extract the deep features of the image and the patch, which are denoted as the state s and the observation oi respectively. As for the global policy network and the critic network, the detailed model architectures are shown in Table 9. The policy network is a fully convolutional neural network (FCN). The parameters of the actor networks of all agents are shared. Although they share parameters, each agent’s policy is not identical. When determining an action, each agent’s policy relies on different activations of the global policy network.
sections.heading: D HYPERPARAMETERS
sections.text: We detail various target models hyperparameters (e.g., batch size, learning rate and training epochs) on CIFAR-10, CIFAR-100, ImageNet, CUB-200-2011, Stanford Cars and FGVC-Aircraft in Table 10. We do not specifically tune these hyperparameters, and all of these are consistent with previous works (Cubuk et al., 2018; Lim et al., 2019; Dabouei et al., 2021; Du et al., 2020; Chen et al., 2019).
  In addition, the time horizon T determines the times of augmentations performed sequentially on a patch. Therefore the agent can output more than one augmentation for a patch by setting T > 1. In our PAA model, we set time horizon T = 1, i.e., the augmentation policies take actions at every time step for saving computational cost and avoiding over-regularization. Even in the single-step decision-making process (i.e., T = 1), RL can still effectively search for the optimal policy as illustrated in (Mnih et al., 2016; Singh et al., 2000). In addition, we use the SGD optimizer with an initial learning rate of 1e-4 to train the actor network and the critic network.
sections.heading: E MORE GRAD-CAM VISUALIZATION
sections.text: In this section, we provide more Grad-CAM (Selvaraju et al., 2017) results of ResNet-50 models trained using baseline augmented data, AA (Cubuk et al., 2018) and our proposed PAA, respectively, as shown in Figure 5. The visualization results demonstrate that our proposed PAA improves the localization ability of the target network and tends to help the target network focus on more parts of the foreground object. In short, our proposed PAA make the target network focus on the important or representative regions closely related to the class within an image.
  F IMPACT OF THE INITIALIZATION OF THE FEATURE EXTRACTOR
  We further study the impact of the initialization of the feature extractor as shown in Table 11. We first observe that the performance of the Non-pretrained ResNet-18 scheme (using non-pretrained ResNet-18 as the state/observation extractor) is still much better than the baseline. In other words, our agents do not rely on the pre-trained ResNet to produce features to be used as the state or observation. We also notice that the performance of the Non-pretrained ResNet-18 scheme is slightly lower than the Pretrained ResNet-18. This may be because that at the very early stage of the training, the features extracted by Non-pretrained ResNet-18 are not as good as Pretrained ResNet-18 in practice.
sections.heading: G DETAILS OF COMPUTATIONAL COST
sections.text: The computational cost of our proposed PAA is estimated on GeForce GTX 1080 Ti while AA (Cubuk et al., 2018) and AdvAA (Zhang et al., 2019b) are on NVIDIA Tesla P100. Besides, FastAA (Lim et al., 2019) is estimated on NVIDIA Tesla V100. The computational cost is mainly used for searching policies and training the target networks.