summaries: This paper target the task of automatically determining the best augmentation method to obtain improved accuracy. While the previous related studies focus on the image-level augmentation and ignore the semantic information of the augmented images, the proposed algorithm augments the grid-wise patches of the given input with the preserved semantic information. To overcome the enlarged number of combinations to consider all the patches, the algorithm utilizes the MARL algorithm with the unified reward function. By MARL algorithm, the number of parameters can be reduced and the training speed can be much improved, compared to the previous auto-augmentation methods. Through the image classification and fine-grained image recognition tasks, the proposed algorithm was validated, and it shows the state-of-the-art performance among the compared methods.

summaries: This paper proposes an automatic data augmentation approach. Different from existing works, they proposed to augment patches in the image rather than the whole image. The approach is formulated as a multi-agent reinforcement learning problem. They empirically show the effectiveness of their approach across several image classification datasets.

summaries: This paper proposed a fine-grained automated data augmentation approach, Patch AutoAugment (PAA), which tries to increase diversity in local regions by divide an image into a grid of patches and search for the joint optimal augmentation policies for the patches. The proposed PAA considers the task as a multi-agent reinforcement learning problem,  and adopt a multi-agent reinforcement learning algorithm to automatically search for the optimal augmentation policies by considering the contextual relationship between the patches. They verify the proposed method on many classification and fine-grained recognition dataset(CIFAR10, CIFAR-100, ImageNet, CUB-200-2011, Stanford Cars and FGVC-Aircraft). The experiments show a good result and visualization results provide some insights that the PAA  help the target network to localize more class-related cues.

summaries: The paper proposes an evolution of the traditional   pipeline of image data augmentation used to reduce ML   model overfitting.    Instead of applying transformations such as shear,   rotate, CutOut, etc. at the image level, the proposed   technique divides the images into a fixed grid and   applies a potentially different transformation to each   cell. The problem of selecting a transform for each cell   is cast as a multi-agent RL (MARL) task, and the agents   learn as the main network trains within a (multi-agent)   Advantage Actor Critic framework.    The agents use a shared reward mechanism, with the reward   defined as the difference between the loss on the   augmented sample and the original loss.    The regular grid is fixed for a dataset and   hyperparameters like the magnitude of the augmentations   follow a fixed schedule; the agents only pick _which_   augmentation to apply on a patch.    Experiments performed on CIFAR-{10,100}, ImageNet,   CUB-200-2011, Stanford Cars, and FGVC-Aircraft show   relatively small but very consistent improvements in   terms of image classification accuracy. Different design   choices (MARL vs. single-agent vs. random, grid size,   etc.) are ablated and discussed in detail.

abstractText: Data augmentation (DA) plays a critical role in improving the generalization of deep learning models. Recent works on automatically searching for DA policies from data have achieved great success. However, existing automated DA methods generally perform the search at the image level, which limits the exploration of diversity in local regions. In this paper, we propose a more fine-grained automated DA approach, dubbed Patch AutoAugment, to divide an image into a grid of patches and search for the joint optimal augmentation policies for the patches. We formulate it as a multi-agent reinforcement learning (MARL) problem, where each agent learns an augmentation policy for each patch based on its content together with the semantics of the whole image. The agents cooperate with each other to achieve the optimal augmentation effect of the entire image by sharing a team reward. We show the effectiveness of our method on multiple benchmark datasets of image classification, fine-grained image recognition and object detection (e.g., CIFAR-10, CIFAR-100, ImageNet, CUB-200-2011, Stanford Cars, FGVC-Aircraft and Pascal VOC 2007). Extensive experiments demonstrate that our method outperforms the state-of-the-art DA methods while requiring fewer computational resources.

authors:
Kai Arulkumaran
Marc Peter Deisenroth
Miles Brundage
Anil Anthony Bharath

title: Deep reinforcement learning: A brief survey

venue: IEEE Signal Processing Magazine,

title: Planning, learning and coordination in multiagent decision processes

venue: In TARK,

authors:
Lucian Busoniu
Robert Babuska
Bart De Schutter

title: A comprehensive survey of multiagent reinforcement learning

venue: IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews),

authors:
Yue Chen
Yalong Bai
Wei Zhang
Tao Mei

title: Destruction and construction learning for finegrained image recognition

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Tianshu Chu
Shuhui Qu
Jie Wang

title: Large-scale multi-agent reinforcement learning using image-based state representation

venue: In 2016 IEEE 55th Conference on Decision and Control (CDC),

authors:
Ekin D Cubuk
Barret Zoph
Dandelion Mane
Vijay Vasudevan
Quoc V Le

title: Autoaugment: Learning augmentation policies from data

venue: arXiv preprint arXiv:1805.09501,

authors:
Ekin D Cubuk
Barret Zoph
Jonathon Shlens
Quoc V Le

title: Randaugment: Practical automated data augmentation with a reduced search space

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,

authors:
Ali Dabouei
Sobhan Soleymani
Fariborz Taherkhani
Nasser M Nasrabadi

title: Supermix: Supervising the mixing data augmentation

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: Imagenet: A large-scale hierarchical image database

venue: IEEE conference on computer vision and pattern recognition,

authors:
Terrance DeVries
Graham W Taylor

title: Improved regularization of convolutional neural networks with cutout

venue: arXiv preprint arXiv:1708.04552,

authors:
Alexey Dosovitskiy
Lucas Beyer
Alexander Kolesnikov
Dirk Weissenborn
Xiaohua Zhai
Thomas Unterthiner
Mostafa Dehghani
Matthias Minderer
Georg Heigold
Sylvain Gelly

title: An image is worth 16x16 words: Transformers for image recognition at scale

venue: arXiv preprint arXiv:2010.11929,

authors:
Ruoyi Du
Dongliang Chang
Ayan Kumar Bhunia
Jiyang Xie
Zhanyu Ma
Yi-Zhe Song
Jun Guo

title: Fine-grained visual classification via progressive multi-granularity training of jigsaw patches

venue: In European Conference on Computer Vision,

authors:
Mark Everingham
John Winn

title: The pascal visual object classes challenge 2007 (voc2007) development kit

authors:
Jakob Foerster
Gregory Farquhar
Triantafyllos Afouras
Nantas Nardelli
Shimon Whiteson

title: Counterfactual multi-agent policy gradients

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Jakob N Foerster
Richard Y Chen
Maruan Al-Shedivat
Shimon Whiteson
Pieter Abbeel
Igor Mordatch

title: Learning with opponent-learning awareness

venue: arXiv preprint arXiv:1709.04326,

authors:
Ryosuke Furuta
Naoto Inoue
Toshihiko Yamasaki

title: Pixelrl: Fully convolutional network with reinforcement learning for image processing

venue: IEEE Transactions on Multimedia,

title: Shake-shake regularization

venue: arXiv preprint arXiv:1705.07485,

authors:
Chengyue Gong
Dilin Wang
Meng Li
Vikas Chandra
Qiang Liu

title: Keepaugment: A simple information-preserving data augmentation approach

venue: arXiv preprint arXiv:2011.11778,

authors:
Raphael Gontijo-Lopes
Sylvia J Smullin
Ekin D Cubuk
Ethan Dyer

title: Affinity and diversity: Quantifying mechanisms of data augmentation

venue: arXiv preprint arXiv:2002.08973,

authors:
Dongyoon Han
Jiwhan Kim
Junmo Kim

title: Deep pyramidal residual networks

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Lei Han
Peng Sun
Yali Du
Jiechao Xiong
Qing Wang
Xinghai Sun
Han Liu
Tong Zhang

title: Grid-wise control for multi-agent reinforcement learning in video game ai

venue: In International Conference on Machine Learning,

authors:
Ryuichiro Hataya
Jan Zdenek
Kazuki Yoshizoe
Hideki Nakayama

title: Faster autoaugment: Learning augmentation strategies using backpropagation

venue: arXiv preprint arXiv:1911.06987,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Dan Hendrycks
Norman Mu
Ekin D Cubuk
Barret Zoph
Justin Gilmer
Balaji Lakshminarayanan

title: Augmix: A simple data processing method to improve robustness and uncertainty

authors:
Daniel Ho
Eric Liang
Xi Chen
Ion Stoica
Pieter Abbeel

title: Population based augmentation: Efficient learning of augmentation policy schedules

authors:
Sungeun Hong
Sungil Kang
Donghyeon Cho

title: Patch-level augmentation for object detection in aerial images

venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops,

authors:
Shaoli Huang
Xinchao Wang
Dacheng Tao

title: Snapmix: Semantically proportional mixing for augmenting fine-grained data

venue: arXiv preprint arXiv:2012.04846,

authors:
Jang-Hyun Kim
Wonho Choo
Hosan Jeong
Hyun Oh Song

title: Co-mixup: Saliency guided joint mixup with supermodular diversity

venue: arXiv preprint arXiv:2102.03065,

authors:
Jonathan Krause
Michael Stark
Jia Deng
Li Fei-Fei

title: 3d object representations for fine-grained categorization

venue: In Proceedings of the IEEE international conference on computer vision workshops,

authors:
Alex Krizhevsky
Geoffrey Hinton

title: Learning multiple layers of features from tiny images

authors:
Yonggang Li
Guosheng Hu
Yongtao Wang
Timothy Hospedales
Neil M Robertson
Yongxin Yang

title: Dada: Differentiable automatic data augmentation

venue: arXiv preprint arXiv:2003.03780,

authors:
Xuan Liao
Wenhao Li
Qisen Xu
Xiangfeng Wang
Bo Jin
Xiaoyun Zhang
Yanfeng Wang
Ya Zhang

title: Iteratively-refined interactive 3d medical image segmentation with multi-agent reinforcement learning

authors:
Sungbin Lim
Ildoo Kim
Taesup Kim
Chiheon Kim
Sungwoong Kim

title: Fast autoaugment

venue: In Advances in Neural Information Processing Systems,

authors:
Chen Lin
Minghao Guo
Chuming Li
Xin Yuan
Wei Wu
Junjie Yan
Dahua Lin
Wanli Ouyang

title: Online hyper-parameter learning for auto-augmentation strategy

venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,

authors:
Aoming Liu
Zehao Huang
Zhiwu Huang
Naiyan Wang

title: Direct differentiable augmentation search

venue: arXiv preprint arXiv:2104.04282,

authors:
Ze Liu
Yutong Lin
Yue Cao
Han Hu
Yixuan Wei
Zheng Zhang
Stephen Lin
Baining Guo

title: Swin transformer: Hierarchical vision transformer using shifted windows

venue: arXiv preprint arXiv:2103.14030,

authors:
Ryan Lowe
Yi Wu
Aviv Tamar
Jean Harb
Pieter Abbeel
Igor Mordatch

title: Multi-agent actorcritic for mixed cooperative-competitive environments

venue: arXiv preprint arXiv:1706.02275,

authors:
Chaofan Ma
Qisen Xu
Xiangfeng Wang
Bo Jin
Xiaoyun Zhang
Yanfeng Wang
Ya Zhang

title: Boundary-aware supervoxel-level iteratively refined interactive 3d image segmentation with multi-agent reinforcement learning

venue: IEEE Transactions on Medical Imaging,

authors:
Jie Ma
Stephen Cameron

title: Combining policy search with planning in multi-agent cooperation

venue: In Robot Soccer World Cup,

authors:
Subhransu Maji
Esa Rahtu
Juho Kannala
Matthew Blaschko
Andrea Vedaldi

title: Fine-grained visual classification of aircraft

venue: arXiv preprint arXiv:1306.5151,

authors:
Volodymyr Mnih
Adria Puigdomenech Badia
Mehdi Mirza
Alex Graves
Timothy Lillicrap
Tim Harley
David Silver
Koray Kavukcuoglu

title: Asynchronous methods for deep reinforcement learning

venue: In International conference on machine learning,

authors:
Samuel G Müller
Frank Hutter

title: Trivialaugment: Tuning-free yet state-of-the-art data augmentation

venue: arXiv preprint arXiv:2103.10158,

title: From single-agent to multi-agent reinforcement learning: Foundational concepts and methods

venue: Learning theory course,

authors:
Tabish Rashid
Mikayel Samvelyan
Christian Schroeder
Gregory Farquhar
Jakob Foerster
Shimon Whiteson

title: Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning

authors:
Shaoqing Ren
Kaiming He
Ross Girshick
Jian Sun

title: Faster r-cnn: Towards real-time object detection with region proposal networks

venue: Advances in neural information processing systems,

authors:
Edgar Riba
Dmytro Mishkin
Daniel Ponsa
Ethan Rublee
Gary Bradski

title: Kornia: an open source differentiable computer vision library for pytorch

venue: In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,

authors:
Ramprasaath R Selvaraju
Michael Cogswell
Abhishek Das
Ramakrishna Vedantam
Devi Parikh
Dhruv Batra

title: Grad-cam: Visual explanations from deep networks via gradient-based localization

venue: In Proceedings of the IEEE international conference on computer vision,

authors:
Connor Shorten
Taghi M Khoshgoftaar

title: A survey on image data augmentation for deep learning

venue: Journal of Big Data,

authors:
Ashish Shrivastava
Tomas Pfister
Oncel Tuzel
Joshua Susskind
Wenda Wang
Russell Webb

title: Learning from simulated and unsupervised images through adversarial training

authors:
Satinder Singh
Tommi Jaakkola
Michael L Littman
Csaba Szepesvári

title: Convergence results for single-step on-policy reinforcement-learning algorithms

venue: Machine learning,

authors:
Sahil Singla
Besmira Nushi
Shital Shah
Ece Kamar
Eric Horvitz

title: Understanding failures of deep networks via robust feature extraction

authors:
Richard S Sutton
Andrew G Barto

title: Reinforcement learning: An introduction

venue: MIT press,

authors:
Ardi Tampuu
Tambet Matiisen
Dorian Kodelja
Ilya Kuzovkin
Kristjan Korjus
Juhan Aru
Jaan Aru
Raul Vicente

title: Multiagent cooperation and competition with deep reinforcement learning

venue: PloS one,

authors:
Hugo Touvron
Matthieu Cord
Matthijs Douze
Francisco Massa
Alexandre Sablayrolles
Hervé Jégou

title: Training data-efficient image transformers & distillation through attention

authors:
Toan Tran
Trung Pham
Gustavo Carneiro
Lyle Palmer
Ian Reid

title: A bayesian data augmentation approach for learning deep models

venue: arXiv preprint arXiv:1710.10564,

authors:
AFM Uddin
Mst Monira
Wheemyung Shin
TaeChoong Chung
Sung-Ho Bae

title: Saliencymix: A saliency guided data augmentation strategy for better regularization

venue: arXiv preprint arXiv:2006.01791,

authors:
Sangdoo Yun
Dongyoon Han
Seong Joon Oh
Sanghyuk Chun
Junsuk Choe
Youngjoon Yoo

title: Cutmix: Regularization strategy to train strong classifiers with localizable features

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
Sergey Zagoruyko
Nikos Komodakis

title: Wide residual networks

venue: arXiv preprint arXiv:1605.07146,

authors:
Hongyi Zhang
Moustapha Cisse
Yann N Dauphin
David Lopez-Paz

title: mixup: Beyond empirical risk minimization

venue: arXiv preprint arXiv:1710.09412,

authors:
Kaiqing Zhang
Zhuoran Yang
Tamer Başar

title: Multi-agent reinforcement learning: A selective overview of theories and algorithms

venue: arXiv preprint arXiv:1911.10635,

authors:
Xinyu Zhang
Qiang Wang
Jian Zhang
Zhao Zhong

title: Adversarial autoaugment

venue: arXiv preprint arXiv:1912.11188,

authors:
Zhun Zhong
Liang Zheng
Guoliang Kang
Shaozi Li
Yi Yang

title: Random erasing data augmentation

venue: In AAAI,

authors:
Bolei Zhou
Aditya Khosla
Agata Lapedriza
Aude Oliva
Antonio Torralba

title: Learning deep features for discriminative localization

authors:
Li Zhou
Zhaohui Yang
Qing Yuan
Zongtan Zhou
Dewen Hu

title: Salient region detection via integrating diffusion-based compactness and local contrast

venue: IEEE Transactions on Image Processing,

authors:
Barret Zoph
Vijay Vasudevan
Jonathon Shlens
Quoc V Le

title: Learning transferable architectures for scalable image recognition

text:
1 INTRODUCTION
Data Augmentation (DA) has been widely used to alleviate the overfitting risk in training deep neural networks by appropriately enriching the diversity of training data (Shorten & Khoshgoftaar, 2019; Gontijo-Lopes et al., 2020). Notable DA methods improve the performance and robustness of the neural networks, such as rotation, Mixup (Zhang et al., 2017) and Cutmix (Yun et al., 2019). But these approaches are typically handcraft and require human prior knowledge, which causes weak transferability of DA across different datasets. To relieve the dependence on manual design and further explore more adaptive augmentation, AutoAugment (AA) (Cubuk et al., 2018), as a new DA paradigm, is proposed to automate the search of the optimal DA policies (i.e., DA operation, probability and magnitude) from the training dataset. To be specific, AA trains a proxy model with the augmentation policy generated by a controller, which is updated through reinforcement
learning using validation accuracy as the reward signal. In spite of the superior performance of AA, its optimization procedure is computationally intensive due to the need to evaluate thousands of policies. Therefore, to reduce the search costs, multiple automated DA approaches (Lim et al., 2019; Ho et al., 2019; Hataya et al., 2019; Zhang et al., 2019b; Lin et al., 2019) have been proposed. For example, (Lim et al., 2019) employs density matching as a search method to accelerate the policy search, and (Zhang et al., 2019b) introduces adversarial learning to organize the target network training and augmentation policies search in an online manner.
Yet the aforementioned automated DA methods all search for policies at the image level. They ignore the exploration of diversity in local regions, which may result in insufficient diversity of dataset and limit the benefits of DA (Gontijo-Lopes et al., 2020). In addition, due to this coarse-grained augmentation, they may damage key semantic information and introduce ambiguity into the training process (see a simple example in Figure 1 row 2, column 2). With those in mind, it is necessary to automatically search for the optimal augmentation policies for different regions by taking regional diversity into account. One straightforward idea is to directly apply image-wise automated DA methods in different regions. But such an intuitive solution ignores the contextual relationship between regions, which may lead to the non-globally optimal effectiveness of DA policies across the entire image. Besides, it may encounter an extremely high computational cost due to the need of optimizing multiple policies for regions respectively.
In this paper, we propose a new approach, named Patch AutoAugment (PAA) (see the last row of Figure 1), to address the above-mentioned problems. We first divide an image into a grid of patches to increase the flexibility of representations for different regions and take “patch” as the basic control unit. Then, we model the search for the augmentation policies of patches as a fully cooperative multiagent task, and we leverage a multi-agent reinforcement learning (MARL) algorithm where agents cooperatively learn the policies. Specifically, based on the content of each patch and the semantics of the entire image, the agent searches for a policy in terms of choosing which transformation to apply out of the pre-defined DA operations. To encourage our policy networks to adaptively learn the beneficial augmentation policies for the target network, we use the feedback of the target network as the team reward signal to guide the policy networks to learn on the fly. All agents wind up benefiting from two mechanisms (i.e., parameter sharing and centralized training with decentralized execution) in MARL. In this way, all agents collaboratively and parallelly learn policies to further achieve the joint optimal DA policy across the whole image and alleviate the computational cost.
In summary, our contributions can be summarized as follows: 1) We pinpoint that exploring diversity in local regions is important for automated learned DA approaches. To our best knowledge, we are the first to propose a more fine-grained automated DA approach, that searches for the optimal policies for the patches according to the content of the patch and the semantics of the entire image. 2) To further achieve the joint optimal policy across the image, we model the DA policy search of patches as a fully cooperative multi-agent task, and adopt a multi-agent reinforcement learning algorithm for Patch AutoAugment by considering the contextual relationship between the patches. 3) Our visualization results provide some insights to the DA community on which augmentation operation should be chosen for patches with different content during the whole training process.

heading: 2 RELATED WORK

heading: 2.1 DATA AUGMENTATION

text:
Despite the remarkable performance of deep learning models in computer vision tasks, they often suffer from overfitting. Data augmentation (DA) as an effective technique has been proved to im-
prove the generalization ability of deep learning models (Shorten & Khoshgoftaar, 2019). Previous works (Cubuk et al., 2020; Gontijo-Lopes et al., 2020) indicate that the main benefit of DA arises from increasing the diversity of images. Popular techniques, such as rotation, flipping, color transformation, have been performed as commonly used augmentation methods. Recently, thanks to the skillful design of human experts, many DA methods (e.g., Cutout (DeVries & Taylor, 2017), Mixup (Zhang et al., 2017), CutMix (Yun et al., 2019)) have been proposed and show significant performance. However, these manually designed methods require additional human prior knowledge on the dataset and sometimes they are limited to certain datasets and target tasks. Naturally, automatically finding DA methods from data have emerged to overcome the limitations of dependence on manual cumbersome exploration. Some works use generative adversarial networks (GANs) to directly generate training data (Shrivastava et al., 2017; Tran et al., 2017).
Furthermore, recent studies aim to automate the search for augmentation policies that choose the optimal transformations for training images. AutoAugment (AA) (Cubuk et al., 2018) adopts a controller to generate an augmentation policy which is used to train a proxy network, then gets the validation accuracy as the reward signal to update the controller using reinforcement learning. Unfortunately, the evaluation of thousands of policies makes AA computationally expensive. Therefore, multiple automated DA approaches focus on reducing the huge complexity and have achieved great progress. For example, PBA (Ho et al., 2019) employs hyperparameter optimization, Fast AA (Lim et al., 2019) uses a density matching algorithm, Adv AA (Zhang et al., 2019b) proposes an adversarial framework to jointly optimize the target network and the augmentation network, and DADA (Li et al., 2020) and DDAS (Liu et al., 2021a) target relaxing the discrete DA policy selection to a differentiable problem. Besides, RandAugment (Cubuk et al., 2020) removes the separate search on a proxy task in the offline automated DA methods (e.g., AA) while also outperforming them. However, these automated DA methods perform the search at the image level, i.e., they use the same policy on the whole image. It inevitably ignores the diversity of different regions in an image, which limits the diversity of data increased by data augmentation, and sometimes causes the damage of critical semantic information. In contrast, our method takes diversity in local regions and contextual relationships into account to search for augmentation policies for multiple regions.
Our proposed method is conceptually orthogonal to most region-based DA methods where DA transformations are performed in a non-automated way. For example, RandomErasing (Zhong et al., 2020), CutMix (Yun et al., 2019) perform cropping or replacement operation on a randomly selected rectangle region, Hard chips (Hong et al., 2019) extracts the object patches from an object pool and pastes them to an image to train the detector, and SECOND (Yan et al., 2018) performs DA by pasting the objects into the 3D point cloud. Some works further exploit class activation map (Zhou et al., 2016) or saliency map (Zhou et al., 2015) to select representative regions which are augmented by a randomly selected DA operation (e.g., SaliencyMix (Uddin et al., 2020), SnapMix (Huang et al., 2020)), AugMix (Hendrycks et al., 2019), KeepAugment (Gong et al., 2020) and TrivialAugment (Müller & Hutter, 2021)). Yet our proposed method automatically searches for the augmentation transformations based on the given datasets and tasks.

heading: 2.2 MULTI-AGENT REINFORCEMENT LEARNING

text: The most significant characteristic of MARL is the cooperation between agents (Tampuu et al., 2017; Ma & Cameron, 2008; Foerster et al., 2017) which is distinct from directly applying reinforcement learning (RL) algorithms to multi-agent systems. Due to the limited observation and action of a single agent, cooperation is necessary in the reinforced multi-agent system to achieve the common goal. Compared with independent agents, cooperative agents can improve the efficiency and robustness of the model (Neto, 2005; Zhang et al., 2019a; Busoniu et al., 2008). Many vision tasks use MARL to interact with the public environment to make decisions, with the goal of maximizing the expected total return of all agents, such as image segmentation (Liao et al., 2020; Han et al., 2019; Ma et al., 2020), image processing (Furuta et al., 2019).

heading: 3 PROPOSED METHOD

text:
As above mentioned, we aim to search augmentation policies for the patches to explore more augmentation diversity. Learning the optimal DA policies often needs to interact with the target network, which aligns with the nature of RL (i.e., RL agents learn to make decisions through interacting with the environment (Sutton & Barto, 2018; Arulkumaran et al., 2017)). In addition, it is of great
necessity to take the patch content and the contextual relationship between patches into account. Therefore, we use multi-agent reinforcement learning (MARL) algorithm. Specifically, the search for patch policies is based on the content of the patch together with the semantics of the image, and policies are encouraged to coordinate to achieve the joint optimal DA policy across the whole image. In this section, we first describe the preliminaries of MARL, then elaborate on our augmentation policy formulation and modelling. Furthermore, we summarize the framework of Patch AutoAugment.

heading: 3.1 PRELIMINARIES OF MULTI-AGENT REINFORCEMENT LEARNING

text:
We first introduce the preliminaries of reinforcement learning (RL). RL models the decision-making problem as a Markov decision process (MDP) which is presented with a tuple (S,A, P,R, γ, T ). In RL framework, given the state s ∈ S, the agent takes an a ∈ A according to its policy π(a |s) : S × A → [0, 1] and then receives a reward r : S × A → R. The environment moves to the next state with a transition function denoted as P : S × A × S → [0, 1]. γ ∈ (0, 1] is a discount factor and T is a time horizon. The agent aims to maximize the long-term reward R over T steps to learn the optimal policy π∗.
Furthermore, multi-agent reinforcement learning (MARL) considers a group of N agents, denoted as N , operating cooperatively in a shared environment towards a common goal. It can be formulated as a multi-agent MDP (MAMDP) (Boutilier, 1996) represented with a tuple (S, {Oi}Ni=1, {Ai}Ni=1, P,R, γ, T ). Here, S describes the shared state space and {O1, · · ·,ON} is a set of observations for agents. In the MARL configuration, each agent receives a private observation correlated with part of state, i.e., oi : S → Oi. According to the global state s and its observation oi, the agent takes its action ai ∈ Ai based on its policy πi(ai|oi, s), Ai is the action space for the i-th agent and A = {A1, · · ·,AN} denotes the joint action space. Then, the joint action a = a1× · · · ×aN produces next state according to transition function P : S ×A×S → [0, 1] and the environment gives the team reward r : S ×A → R to agents N . The objective of each agent is to maximize the total accumulative reward R to cooperatively learn the globally optimal policy π∗ = {π∗1, · · ·, π∗N} that consists of the optimal policy of each agent. In this paper, as mentioned before, we employ MARL to search optimal policy for each patch, and achieve the optimal effectiveness of DA across the entire image to further improve the performance of the target network as possible.

heading: 3.2 PATCH AUTOAUGMENT

text:
In our proposed Patch AutoAugment (PAA), we formulate the task of policy search for the patches as a cooperative multi-agent decision-making problem and adopt multi-agent reinforcement learning (MARL) to solve it. In the following, We clarify the detailed formulation i) the state, observation and action modeling for the policy, ii) an effective team reward function design and iii) the detailed MARL algorithm for policy learning) of Patch AutoAugment.
Policy Modeling. As illustrated in Figure 2, given the original input batch x and the corresponding label y (i.e., {x, y} = {xj , yj}bj=1 and b is the batch size), we divide an image xj into N equalsized and non-overlapping patches, denoted as xj = {Pij}Ni=1 whereP j j is the i-th patch of the image xj . In our proposed method, we aim to search the augmentation policy for each patch. Therefore, in MARL formulation, the augmentation policy of the patch Pi is controlled by an agent i ∈ N and we detail the state, observation and action for the augmentation policy as below.
State. As aforementioned, the selection of augmentation operation for a patch is closely bound up with the contextual relationship between regions. Therefore, the augmentation policy needs to perceive the image semantics and we take the deep features of the whole image extracted by a backbone (e.g., ImageNet pre-trained ResNet-18 (He et al., 2016)) as the global state s which is visible to all agents. We analyze the impact of feature extractor initialization in Appendix F.
Observation. Apart from capturing state (i.e., the global information), the agent only use their own observation (i.e., the local information) which is invisible to other agents. The augmentation policy seeks to choose augmentation operation based on the content of the patch and the observation is generally part of the state. Considering all these factors, we utilize the deep features of the i-th patch as observation oi, which are extracted by the same backbone as the state extractor.
Action. The augmentation policy is responsible for choosing which transformations to apply from pre-defined operations. Following the previous automated DA methods (Cubuk et al., 2018; Lim et al., 2019), we define the fifteen operation functions (i.e., ShearX/Y, TranslateX/Y, Rotate, Invert, Equalize, Solarize, Posterize, Contrast, Color, Brightness, Sharpness, RandomErasing, Cutout, Mixup, Cutmix) to construct the action spaceA. Given the state s and the observation oi, according to policy πi(ai|oi, s), each agent i determines an action ai(·) ∈ A, which is the operation performed on the patch. Each operation is associated with two hyperparameters: probability p and magnitude m. In order to dramatically and effectively reduce the action space for augmentation policy, similar with (Cubuk et al., 2020; Ho et al., 2019; Lim et al., 2019), we take a fixed probability and magnitude schedule. Among them, the probability of applying the operation is sampled from the uniform distribution (i.e., pi ∼ U(0, 1). Following (Ho et al., 2019), we employ the same linear scale to be the magnitude schedule. In summary, the processed i-th patch in image xj is denoted as P̃ ij = ai(P i j) with the probability pi and the magnitude of ai(·) is mi, otherwise P̃ ij = P ij . Note that most operations are label-invariant, except Mixup (Zhang et al., 2017) and CutMix (Yun et al., 2019) are label-disturbing operations that combine different patches as well as their labels. We take Mixup as an example, then P̃ij = λPij + (1 − λ)Pit with probability pi, where Pit is a patch from another image xt, t 6= j, λ ∼ Beta(α, α), for α ∈ (0,∞), and the one-hot label is modified as ỹj = λ N yj+ (1−λ) N yt. Once all patches are processed by the corresponding operations chosen by the augmentation policies, we obtain the image augmented by our PAA, denoted as x̃j = {P̃1j , ···, P̃Nj }, and the final label ỹj . More operation details are shown in Appendix A.
Reward Function. The reward function is of importance to guide the agents to learn so that they follow desired behaviors. The previous work, AdvAA (Zhang et al., 2019b), attempts to increase the training loss of the target network to generate harder augmentation policies and explore the weakness of the target network. Inspired by AdvAA, we reformulate the reward design appropriately under our configuration. In our proposed PAA, the common objective of all agents is to improve the performance of the mainstream target task through enhancing the benefits of DA. Therefore, we compare the feedback of target network φ(·) on the augmented data processed by our proposed PAA x̃ with the original data x and take their difference on the training losses as the reward for the policy in MARL, as in Eq. (1):
r = l(φ(x̃), ỹ)− l(φ(x),y), (1)
where x and y denote raw inputs and labels in supervision tasks. In our PAA model, all agents are encouraged to cooperate to achieve the common goal, thus we adopt the team reward function design (i.e., the shared reward mechanism in MARL) as Eq. (1) for all agents to make the joint augmentation policy achieve the optimal effectiveness.
Policy Learning. Here, we introduce the training for the augmentation policies mentioned above. Considering that the action space is discrete, we adopt multi-agent Advantage Actor-Critic algorithm (Lowe et al., 2017) to learn the augmentation policies and encourage the coordination behaviors. In MARL, the framework of centralized training with decentralized execution (Foerster et al., 2018; Rashid et al., 2018) is generally adopted. More concretely, each agent i has an actor which is to learn discrete policy πi(ai|oi, s) and agents share a common critic which aims to estimate the value of global state V π(s). And we use the centralized critic to train decentralised actors. Here, we reformulate it appropriately for our task. We model the centralized action-value Q function that takes the actions of all agents in addition to state information s and outputs the Q-value for the team, formulated as:
Qπ(s,a) = Eπ[Rt|s, a1, · · ·, aN ], (2)
where a is the joint action of all agents a = {ai, · · ·, aN} and Rt = ∑T l=0 γ
lrt+l is the long-term discounted reward. Then, the advantage function on the augmentation policy is given as follows:
Aπ(s,a) = Qπ(s,a)− V π(s). (3)
The critic network is a Multi-layer Perceptron (MLP). And we use ϕ to denote the its parameters. We take the square value of the advantage function Aπ as the loss function to update ϕ:
L(ϕ) = (Aπ(s,a)) 2 . (4)
The global policy network is a fully convolutional network (FCN), denoted as θ. Besides, to further achieve the ability to cooperate, similar to (Foerster et al., 2018; Rashid et al., 2018), the parameters of the actor networks of all agents are shared. The loss function for updating θ is defined as: L(θ) = − log πθ(a|s)Aπ(s,a). (5) Framework Summary. In this part, we summarize the overall framework of our proposed PAA. As shown in Figure 2, PAA first divides an image into a grid of patches. Then, we use a feature extractor to obtain the deep features of the whole image as the state. Each agent draws its individual observation, i.e., the deep features of the patch. According to the global state (i.e., the semantics of the entire image) and the local observation (i.e., the content of the patch), the actor networks output the augmentation operations of patches to further construct the joint operation map performed on the whole image. The augmented images are processed by our proposed PAA and then we input them to the target task network for parameters updating. Moreover, the feedback of the target network is used as the team reward signal to update the policy network.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENT OVERVIEW

text: In this section, to study the effectiveness of Patch AutoAugment (PAA), we conduct experiment on image classification, fine-grained image recognition and local tasks where local information is of importance, e.g., object detection. Specifically, we focus on CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009) datasets as well as three fine-grained object recognition datasets, i.e., CUB-200-2011 (Wah et al., 2011), Stanford Cars (Krause et al., 2013), FGVCAircraft (Maji et al., 2013) and Pascal VOC 2007 (Everingham & Winn, 2009). We describe the datasets in detail in Appendix B. We compare PAA with baseline pre-processing, Cutout (DeVries & Taylor, 2017), Mixup (Zhang et al., 2017), Cutmix (Yun et al., 2019), Co-Mixup (Co-Mix) (Kim et al., 2021), AutoAugment (AA) (Cubuk et al., 2018), Fast AutoAugment (FastAA) (Lim et al., 2019), RandAugment (RA) (Cubuk et al., 2020), DADA (Li et al., 2020) and Adversarial AA (AdvAA). The baseline follows (Zoph et al., 2018; Yamada et al., 2018; Gastaldi, 2017): standardizing the data, horizontally flipping with 0.5 probability, zero-padding and random cropping. Our policy model architecture design mainly follows (Chu et al., 2016), and we make some adjustments according to our task scenario. More details about our policy network architectures and the hyperparameters of the target model and our policy model are supplied in Appendix C and D, respectively. Moreover, we default the number of patches N to 16 except for CIFAR tasks N = 4. To ensure the reliability of our experiments, we run each experiment four times using different random seeds.

heading: 4.2 RESULTS AND ANALYSIS

text:
Classification Results on CIFAR-10 and CIFAR-100. For CIFAR-10 and CIFAR-100, we examine on Wide-ResNet-28-10 (WRN-28-10) (Zagoruyko & Komodakis, 2016), Shake-Shake (SS) (Gastaldi, 2017) and Pyramid-Net+ShakeDrop (Pyramid+SD) (Han et al., 2017; Yamada et al., 2018) models. The results are reported in Table 1, which shows the proposed approach consistently outperforms several state-of-the-art DA methods. We observe that the improvement of performance is relatively slight, due to the small image size of CIFAR which is 32 × 32. In the following, we further apply our proposed PAA on datasets with larger image sizes and other networks.
Classification Results on ImageNet. As shown in Table 2, we evaluate our method on ResNet-50 and ResNet-200 (He et al., 2016) backbone on ImageNet, and our PAA significantly improves the performance of the target networks. The results further demonstrate that our proposed method is an effective DA technique for consistent and expressive benefits for datasets with larger image sizes.
Effectiveness of Fine-grained Classification. Furthermore, we evaluate our proposed method on fine-grained image recognition tasks. According to previous work (Du et al., 2020; Chen et al., 2019), we take ResNet-50 and ResNet-101 as the backbones. The results are shown in Table 4, which illustrates that the performance of PAA is consistently better than other methods and PAA has achieved remarkable performance on these challenging fine-grained tasks.
Effectiveness of Local Tasks. In order to further verify the effectiveness of our proposed method, we conduct experiments on the object detection task. We adopt the mainstream detector Faster RCNN (Ren et al., 2015) (ResNet-50) on the dataset Pascal VOC 2007 (Everingham & Winn, 2009) and show the mAP results on the Table 5. We observe that the proposed PAA achieves consistent improvements and PAA is indeed effective for the local tasks like object detection.

heading: 4.3 COMPLEXITY ANALYSIS

text:
In this section, in order to further demonstrate the performance of PAA in terms of complexity, we compare the policy search time and training time of PAA with AA (Cubuk et al., 2018), FastAA (Lim et al., 2019) and AdvAA (Zhang et al., 2019b), as illustrated in Table 3. As shown in Table 3, compared to the previous works, PAA requires the fewest total computational resources, and the search time is almost negligible. As for the parameters, the total number of PAA model parameters (about 0.23M) is less than 1% of the target network (e.g., ResNet50: about 25.5M).
In summary, the main methods to reduce the computational cost lie in three aspects: 1) The augmentation policy network is jointly optimized with the target network, similar to (Zhang et al., 2019b; Lin et al., 2019). Namely, our proposed method searches for policies in an online manner, obviating thousands of policies validation on a small proxy network and the requirement for retraining the target network. Besides, we use fixed schedules for the two corresponding hyperparameters (i.e., probability and magnitude) of each transformation to effectively reduce the search space, which makes it easier to search for effective policies. By these means, PAA compresses most of the policy search time 2) As mentioned before, a MARL algorithm is adopted, in which all agents parallelly learn the augmentation policies, to reduce the training time. 3) Compared with the previous DA methods using image-by-image sequential transformations, we perform parallel transformations on tensor. Specifically, we pick the patches in a batch performing the same operation to reconstruct
a new tensor. And we use Kornia1 to realize tensor transformations on GPU to further reduce the processing time which is included in the training time.

heading: 4.4 ABLATION STUDY

text:
In this section, we study the effectiveness of MARL and also discuss the design of patch numbers, through several ablation experiments.
Performance of random policies. We randomize the augmentation policy for each patch, dubbed Patch RandAugment and compare it with our proposed PAA. To be specific, the augmentation policy is randomly selected from predefined transformations with uniform probability. As shown in Table 6, Patch RandAugment (PRA) leads to considerable appreciable improvements. However, our Patch AutoAugment with meaningful guidance significantly surpasses the random patch policies.
Performance of the policies searched using single-agent RL. Furthermore, we directly use the same single-agent RL to search for each patch’s policy, termed as Patch SingleAutoAugment (PSAA) and compare it to our PAA, where the search task is a cooperative multi-agent problem. In short, PSAA ignores the contextual relationship between patches, where the agents are independent and patches are non-cooperative. The results (see Table 6) indicate that taking the contextual information into account with cooperative RL-agents has improved the joint effectiveness of DA.
Impact of the number of patches. We explore the effect of the number of patches on the target model performance. As shown in Table 7, we respectively set the number of patches N = {4, 16, 64, 256, 1024}. The results show that on ImageNet / CUB-200-2011, when patch numbers N increase, the performance accuracy first increases and then decreases. In addition, when N = 16, PAA achieves the best performance. We analyze that the small number of patches may cause PAA to be unable to effectively explore local diversity, and its advantages would be limited. In particular, when N = 1, PAA degenerates into image-wise automated DA. In contrast, under too larger values (e.g., the extreme case is to search for DA policy for each pixel), the local semantic consistency is broken and the benefit brought by the consideration of contextual relationship between patches is gradually overtaken.

heading: 4.5 VISUALIZATION

text:
Policy Visualization. Grad-CAM (Selvaraju et al., 2017) is used to localize the important regions in the image. Therefore we can calculate the importance score of each patch, and we divide the importance scores into four bins, i.e., very important, important, normal and not important. Then, we categorize patches into four groups and draw four stacked area charts showing the percentages of operations selected by PAA augmentation policies over time. We take ResNet-50 backbone trained on CUB-200-2011 as an example. Since our proposed PAA searches for patch policies in an online manner, the strategies change dynamically over time as shown in Figure 4. At the beginning of the training process, the selected actions are messy since the MARL network is in the exploratory stage. At the tail end of the training, the target network has converged, causing the percentage of all operations to be almost the same and the percentage to flatten out.
In addition, we have some interesting findings which may provide some insights to the DA community. i) The optimal augmentation strategies of patches vary by their importance levels. Therefore, it is necessary to take the image content into account when performing data augmentation. ii) In
1Kornia (Riba et al., 2020) is a differentiable computer vision library for PyTorch. We use it to accelerate augmentation operation on tensors.
Table 7: Ablation study: performance comparisons (Top-1 accuracy (%)) under five different patch numbers N on CIFAR-10 (image size: 32 × 32) with WRN-28-10 and on ImageNet (image size: 224×224) together with CUB-200-2011 (CUB) (image size: 448×448) with ResNet-50 backbone.
N 4 16 64 256 1024 CIFAR-10 97.5 97.3 97.2 97.1 -
CUB 87.2 87.5 87.3 87.1 86.8
Baseline AutoAugment Ours
Figure 3: Grad-CAM visualization on the examples from CUB with ResNet-50 trained by baseline, AutoAugment and our PAA. PAA tends to help the target network focus on task-related features.
O pe
ra tio
n Pe
rc en
ta ge
(% )
O pe
ra tio
n Pe
rc en
ta ge
(% )
brightness (c) Normal
cutmix color
invert
Epochs (a) Very Important
Epochs (b) Important
Epochs Epochs
100
80
60
40
20
0
100
80
60
40
20
0
100
80
60
40
20
0
100
80
60
40
20
0 0 20 40 60 80 100 120 140 160 180 2000 20 40 60 80 100 120 140 160 180 200
0 20 40 60 80 100 120 140 160 180 200 0 20 40 60 80 100 120 140 160 180 200
contrast (d) Not Important
equalizemixup shear randomerasing translate cutout sharpnessrotation posterize sloarize
Figure 4: We categorize patches into four groups according to the patch importance calculated through Grad-CAM. We draw four stacked area chart of the operations’ percentages over time. The area represents the percentage of each operation.
the middle of the training process, different types of patches prefer to select different augmentation operations. Concretely, as illustrated in Figure 4, for the important patches, color transformation is mostly picked. For the unimportant patches, RandomErasing and Cutout are usually chosen by PAA. Important patches commonly take along semantic information that is related to mainstream tasks. It is better to choose the mild transformations (e.g., color) for them, which can effectively protect the semantic information from being damaged. In contrast, unimportant patches typically carry unexpected features (Singla et al., 2021) which are causally unrelated to the desired class. Severe transformations (e.g., Cutout, RandomErasing) could be chosen for them, which introduce noise and disturbance to reduce the impact of unexpected features on the target network learning.
Grad-CAM Visualization. Here, we adopt Grad-CAM (Selvaraju et al., 2017) to visualize the learned features to intuitively show the impact of PAA, as shown in Figure 3. We take the ResNet-50 as the backbone which is trained with dataset processed by 1) Baseline 2) AutoAugment and 3) Patch AutoAugment, respectively. We observe that the model trained with PAA focus on more task-related areas rather than spurious correlations (e.g., the branch where the bird stands) or overemphasized features (e.g., birds’ claws). More visualization results are provided in the Appendix E.

heading: 5 CONCLUSION

text: In this paper, we propose Patch AutoAugment (PAA), a more fine-grained automated data augmentation approach. Our method adopts multi-agent reinforcement learning to automatically search for the optimal augmentation policies for patches, and encourages agents to cooperate with each other to further achieve the joint optimal policy across the entire image. Extensive experiments demonstrate that PAA improves the target network performance with low computational cost in various tasks. Meanwhile, we use visualization to show that PAA is beneficial for the target network to localize more class-related cues. Furthermore, we hope that our visual observations of policies will be useful for future development. In future work, we will investigate different schemes on dividing different regions. Additionally, our method is naturally aligned with the patch token mechanism in the current vision transformers (Dosovitskiy et al., 2020; Touvron et al., 2021; Liu et al., 2021b) and the data augmentation specific to vision transformers has not been extensively studied. Therefore, we leave the automated data augmentation for vision transformers to future work, which may provide some interesting insights to the community.

heading: A OPERATIONS DETAILS

text:
Following (Cubuk et al., 2018; Ho et al., 2019; Lim et al., 2019; Zhang et al., 2019b), we define the fifteen common augmentation operations to form the action space. Here, we detail the description of these operations, as illustrated in Table 8. In addition, we give magnitudes range of augmentation operations corresponding to hyperparameters of the functions in the Kornia PyTorch library. Some operations, such as Cutout and CutMix, have no parameters.
Additionally, in our implementation, we pick out the patches that perform the same operation and put them into a new tensor. Then, we speed up the process by performing parallel transformations on the tensor. Tensor transformations on GPU can be realized by Kornia (Riba et al., 2020) to reduce the computational costs. In particular, when performing the label-disturbing operations (i.e., Mixup and CutMix), a patch needs to mix with another patch that is randomly selected from the new tensor. Namely, a patch is mixed with another patch that performs the same operation.

heading: B DATASETS

text:
We evaluate Patch AutoAugment (PAA) on the following datasets: CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), ImageNet (Deng et al., 2009) and three fine-grained
image recognition datasets (CUB-200-2011 (Wah et al., 2011), Stanford Cars (Krause et al., 2013), FGVC-Aircraft (Maji et al., 2013)) and Pascal VOC (Everingham & Winn, 2009).
To be specific, both CIFAR-10 and CIFAR-100 have 50,000 training examples. Each image of size 32 × 32 belongs to one of 10 categories. ImageNet dataset has about 1.2 million training images and 50,000 validation images with 1000 classes. Original ImageNet data have different sizes and we resize them to 224 × 224. In addition, we evaluate the performance of our proposed PAA on three standard fine-grained object recognition datasets. CUB-200-2011 (Wah et al., 2011) consists of 6,000 train and 5,800 test bird images distributed in 200 categories. Stanford Cars (Krause et al., 2013) contains 16,185 images in 196 classes. The FGVC-Aircraft dataset contains 10,200 images of aircraft, with 100 images for each of 102 different aircraft model variants, most of which are airplanes. The image size in the above three datasets is 448× 448. The Pascal VOC dataset consists of a set of images and each image has an annotation file giving a bounding box and object class label for each object in one of the twenty classes present in the image. The data has been split into 50% for training and 50% for testing. In total there are 9,963 images, containing 24,640 annotated objects.

heading: C MODEL ARCHITECTURE

text: Here, we provide the detailed model architecture for each component in our PAA augmentation policy model, including feature extractor network, actor network and critic network. We use the pretrained on ImageNet ResNet-18 backbone (excluding the final avgpool and softmax layer) to extract the deep features of the image and the patch, which are denoted as the state s and the observation oi respectively. As for the global policy network and the critic network, the detailed model architectures are shown in Table 9. The policy network is a fully convolutional neural network (FCN). The parameters of the actor networks of all agents are shared. Although they share parameters, each agent’s policy is not identical. When determining an action, each agent’s policy relies on different activations of the global policy network.

heading: D HYPERPARAMETERS

text:
We detail various target models hyperparameters (e.g., batch size, learning rate and training epochs) on CIFAR-10, CIFAR-100, ImageNet, CUB-200-2011, Stanford Cars and FGVC-Aircraft in Table 10. We do not specifically tune these hyperparameters, and all of these are consistent with previous works (Cubuk et al., 2018; Lim et al., 2019; Dabouei et al., 2021; Du et al., 2020; Chen et al., 2019).
In addition, the time horizon T determines the times of augmentations performed sequentially on a patch. Therefore the agent can output more than one augmentation for a patch by setting T > 1. In our PAA model, we set time horizon T = 1, i.e., the augmentation policies take actions at every time step for saving computational cost and avoiding over-regularization. Even in the single-step decision-making process (i.e., T = 1), RL can still effectively search for the optimal policy as illustrated in (Mnih et al., 2016; Singh et al., 2000). In addition, we use the SGD optimizer with an initial learning rate of 1e-4 to train the actor network and the critic network.

heading: E MORE GRAD-CAM VISUALIZATION

text:
In this section, we provide more Grad-CAM (Selvaraju et al., 2017) results of ResNet-50 models trained using baseline augmented data, AA (Cubuk et al., 2018) and our proposed PAA, respectively, as shown in Figure 5. The visualization results demonstrate that our proposed PAA improves the localization ability of the target network and tends to help the target network focus on more parts of the foreground object. In short, our proposed PAA make the target network focus on the important or representative regions closely related to the class within an image.
F IMPACT OF THE INITIALIZATION OF THE FEATURE EXTRACTOR
We further study the impact of the initialization of the feature extractor as shown in Table 11. We first observe that the performance of the Non-pretrained ResNet-18 scheme (using non-pretrained ResNet-18 as the state/observation extractor) is still much better than the baseline. In other words, our agents do not rely on the pre-trained ResNet to produce features to be used as the state or observation. We also notice that the performance of the Non-pretrained ResNet-18 scheme is slightly lower than the Pretrained ResNet-18. This may be because that at the very early stage of the training, the features extracted by Non-pretrained ResNet-18 are not as good as Pretrained ResNet-18 in practice.

heading: G DETAILS OF COMPUTATIONAL COST

text: The computational cost of our proposed PAA is estimated on GeForce GTX 1080 Ti while AA (Cubuk et al., 2018) and AdvAA (Zhang et al., 2019b) are on NVIDIA Tesla P100. Besides, FastAA (Lim et al., 2019) is estimated on NVIDIA Tesla V100. The computational cost is mainly used for searching policies and training the target networks.