## Greedy AutoAugment

Alireza Naghizadeh , a Mohammadsajad Abavisani b and Dimitris N. Metaxas a a Department of Computer Science, Rutgers University, CBIM, Piscataway Township, NJ 08854.

b Department of Electrical and Computer Engineering, Georgia Institute of Technology, TReNDs center, 55 Park Pl NE, Atlanta, GA 30303.

## ARTICLE INFO

## ABSTRACT

Keywords : AutoAugment Augmentation ANN Neural Networks Vision

Classification

A major problem in data augmentation is to ensure that the generated new samples cover the search space. This is a challenging problem and requires exploration for data augmentation policies to ensure their effectiveness in covering the search space. In this paper, we propose Greedy AutoAugment as a highly efficient search algorithm to find the best augmentation policies. We use a greedy approach to reduce the exponential growth of the number of possible trials to linear growth. The Greedy Search also helps us to lead the search towards the sub-policies with better results, which eventually helps to increase the accuracy. The proposed method can be used as a reliable addition to the current artifitial neural networks. Our experiments on four datasets (Tiny ImageNet, CIFAR-10, CIFAR-100, and SVHN) show that Greedy AutoAugment provides better accuracy, while using 360 times fewer computational resources.

## 1. Introduction

Data augmentation 1 is an important technique that can help to improve the performance of various data analysis algorithms in the presence of insufficient data. For instance, a common practice in medical applications is class-specific data augmentation. In this case, gathering sufficient labeled data to train a deep neural network model is impractical (Ronneberger et al., 2015; Kuo et al., 2019). This is a classic type of long-tail distribution data, which is also prevalent in natural images (Zhu et al., 2014; Tang et al., 2015) and can be addressed by data augmentation methods (Wang et al., 2017). Other important usages of data augmentation methods include unsupervised learning (Dosovitskiy et al., 2014; DeVries and Taylor, 2017a; Bengio et al., 2013; Ozair and Bengio, 2014; Naghizadeh et al., 2020b; Naghizadeh and Metaxas, 2020) and in the improved training of generative adversarial networks (Peng et al., 2018; Luan et al., 2017). In this paper, we focus on the problem of data augmentation for image classification . The main goal is to increase the accuracy of image classification by applying the right augmentation techniques on training data.

Data augmentation in image classification is directly related to image-based object transformations. In the classification process, it is desirable to take into account a variety of such transformations to improve image classification. In other words, we want the perception of an object to be invariant to the properties such as scale, brightness, rotation, and viewing angle. Estimating important object transformations and applying them in the learning process is a critical problem in Artificial Neural Networks (ANNs). For instance, it is desirable that a network, after learning an object from its original form, recognizes the same object in a modified location where its scale, rotation, and other properties have

&lt; Corresponding author:

&lt;&lt; Corresponding author

ar.naghizadeh@cs.rutgers.edu (A. Naghizadeh)

ORCID(s):

1 This is the the arxiv version of the Greedy AutoAugment paper (Naghizadeh et al., 2020a).

been changed. Currently, there are two ways to deal with this problem. First, by designing network architectures that can inherently be invariant to important image-based object transformations. Second, with the use of data augmentation methods.

The most basic network which considers the transformations of the input data is the Convolutional Neural Network (CNN). The CNN architecture, with the concept of convolutional layers, tries to be translation invariant (LeCun et al., 1990; Krizhevsky et al., 2012). This network was very successful in its approach and has been used as a basis for the development of more advanced architectures (Szegedy et al., 2015; He et al., 2016a; Huang et al., 2017). Another example of this approach is CapsuleNet, which tries to find the relevant pose information automatically (Hinton et al., 2011, 2018; Sabour et al., 2017). While the design of CapsuleNet improved the results of basic datasets (LeCun, 1998), unfortunately, it could not improve the accuracy for more complicated datasets such as ImageNet (Deng et al., 2009).

The second method for considering different transformations of the input data is to use data augmentation. In this method, the objective is to achieve invariance by applying different image transformations such as geometry transformation, kernel filtering, color transformation, image mixing, random erasing (DeVries and Taylor, 2017b), etc. The main advantage of this method is simplicity and supporting all forms of ANN architectures. Additionally, there is a possibility to use transformation techniques in which current ANN architectures do not support.

One of the most important factors for data augmentation techniques is the constraint on the number of possibilities for applying augmentation techniques. In this regard, only a subset of the possible techniques can be used for data augmentation. Therefore, a search mechanism is needed to find the best possible techniques. The most common method to find the best data augmentation techniques is to find them manually (Krizhevsky et al., 2012; Cire/uni015Fan et al., 2012; Huang et al., 2016) which needs prior knowledge and expertise. Recently, the AutoAugment (Cubuk et al., 2018) is proposed to

automate the process of finding the best augmentations. In this method, finding the augmentation policy is reduced to a discrete search problem over various augmentation techniques, each having hyperparameters of the probability of applying the operation and the magnitude to which the operation is applied. Because of the computational requirements for searching with AutoAugment, this method relies on transferability of the augmentation techniques. This means that the policies that are found with one dataset and architecture can be used for similar datasets with different architectures (Cubuk et al., 2018).

Using the same policies which are found for a specific scenario and trying to use them for others is risky and may lead to worse results (see the Results). To solve this problem, it is desired to perform the search more effectively, so that searching can be performed for each dataset separately. In this paper, we address the problem of how to effectively search for the most appropriate data augmentation techniques and apply them to training data. For this purpose, we propose Greedy AutoAugment. In this method, we develop a greedy-based search algorithm, which reduces the searching space from the exponential growth of possible trials to linear growth. With the proposed method, instead of searching for all possible outcomes, the search expands dynamically, and for each sub-policy, it is pushed towards trials with the best outcomes. To achieve this goal, we search among subpolicies with only two elements, the techniques of operations and the magnitudes for those techniques. The probability is applied after the search is finished with a methodology that gives weights to better policies. Our experimental results show that this approach is effective in providing higher accuracies. The Greedy AutoAugment, on four datasets, Tiny ImageNet, CIFAR-10, CIFAR-100, and SVHN, could reliably provide higher accuracies while using 360 times fewer computational resources.

## 2. Data Augmentation

Data augmentation refers to the practice of applying a series of standard transformations (wiredfool et al., 2016; DeVries and Taylor, 2017b; Inoue, 2018) to the given image data. In order to use these transformations, for each epoch in the training phase, a percentage of the data receives one or a combination of these techniques.

Figure 1: Random data augmentation techniques applied to samples from two real datasets. Each row receives the same augmentation technique with different magnitudes.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00704v2-figure-d20eef75-d31d-4839-811b-014f6326b1a9.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00704v2-figure-80ad9ce5-1ee8-4238-a7b3-9d565fdc5358.jpg)

The effect of applying data augmentations to the images from ImageNet and CIFAR-100 datasets are shown in Figure 1. In each row, one specific combination of augmentation techniques with different magnitudes is chosen randomly and is applied to the columns of images. As we can see, in some instances, the change is not noticeable, and in others, the change could completely change the original data. The goal is to discard combinations that would decrease the generalization ability of the network and select the best combinations that would increase it.

The augmentation techniques in their simplest form have been used in prominent artificial neural networks (Krizhevsky et al., 2012; Cire/uni015Fan et al., 2012; Huang et al., 2016). These networks mostly used a trial and error approach to manually find the best combinations. A more advanced approach is proposed in (Cubuk et al., 2018), which automates the searching process for finding the best augmentation techniques. There are also methods that do not perform a real search to find the best augmentations. Instead, these methods are created to be resilient against randomly selected augmentations (Ho et al., 2019; Lim et al., 2019).

Figure 2: The general scheme of AutoAugment algorithm.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00704v2-figure-575ea2ef-dec1-4391-9aaa-45fb87a78b34.jpg)

Thesearching process of the AutoAugment method heavily relies on NasNet (Zoph et al., 2018) as a controller to direct the search. The controller predicts a decision by using a one-layer LSTM, which contains 100 hidden units and 30 units softmax predictions. The prediction is then fed into the next step as an embedding. In the end, the controller uses 30 softmax predictions in order to select the best policies. The LSTM should be in direct contact with some child networks to train its own network. The child networks are usually a subset of the training networks. The accuracy updates from child networks are used to update the main LSTM network. To use the accuracies, a policy gradient method called Proximal Policy Optimization algorithm (PPO) is employed. The whole process is shown in Figure 2. The strategies (policies) have three elements 1- data augmentation techniques, 2- the probability of applying each operation, and 3- the magnitude of the operation. The policy gradient receives the values of reward R to update the controller. The LSTM uses the softmax predictions of its trained networks to select the best polices. In the end, the best policies are used to train the actual network.

Table 1

Augmentation techniques with their descriptions which are used in GAutoAugment.

|    | Technique    | Description                                                      |    | Technique   | Description                                                 |
|----|--------------|------------------------------------------------------------------|----|-------------|-------------------------------------------------------------|
|  1 | FlipLR       | Filliping the image along the vertical axis.                     | 11 | Contrast    | Changing the contrast of the image.                         |
|  2 | FlipUD       | Filliping the image along the horizontal axis.                   | 12 | Brightness  | Adjusting the brightness of the image.                      |
|  3 | AutoContrast | Increasing the contrast of the image.                            | 13 | Sharpness   | Adjusting the sharpness of the image.                       |
|  4 | Equalize     | Equalizing the histogram of the image.                           | 14 | ShearX      | Sheering the image in horizontal axis.                      |
|  5 | Invert       | Inverting the color of the pixels in the image.                  | 15 | ShearY      | Sheering the image in vertical axis.                        |
|  6 | Rotate       | Rotating the image by certain degrees.                           | 16 | TranslateX  | Translating the image in horizontal axis.                   |
|  7 | Posterize    | Redicing the number of Bits for each pixel.                      | 17 | TranslateY  | Translating the image in vertical axis.                     |
|  8 | CropBilinear | Croping with bilinear interpolation strategy.                    | 18 | Cutout      | Changing a random square patch of the image to gray pixels. |
|  9 | Solarize     | Inverting the color of all the pixels above a certain threshold. | 19 | Blur        | Blurring the image.                                         |
| 10 | Color        | Changing the color balance of the image.                         | 20 | Smooth      | Smoothing the image(Low-pass filtering).                    |

## 3. Searching Environment

To perform data augmentation on image data, we use policies. Let us define policy function GLYPH&lt;130&gt; f i that may include one or more sub-policy functions. The sub-policy is used as the functionality of applying augmentation techniques on image data. Each sub-policy has three essential elements, 1the augmentation technique, 2- the magnitude of the operation, and 3- the probability of applying the operation. For a complete list of augmentation techniques that we use in this paper, see Table 1. The magnitude is the degree in which an operation is applied. For instance, in the rotate augmentation, the magnitude specifies how much we should rotate an image. The third element specifies the probability of applying the augmentation on the image.

ularly, given base . t n GLYPH&lt;157&gt; p n GLYPH&lt;157&gt; mn / the possible number of trials increases exponentially with the increase of l . To solve this problem, we transform the base of the search space into a two-variable setup. Among the three variables, t; p , and m in the sub-policies, we do not search for the variable p and fixate its value to one. Instead of searching for the best probability values among sub-policies, we apply the probability with a methodology that gives more weights to the policies with better accuracy results. This simple design decision helps us to reduce the computational requirements of the searching algorithm considerably.

We define each image as a multivariate data point x . For augmentation on image data we use GLYPH&lt;130&gt; f . x / GLYPH&lt;153&gt; x ¤ which transforms the original image x into the augmented image x ¤ . The GLYPH&lt;130&gt; f i is separated into one or a multiple number of sub-policies. The sub-policy function is denoted by f . Each sub-policy receives three input values, f t; p; m . / , where the t , p , and m variables represent the augmentation technique, probability, and magnitude. The search space is defined as all of the possible combinations of concatenated sub-policies. The search space for a single f , includes all of the possible combinations of discrete values of t; p and m . Accordingly, the search space for any f is . t n GLYPH&lt;157&gt; p n GLYPH&lt;157&gt; mn / where t n ; p n , and mn are the maximum values for t; p , and m .

To represent the search space, we can divide it into different layers. A search space that has only one layer includes an augmentation function GLYPH&lt;130&gt; f i that has only one f as its subpolicy. We know that the search space for sub-policy f has a maximum number of possible elements of . t n GLYPH&lt;157&gt; p n GLYPH&lt;157&gt; mn / . The search space with two layers is defined for GLYPH&lt;130&gt; f i s that can concatenate two f s as their sub-policies. Respectively, the maximum number of possible elements for the search space expands to . t n GLYPH&lt;157&gt; p n GLYPH&lt;157&gt; mn / 2 . We generalize the number of layers for any search space with the variable l . The l is a discreet integer value such that l g 1 . The search space is defined as concatenated layers, with a maximum size of . t n GLYPH&lt;157&gt; p n GLYPH&lt;157&gt; mn / l , where l indicates the number of allowed sub-policies for GLYPH&lt;130&gt; f i .

The important point in the defined searching environment is that the search space expands exponentially. Partic-

Even with reducing the number of variables for the base, the growth of the number of possible trials is still exponential. To tackle this problem, when going from one layer to the other (concatenating two sub-policies), we use a greedy search. To use the greedy search, we replace the reinforcement learning in AutoAugment with Breadth-First Search, which is an explorer for tree-based data structures. Accordingly, the Greedy Breadth-First Search is used to explore the defined layered environment. The new number of the possibilities is defined as follows,

$$\sum _ { 1 } ^ { k } \left ( t _ { n } \times m _ { n } \right )$$

In this notation, k is an arbitrary integer number, which indicates the number of iterations the algorithm is allowed to perform in the search. The higher values of the k , allow algorithm to perform more trials, which may lead to higher accuracy. Therefore, the value of k should be set based on the available computational resources. The greedy approach helps us to convert the exponential growth of base . t n GLYPH&lt;157&gt; p n GLYPH&lt;157&gt; mn / with l to linear growth of base . t n GLYPH&lt;157&gt; mn / times the value of k which allows GLYPH&lt;130&gt; f i to potentially encompass many layers. For numerical values, we follow the discretization setup, which is introduced in AutoAugment. The number of augmentation techniques that we use in this paper is t n = 20 . The discretization of probabilities is with p n = 11 values with uniform space, and the discretization of magnitudes is with mn = 10 values with uniform space.

Algorithm 1 The proposed Greedy AutoAugment algorithm.

Input: Dataset X .

Output: Best augmentation policy functions GLYPH&lt;130&gt; f ; 1 § ; f GLYPH&lt;130&gt; d . 1: for all augmentation techniques t = 1,..., t n do 2: for all magnitudes m = 1, . . . , mn do 3: set probability to 1; 4: X GLYPH&lt;153&gt; X tr ; X te ; 5: GLYPH&lt;130&gt; f = f t; p; m . / ; 6: apply GLYPH&lt;130&gt; f on X tr ; 7: store policy with its score on X te . 8: end for 9: store best policy 10: end for 11: for counter = 2,..., k do 12: GLYPH&lt;130&gt; f = best policy; 13: for all augmentation techniques t = 1,..., t n do 14: for all magnitudes m = 1, . . . , mn do 15: set probability to 1; 16: X GLYPH&lt;153&gt; X tr ; X te ; 17: GLYPH&lt;130&gt; f = GLYPH&lt;130&gt; f + f t; p; m . / ; 18: apply GLYPH&lt;130&gt; f on X tr ; 19: store policy with its score on X te . 20: end for 21: end for 22: store best policy 23: end for 24: select GLYPH&lt;130&gt; f ; 1 § ; f GLYPH&lt;130&gt; d from all policies which have highest scores; 25: return GLYPH&lt;130&gt; f ; 1 § ; f GLYPH&lt;130&gt; d

## 4. Greedy AutoAugment

Training with the Greedy AutoAugment algorithm includes two main steps. The first step is to search and find the best GLYPH&lt;130&gt; f i s. The second step is to apply the GLYPH&lt;130&gt; f i s to X and increase the generalization ability of the network. To perform the greedy search, we use Algorithm 1. The input of the searching process is X , and the output is . GLYPH&lt;130&gt; f ; 1 § ; f GLYPH&lt;130&gt; d / , which are the best augmentation policies found in the search. The number of iterations of the algorithm needs to be at least equal to one ( k g 1 ).

The lines 1-10 are for k = 1 . The Breadth-First Search takes place in lines 1,2, where we go through all of the augmentation techniques and their respective magnitudes. In line 3, we use fixed value of 1 for the probability. In line 4, X is divided into two parts X tr ; X te for training and testing. In line 5, the sub-policy is considered as a complete policy. In line 6, sub-policy f(t,p,m) are applied on X tr . The classification of X te is determined in line 7 and the score of the sub-policy and its respective score is stored. Line 9 is for the greedy part of the algorithm, where the policy with the highest score is stored to act as a base sub-policy for more iterations.

For the next step (lines 11-23) the algorithm is for iterations with k &gt; 1 . As it is shown in line 11, the algorithm repeats its steps until the counter is equal to k . In line 12, the GLYPH&lt;130&gt; f is equal to the best policy with the highest score, which is not selected before. In this way, the next iteration starts from a base that has the best score. As we can see in line 17, the f(t,p,m) is added on top of the best policy. The other parts of the algorithm are the same as the first part. In the end, we select the d best GLYPH&lt;130&gt; f i s, which shows the highest accuracy results among all policies. In Algorithm 1, the probabilities for all policies were always set to the value of one. To determine the p values for GLYPH&lt;130&gt; f i s we propose the following process.

The goal is that the policy which has a higher score to receive a higher probability. For this purpose, let us define S ; 1 § ; S d \_ GLYPH&lt;21&gt; , where Si , is a set with GLYPH&lt;21&gt; number of probabilities, S 1 = ^ p ; 1 § ; p GLYPH&lt;21&gt; ' § ; ; S d \_ GLYPH&lt;21&gt; = ^ p d * GLYPH&lt;21&gt; ; § ; p d ' . In this regard, we define vector GLYPH&lt;146&gt; v = [ v ; 1 § ; v d \_ GLYPH&lt;21&gt; ; v . d \_ GLYPH&lt;21&gt; /+1 ] which represents the probability of choosing a set Si or the original data. The v 1 corresponds to selecting the original data and v ; 2 § ; v d \_ GLYPH&lt;21&gt; ; v . d \_ GLYPH&lt;21&gt; /+1 correspond to the selection of S ; 1 § ; S d \_ GLYPH&lt;21&gt; . To fill the values of GLYPH&lt;146&gt; v , we use the Pareto Distribution (Arnold, 2015), as follows,

$$\to v _ { i } = \begin{cases} \ ( \frac { 1 } { i } ) ^ { \alpha } & \quad i > 1 \\ 1 & \quad i \leq 1 \end{cases}$$

In this equation, a positive parameter ( GLYPH&lt;11&gt; ) is used to assign the highest probability to v 1 and lowest probability to v d . \_ GLYPH&lt;21&gt; /+1 . When i = 1 the probability is one, and for i &gt; 1 the probability is less than one but not zero. To choose the best set Si or original data, we start from the rightmost element of GLYPH&lt;146&gt; v and go to the leftmost element of GLYPH&lt;146&gt; v . Each of these elements has a chance to be selected based on their respective v i values. After choosing the Si , a policy is chosen randomly from the members of the set, with uniform distribution.

## 5. Results

In this section, we compare our method with current solutions. For this purpose, first, we show the accuracy results of our method compared to the other methods. Next, we provide a computational analysis of the overall augmentation process. The AutoAugment paper uses state-of-theart record-breaking networks also to test their method. This encompasses several architectural advancements, which increases RAM requirements in GPUs and require at least 1800 epochs for each test-case. Using such a vast requirement to test augmentation policies is actually not necessary and would limit us to analyze the methods thoroughly. Therefore, in order for the experiments to be in our available resources, we have used a smaller and reasonable infrastructure to test both our methods and the AutoAugment results in a uniform setting. According to the AutoAugment paper, their method is transferable. Therefore, this change is fair and should not affect their method.

In these experiments, we use eleven different prominent ANN architectures. The DenseNet121 (Huang et al., 2017), GoogLeNet(Szegedy et al., 2015), MobileNet (Howard et al., 2017), MobileNetV2 (Sandler et al., 2018), PreActResNet18

Figure 3: Samples from real datasets used in our experiments: (a) Tiny ImageNet (b) CIFAR-10 (c) CIFAR-100 (d) SVHN.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00704v2-figure-3dea61f0-0cb7-4206-97cc-c726efcb5c25.jpg)

Table 2

The result table for accuracy analysis. Abbreviations include: Manual = Manual Augmentation, GAutoAugment = Greedy AutoAugment.

| Network        | Manual   | Manual   | Manual   | Manual   | AutoAugment   | AutoAugment   | AutoAugment   | AutoAugment   | GAutoAugment   | GAutoAugment   | GAutoAugment   | GAutoAugment   |
|----------------|----------|----------|----------|----------|---------------|---------------|---------------|---------------|----------------|----------------|----------------|----------------|
| Network        | Tiny     | Cifar10  | Cifar100 | SVHN     | Tiny          | Cifar10       | Cifar100      | SVHN          | Tiny           | Cifar10        | Cifar10        | SVHN           |
| DenseNet121    | 34.65    | 80.67    | 53.91    | 91.69    | 29.15         | 80.20         | 54.80         | 91.47         | 37.75          | 83.36          | 59.32          | 93.26          |
| GoogLeNet      | 33.78    | 79.85    | 38.96    | 89.63    | 27.81         | 79.92         | 53.28         | 90.79         | 28.70          | 79.25          | 50.51          | 91.44          |
| MobileNet      | 14.58    | 68.50    | 40.75    | 81.43    | 10.50         | 63.95         | 37.28         | 79.53         | 17.38          | 64.90          | 43.09          | 88.55          |
| MobileNetV2    | 21.66    | 70.81    | 39.92    | 84.33    | 15.00         | 67.95         | 39.14         | 83.42         | 21.07          | 73.67          | 44.92          | 88.17          |
| PreActResNet18 | 29.45    | 77.16    | 44.57    | 89.06    | 26.01         | 83.05         | 53.28         | 89.77         | 32.60          | 83.43          | 48.26          | 93.60          |
| ResNet18       | 29.61    | 78.68    | 42.34    | 87.08    | 27.51         | 80.22         | 55.29         | 89.83         | 35.84          | 80.62          | 48.25          | 93.92          |
| ResNeXt29      | 31.87    | 76.95    | 47.71    | 88.57    | 23.85         | 78.15         | 50.90         | 81.63         | 35.04          | 81.87          | 56.92          | 84.08          |
| SENet18        | 29.18    | 78.66    | 42.16    | 90.17    | 25.63         | 80.55         | 54.18         | 91.71         | 31.96          | 79.90          | 51.69          | 93.07          |
| ShuffleNetG2   | 22.01    | 71.67    | 48.93    | 85.54    | 17.27         | 70.58         | 38.40         | 81.05         | 27.72          | 74.56          | 48.85          | 91.14          |
| ShuffleNetV2   | 24.89    | 72.95    | 49.26    | 87.44    | 20.50         | 71.25         | 46.44         | 86.18         | 27.10          | 75.27          | 53.19          | 91.88          |
| VGG            | 21.03    | 76.66    | 43.41    | 90.07    | 16.73         | 77.60         | 42.15         | 88.61         | 23.16          | 80.75          | 43.37          | 93.06          |

(He et al., 2016b), ResNet18 (He et al., 2016a), ResNeXt29 (Xie et al., 2017), ShuffleNetG2 (Zhang et al., 2018), ShuffleNetV2 (Ma et al., 2018), and VGG (Simonyan and Zisserman, 2014) are used as prominent networks which contain both normal and lightweight networks. To implement these networks, we forked the implementations from (Liu). The default settings of the network implementations are not changed. The networks accept 32 GLYPH&lt;157&gt; 32 images and provide an output based on the number of classes.

In the experiments, we also use four real datasets, 1Tiny ImageNet (Le and Yang, 2015) includes 120000 natural images in 200 classes with each class having a training set of 500 images a test set of 50 images along with 50 validation images. 2,3- CIFAR-10 and CIFAR-100 datasets (Krizhevsky et al., 2009), both containing 60000 images of size 32 GLYPH&lt;157&gt; 32 in 10 and 100 classes respectively. 4- SVHN (Netzer et al., 2011) which contains over 600000 images of real-world images of digits 0*9 . These selected datasets are used for three main reasons. First, while they are complex datasets, they have a reasonable number of images and features, which makes working with them with our available computational resources feasible. Second, they are wellknown datasets with known and predictable results on a variety of ANN architectures. Third, they are compatible with the official experiments of AutoAugment paper (Cubuk et al.).

The proposed method can be used with all architectures and all datasets with reasonable training size. For training, we have two completely separate steps, 1-finding policies, 2-applying those policies. For finding policies, we use Algorithm 1, and then we perform normal training with the new policies. The default learning rate for networks in (Liu) is 0.1. We use the same learning rate throughout the experi- ments for finding policies and training networks. The number of epochs for all of the training scenarios was 200 . To find the best policies, we need to create child networks. The child networks and training networks share the same infrastructures. To obtain the accuracies from child networks, we divided training data into two parts. The training part and the testing part. For Tiny Imagenet and SVHN, 5000 images are used for testing. For CIFAR-10 and CIFAR-100, 2500 images are used for testing. The images which are not used for testing part are used for training part. All of the images are selected randomly with i.i.d. distribution. The GLYPH&lt;11&gt; value for Pareto Distribution was always 2 . We also used d = 25 and GLYPH&lt;21&gt; = 5 .

## 5.1. Accuracy

In this section, we test the accuracy of our proposed method using four different datasets, 1- Tiny ImageNet, 2- CIFAR10, 3- CIFAR-100, and 4- SVHN. The results are shown in Table 2. In this table, the "Manual" section stands for the images with common augmentation methods. These techniques include zero padding, cropping, random-flip, and cutout. To prevent probable errors, the same source code released in AutoAugment for augmentation techniques is also used for manual augmentation (Cubuk et al.). This helps to provide a fair environment for all methods. The only extra preprocessing step that we used is for resizing Tiny Imagenet from 64 GLYPH&lt;157&gt; 64 to 32 GLYPH&lt;157&gt; 32 . This helped us to use the same infrastructure for all datasets without having adverse effects on the experiments. The values in the table are the average results from five trials.

Thesection "AutoAugment" stands for the AutoAugment method. For CIFAR-10, CIFAR-100, and SVHN, we use the

same policies that are found from AutoAugment method. For Tiny ImageNet, we use the policies that are found for ImageNet dataset. Because Tiny ImageNet is a subset of ImageNet, it can test the generalization of the AutoAugment method. The section "GAutoAugment" stands for Greedy AutoAugment, which is the proposed method. In our method, we suggest a specific searching process for each scenario to find the best policies. This is possible because (as we will see in the next section), our search method is computationally much more efficient than AutoAugment method.

For Tiny ImageNet, as we can see, the policies from AutoAugment are not effective when they are applied to a smaller subset of the same dataset. All of the networks had worse results compared to the Manual augmentation. Overall, the AutoAugment reduced the accuracy with 52 73~ : compared to Manual augmentation. Comparatively, our method increased accuracies for nine out of eleven available networks. Thehighest increase of accuracy is for ResNet18, with 6 23~ : higher accuracy. The least increase of accuracy is for VGG with 2 12~ : higher accuracy. Overall, compared to the AutoAugment, the proposed method provided 78 34~ : higher accuracy for eleven networks. Respectively, compared to the Manual augmentation, the proposed method provided 25 60~ : higher accuracy for eleven networks.

For CIFAR-10, the transition is better for AutoAugment policies. From eleven networks, six networks had better results with at most 5 89~ : better accuracy and at least 0 06~ : better accuracy than the manual augmentation. On the other hand, the manual augmentation showed better results for five networks with at most 4 54~ : better accuracy for MobileNet and at least 0 46~ : better accuracy in DenseNet121. The overall increase of the accuracy was 0 86~ : in favor of the AutoAugment. Comparatively, our method increased the accuracies for nine networks, with at most 6 27~ : better accuracy for PreActResNet18 and at least 1 24~ : better accuracy for SENet18. Overall, we could increase the accuracies, 25 02~ : better than manual augmentation and 24 15~ : better than AutoAugment.

The results for CIFAR-100 show that when policies from AutoAugment were applied to original images, the accuracies could improve for six networks. The best increase of the accuracy is for GoogLeNet with at most 7 49~ : better accuracy, and the least accuracy is for DenseNet121 with at least 0 91 : better accuracy. The overall increase of the accuracy was 1 04~ : for AutoAugment compared to the manual augmentation. When the policies from the proposed method are applied to the original network, we could improve the results for nine networks. The accuracy could be up to 11 54~ : and down to 2 34~ : better than the original images. The overall increase of the accuracy for the proposed method is 56 46~ : compared to the manual augmentation. The overall increase of the accuracy for all of the networks is 55 422~ : for the proposed method compared to the AutoAugment.

For SVHN, the AutoAugment provide higher accuracies for four networks when it is compared to the manual augmentation. The highest accuracy is for ResNet18 with 2 75~ : better accuracy and the least higher accuracy is 0 70~ : for

The result table for comparing the search space between our

Table 3 method and AutoAugment.

| Layers   | AutoAugment   |   GAutoAugment | Comparison     |
|----------|---------------|----------------|----------------|
| l = 1    | 2200 1        |            200 | 11 : 0         |
| l = 2    | 2200 2        |           4200 | 1152 : 3       |
| l = 3    | 2200 3        |           8200 | 1298536 : 5    |
| l = 4    | 2200 4        |          12200 | 1920131147 : 5 |

PreActResNet18. For the six other networks, the manual augmentation was, on average 2 45~ : better than AutoAugment. It was at least 0 21~ : and at most 6 93~ : better than AutoAugment. Overall, manual augmentation provided 11 00~ : better cumulative results compared to the AutoAugentation. Comparatively, the proposed method provided better accuracies for ten networks compared to the manual augmentation. The highest and lowest accuracies were 7 11~ 1 57~ : ; : better than manual augmentation. Overall, the cumulative accuracies are 37 16~ : ; and 48 16~ : better than manual augmentation, and AutoAugment. The better accuracy comes from the fact that SVHN needed policies that are in further layers of the search space, and this could only be achieved with Greedy AutoAugment.

## 5.2. Computational Analysis

As described in Section 3, the search space is reduced from the exponential growth of base . t n GLYPH&lt;157&gt; p n GLYPH&lt;157&gt; mn / with the value of l to the linear growth of . t n GLYPH&lt;157&gt; mn / with the value of k . We already saw that this approach is effective in improving the accuracy of the networks. The important question is now how much computational resources we needed to perform the search. To allocate the resources, we are concerned about two aspects, 1-the search space, 2-real resources allocated for the searching process. We separate this two processess because the algorithms may not use all of the trials is searching spaces.

The comparison between the two spaces is shown in Table 3. This table has three columns. The first column shows the number of the layers, which is ranged from l = 1 to l = 4 . The second column is for the search space for the AutoAugment algorithm. This shows the maximum number of possible selections that AutoAugment can choose. As we can see, the values for each row exponentially increases with the increase of the l from a base of 2200 . The 2200 is from calculating . t n GLYPH&lt;157&gt; p n GLYPH&lt;157&gt; mn / . The third column represents the search space for Greedy AutoAugment. To calculate the search space, we use the (2). In this case, if each layer is fully explored, the number of possible trials is only 4000 . The last column shows the comparison between column two and column three. The values show that the number of times the search space of the proposed method is smaller than the search space of AutoAugment.

For the real experiments, AutoAugment only searches for the 2-layer search space. This means that AutoAugment can have 2200 2 possible trials. Because searching all of the space for 2200 2 trials is impractical, the AutoAugment uses a sample of 15000 child networks. Comparatively, our

method, in each step, explores 200 trials. The number of steps is determined by k . In these experiments, we used k = 5 , which gives us 1000 trials. This number is used to limit the search within our computational resources. Also, AutoAugment uses 120 epochs to evaluate the accuracies of child networks. The number of epochs used for our child networks was only 5 epochs. Since the exact infrastructure for different datasets is not known for AutoAugment, and child networks are interchangeable, we consider the child networks to have the same efficiency. Therefore, overall we had .15000 GLYPH&lt;157&gt; 120/ GLYPH&lt;159&gt; .1000 GLYPH&lt;157&gt; 5/ = 360 less computational requirement.

## 6. Conclusion

A major problem in data augmentation is to ensure that the generated new samples cover the search space. In this paper, we proposed Greedy AutoAugment as a highly efficient method to find the best augmentation policies. We combined the searching process with a simple procedure to add augmentation policies to the training data. For the experiments on the classification accuracy, we used four real datasets and eleven networks. Our results show that the proposed method could reliably improve the accuracy of classification results. The higher accuracy could be achieved while using 360 times less computational resources.

## References

Arnold, B.C., 2015. Pareto Distribution. Wiley Online Library.

- Bengio, Y., Mesnil, G., Dauphin, Y., Rifai, S., 2013. Better mixing via deep representations, in: International conference on machine learning, pp. 552-560.
- Cire/uni015Fan, D., Meier, U., Schmidhuber, J., 2012. Multi-column deep neural networks for image classification. arXiv preprint arXiv:1202.2745 .
- Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V., . Training deep models on cifar-10 and cifar-100 using autoaugment. https:// github.com/tensorflow/models/tree/master/research/autoaugment . Accessed: 2019-07-26.
- Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V., 2018. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501 .
- Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L., 2009. ImageNet: A Large-Scale Hierarchical Image Database, in: CVPR09.
- DeVries, T., Taylor, G.W., 2017a. Dataset augmentation in feature space. arXiv preprint arXiv:1702.05538 .
- DeVries, T., Taylor, G.W., 2017b. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 .
- Dosovitskiy, A., Springenberg, J.T., Riedmiller, M., Brox, T., 2014. Discriminative unsupervised feature learning with convolutional neural networks, in: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (Eds.), Advances in Neural Information Processing Systems 27. Curran Associates, Inc., pp. 766-774.
- He, K., Zhang, X., Ren, S., Sun, J., 2016a. Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778.
- He, K., Zhang, X., Ren, S., Sun, J., 2016b. Identity mappings in deep residual networks, in: European conference on computer vision, Springer. pp. 630-645.
- Hinton, G.E., Krizhevsky, A., Wang, S.D., 2011. Transforming autoencoders, in: International Conference on Artificial Neural Networks, Springer. pp. 44-51.
- Hinton, G.E., Sabour, S., Frosst, N., 2018. Matrix capsules with em routing

.

- Ho, D., Liang, E., Stoica, I., Abbeel, P., Chen, X., 2019. Population based augmentation: Efficient learning of augmentation policy schedules, in: ICML.
- Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H., 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 .
- Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q., 2017. Densely connected convolutional networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708.
- Huang, G., Sun, Y., Liu, Z., Sedra, D., Weinberger, K.Q., 2016. Deep networks with stochastic depth, in: European conference on computer vision, Springer. pp. 646-661.
- Inoue, H., 2018. Data augmentation by pairing samples for images classification. arXiv preprint arXiv:1801.02929 .
- Krizhevsky, A., Hinton, G., et al., 2009. Learning multiple layers of features from tiny images. Technical Report. Citeseer.
- Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012. Imagenet classification with deep convolutional neural networks, in: Advances in neural information processing systems, pp. 1097-1105.
- Kuo, C.C., Chang, C.M., Liu, K.T., Lin, W.K., Chiang, H.Y., Chung, C.W., Ho, M.R., Sun, P.R., Yang, R.L., Chen, K.T., 2019. Automation of the kidney function prediction and classification through ultrasound-based kidney imaging using deep learning. npj Digital Medicine 2, 29.

Le, Y., Yang, X., 2015. Tiny imagenet visual recognition challenge.

- LeCun, Y., 1998. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ .
- LeCun, Y., Boser, B.E., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W.E., Jackel, L.D., 1990. Handwritten digit recognition with a back-propagation network, in: Advances in neural information processing systems, pp. 396-404.
- Lim, S., Kim, I., Kim, T., Kim, C., Kim, S., 2019. Fast autoaugment. arXiv preprint arXiv:1905.00397 .
- Liu, K., . Deep model infrastructures used for training gautoaugment. https://github.com/kuangliu/pytorch-cifar . Accessed: 2019-07-26.
- Luan, F., Paris, S., Shechtman, E., Bala, K., 2017. Deep photo style transfer, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4990-4998.
- Ma, N., Zhang, X., Zheng, H.T., Sun, J., 2018. Shufflenet v2: Practical guidelines for efficient cnn architecture design, in: Proceedings of the European Conference on Computer Vision (ECCV), pp. 116-131.
- Naghizadeh, A., Abavisani, M., Metaxas, D.N., 2020a. Greedy autoaugment. Pattern Recognition Letters .
- Naghizadeh, A., Berenjian, S., Margolis, D.J., Metaxas, D.N., 2020b. Gnm: Gridcell navigational model. Expert Systems with Applications 148, 113217.
- Naghizadeh, A., Metaxas, D.N., 2020. Condensed silhouette: An optimized filtering process for cluster selection in k-means. Procedia Computer Science 176, 205-214.
- Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y., 2011. Reading digits in natural images with unsupervised feature learning .
- Ozair, S., Bengio, Y., 2014. Deep directed generative autoencoders. arXiv preprint arXiv:1410.0630 .
- Peng, X., Tang, Z., Yang, F., Feris, R.S., Metaxas, D., 2018. Jointly optimize data augmentation and network training: Adversarial data augmentation in human pose estimation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2226-2234.
- Ronneberger, O., Fischer, P., Brox, T., 2015. U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical image computing and computer-assisted intervention, Springer. pp. 234-241.
- Sabour, S., Frosst, N., Hinton, G.E., 2017. Dynamic routing between capsules, in: Advances in neural information processing systems, pp. 38563866.
- Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C., 2018. Mobilenetv2: Inverted residuals and linear bottlenecks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520.

Simonyan, K., Zisserman, A., 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 .

Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., 2015. Going deeper with convolutions, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9.

Tang, Z., Zhang, Y., Li, Z., Lu, H., 2015. Face clustering in videos with proportion prior, in: Twenty-Fourth International Joint Conference on Artificial Intelligence.

Wang, Y.X., Ramanan, D., Hebert, M., 2017. Learning to model the tail, in: Advances in Neural Information Processing Systems, pp. 7029-7039. wiredfool, Clark, A., Hugo, Murray, A., Karpinsky, A., Gohlke, C., Crowell, B., Schmidt, D., Houghton, A., Johnson, S., Mani, S., Ware, J., Caro, D., Kossouho, S., Brown, E.W., Lee, A., Korobov, M., Górny, M., Santana, E.S., Pieuchot, N., Tonnhofer, O., Brown, M., Pierre, B., Abela, J.C., Solberg, L.J., Reyes, F., Buzanov, A., Yu, Y., eliempje, Tolf, F., 2016. Pillow: 3.1.0. URL: https://doi.org/10.5281/zenodo.44297 , doi: 10.5281/zenodo.44297 .

Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K., 2017. Aggregated residual transformations for deep neural networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500.

Zhang, X., Zhou, X., Lin, M., Sun, J., 2018. Shufflenet: An extremely efficient convolutional neural network for mobile devices, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848-6856.

Zhu, X., Anguelov, D., Ramanan, D., 2014. Capturing long-tail distributions of object subcategories, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 915-922.

Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V., 2018. Learning transferable architectures for scalable image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710.
Alireza Naghizadeh Mohammadsajad Abavisani Dimitris N. Metaxas
Greedy AutoAugment
A major problem in data augmentation is to ensure that the generated new
samples cover the search space. This is a challenging problem and requires
exploration for data augmentation policies to ensure their effectiveness in
covering the search space. In this paper, we propose Greedy AutoAugment as a
highly efficient search algorithm to find the best augmentation policies. We
use a greedy approach to reduce the exponential growth of the number of
possible trials to linear growth. The Greedy Search also helps us to lead the
search towards the sub-policies with better results, which eventually helps to
increase the accuracy. The proposed method can be used as a reliable addition
to the current artifitial neural networks. Our experiments on four datasets
(Tiny ImageNet, CIFAR-10, CIFAR-100, and SVHN) show that Greedy AutoAugment
provides better accuracy, while using 360 times fewer computational resources.