summaries: This paper addresses a limitation of BatchNorm: vulnerability to adversarial perturbations. The authors propose a possible explanation of this issue and correspondingly an alternative called RobustNorm to tackle this problem. Specifically, the authors observe that the statistics of BatchNorm for training and inference are different, resulting in different data distributions for training and inference. To solve this problem, the authors propose to use min-max rescaling instead of normalization. In addition, the running average is calculated with mean and the running mean of the denominator during inference. Experimental results show significant improvement of robustness and also comparable accuracy for clean data. Review: This paper investigates the reason behind the vulnerability of BatchNorm and proposes a Robust Normalization. They experimentally show that it is the moving averages of mini-batch means and variances (tracking) used in Normalization that cause the adversarial vulnerability. Based on this observation, they propose a new normalization method not only achieves significantly better results under a variety of attack methods but ensures a comparable test accuracy to that of BatchNorm on unperturbed datasets. The paper is clearly written, easy to read. This paper proposes an interesting perspective that BatchNorm may introduce the adversarial vulnerability, and probes why BatchNorm performs like that (the tracking part in BatchNorm). In experiment, the robustness of the networks increases by 20% when removing the tracking part, but the test accuracy on the clean images drops a lot. Afterwards, the authors propose RobustNorm, which performs better than BatchNorm for both natural and adversarial scenarios.
abstractText: Batch Normalization (BatchNorm) has shown to be effective for improving and accelerating the training of deep neural networks. However, recently it has been shown that it is also vulnerable to adversarial perturbations. In this work, we aim to investigate the cause of adversarial vulnerability of the BatchNorm. We hypothesize that the use of different normalization statistics during training and inference (mini-batch statistics for training and moving average of these values at inference) is the main cause of this adversarial vulnerability in the BatchNorm layer. We empirically proved this by experiments on various neural network architectures and datasets. Furthermore, we introduce Robust Normalization (RobustNorm) and experimentally show that it is not only resilient to adversarial perturbation but also inherit the benefits of BatchNorm.
references.authors: Devansh Arpit Yingbo Zhou Bhargava U Kota Venu Govindaraju
references.title: Normalization propagation: A parametric technique for removing internal covariate shift in deep networks
references.venue: arXiv preprint arXiv:1603.01431,
references.authors: Nils Bjorck Carla P Gomes Bart Selman Kilian Q Weinberger
references.title: Understanding batch normalization
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Nicholas Carlini David Wagner
references.title: Towards evaluating the robustness of neural networks
references.venue: IEEE Symposium on Security and Privacy (SP),
references.authors: Gavin Weiguang Ding Kry Yik Chau Lui Xiaomeng Jin Luyu Wang Ruitong Huang
references.title: On the sensitivity of adversarial robustness to input data
references.authors: Laurent Dinh Razvan Pascanu Samy Bengio Yoshua Bengio
references.title: Sharp minima can generalize for deep nets
references.venue: In Proceedings of the 34th International Conference on Machine Learning-Volume
references.authors: Christian Etmann Sebastian Lunz Peter Maass Carola-Bibiane Schönlieb
references.title: On the connection between adversarial robustness and saliency map interpretability
references.authors: Angus Galloway Anna Golubeva Thomas Tanay Medhat Moussa Graham W Taylor
references.title: Batch normalization is a cause of adversarial vulnerability
references.authors: Ian J Goodfellow Jonathon Shlens Christian Szegedy
references.title: Explaining and harnessing adversarial examples
references.venue: arXiv preprint arXiv:1412.6572,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Sergey Ioffe
references.title: Batch renormalization: Towards reducing minibatch dependence in batch-normalized models
references.venue: In Advances in neural information processing systems,
references.authors: Sergey Ioffe Christian Szegedy
references.title: Batch normalization: Accelerating deep network training by reducing internal covariate shift
references.venue: arXiv preprint arXiv:1502.03167,
references.authors: J”orn-Henrik Jacobsen Jens Behrmann Richard Zemel Matthias Bethge
references.title: Excessive invariance causes adversarial vulnerability
references.venue: arXiv preprint arXiv:1811.00401,
references.authors: Jörn-Henrik Jacobsen Jens Behrmannn Nicholas Carlini Florian Tramer Nicolas Papernot
references.title: Exploiting excessive invariance caused by norm-bounded adversarial robustness
references.authors: Nitish Shirish Keskar Dheevatsa Mudigere Jorge Nocedal Mikhail Smelyanskiy Ping Tak Peter Tang
references.title: On large-batch training for deep learning: Generalization gap and sharp minima
references.venue: arXiv preprint arXiv:1609.04836,
references.authors: Alex Krizhevsky
references.title: Learning multiple layers of features from tiny images
references.venue: Technical report, Citeseer,
references.authors: Alexey Kurakin Ian Goodfellow Samy Bengio
references.title: Adversarial machine learning at scale
references.venue: arXiv preprint arXiv:1611.01236,
references.authors: Hao Li Zheng Xu Gavin Taylor Christoph Studer Tom Goldstein
references.title: Visualizing the loss landscape of neural nets
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Xiang Li Shuo Chen Xiaolin Hu Jian Yang
references.title: Understanding the disharmony between dropout and batch normalization by variance shift
references.venue: arXiv preprint arXiv:1801.05134,
references.authors: Ping Luo Xinjiang Wang Wenqi Shao Zhanglin Peng
references.title: Towards understanding regularization in batch normalization
references.authors: Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras Adrian Vladu
references.title: Towards deep learning models resistant to adversarial attacks
references.venue: arXiv preprint arXiv:1706.06083,
references.authors: Anh Nguyen Jason Yosinski Jeff Clune
references.title: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Nicolas Papernot Patrick McDaniel Xi Wu Somesh Jha Ananthram Swami
references.title: Distillation as a defense to adversarial perturbations against deep neural networks
references.venue: In 2016 IEEE Symposium on Security and Privacy (SP),
references.authors: Tiberiu Popoviciu
references.title: Sur les équations algébriques ayant toutes leurs racines réelles
references.venue: Mathematica, 9:129–145,
references.authors: Tim Salimans Durk P Kingma
references.title: Weight normalization: A simple reparameterization to accelerate training of deep neural networks
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Shibani Santurkar Dimitris Tsipras Andrew Ilyas Aleksander Madry
references.title: How does batch normalization help optimization
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Carl-Johann Simon-Gabriel Yann Ollivier Léon Bottou Bernhard Schölkopf David Lopez-Paz
references.title: Adversarial vulnerability of neural networks increases with input dimension
references.venue: arXiv preprint arXiv:1802.01421,
references.authors: Karen Simonyan Andrew Zisserman
references.title: Very deep convolutional networks for large-scale image recognition
references.venue: arXiv preprint arXiv:1409.1556,
references.authors: Florian Tramèr Alexey Kurakin Nicolas Papernot Ian Goodfellow Dan Boneh Patrick McDaniel
references.title: Ensemble adversarial training: Attacks and defenses
references.venue: arXiv preprint arXiv:1705.07204,
references.authors: Cihang Xie Yuxin Wu Laurens van der Maaten Alan L Yuille Kaiming He
references.title: Feature denoising for
references.venue: Vision (ECCV),
sections.heading: 1 INTRODUCTION
sections.text: In spite of their impressive performance on challenging tasks in computer vision such as image classification and semantic segmentation, deep neural networks (DNNs) are shown to be highly vulnerable to adversarial examples, i.e. carefully crafted samples which look similar to natural images but designed to mislead a trained neural network model (Goodfellow et al., 2014; Nguyen et al., 2015; Carlini & Wagner, 2017). Designing defense mechanisms against these adversarial perturbations has been subjected to much research recently (Xie et al., 2019; Madry et al., 2017; Tramèr et al., 2017; Papernot et al., 2016).
  Meanwhile, Batch Normalization (BatchNorm or BN) (Ioffe & Szegedy, 2015) has successfully proliferated throughout all areas of deep learning as it enables stable training, higher learning rates, faster convergence, and higher generalization accuracy. Initially, the effectiveness of the BatchNorm has been attributed to its ability to eliminate the internal covariate shift (ICS), the tendency of the distribution of activations to drift during training. However, later on, alternative reasons including avoiding exploding activations, smooth loss landscape, reducing the sensitivity to initialization, etc. have also been proposed as the basis of BatchNorm’s success (Santurkar et al., 2018; Bjorck et al., 2018; Luo et al., 2018).
  While there exist a plethora of reasons for the adversarial vulnerability of deep neural networks (Jacobsen et al., 2018; Simon-Gabriel et al., 2018), a recent study by Galloway et al. (2019) showed that BatchNorm is one of the reasons for this vulnerability. Specifically, they empirically showed that removing the BatchNorm layer enhances robustness against adversarial perturbations. However, removal of BatchNorm also means a sacrifice of benefits such as the use of higher learning rates, faster convergence, and significant improvement in the clean test set accuracy among many others.
  In this paper, we propose a new perspective regarding the adversarial vulnerability of the BatchNorm layer. Specifically, we probe why BatchNorm layer causes the adversarial vulnerability. We hypothesize that the use of different normalization statistics during training and inference phase (mini-batch statistics for training and moving average of these statistics also called tracking, at inference time) is the cause of this adversarial
  vulnerability of the BatchNorm layer. Our experiments show that by removing this part of the BatchNorm, the robustness of the network increases by 20%. Similarly, robustness can further be enhanced by up to 30% after adversarial training. However, by removing the tracking part, the test accuracy on the clean images drops significantly ( though better than without normalization). To circumvent this issue, we propose Robust Normalization (RobustNorm or RN). Our experiments demonstrate that RobustNorm not only significantly improve the test performance of adversarially-trained DNNs but is also able to achieve the comparable test accuracy to that of BatchNorm on unperturbed datasets. We perform numerical experiments over standard datasets and DNN architectures. In almost all of our experiments, we obtain a better adversarial robustness performance on perturbed examples for training with natural as well as adversarial training.
sections.heading: 2 BACKGROUND
sections.text: We consider a standard classification task for data, having underlying distribution denoted as D, over the pair of examples x ∈ Rn and corresponding true labels y ∈ {1, 2, ..., k} where k represents different labels. We denote deep neural network (DNN) as a function Fθ(x), where θ denotes trainable parameters. θ is learned by minimizing a loss function L(x, y) with training data x, y. The output of the DNN is a feature representation f ∈ Rd, that we give input to a classifier C : Rn → {1, 2, ..., k}. The objective of the adversary is to add the additive perturbation δ ∈ Rn under the constrain that the generated adversarial sample xadv = x+ δ that looks visually similar to the true image x, and for which the corresponding labels are not same i.e. C(x) 6= C(xadv). In this work, we have added the perturbation via following well-known adversarial attack approaches.
  Fast Gradient Sign Method: Given an input image x along with its corresponding true label y, FGSM Goodfellow et al. (2014) aims to generate the adversarial image xadv as,
  xadv = x+ · sign(∇xL(x, y), (1) where is the perturbation budget that is chosen to be sufficiently small. We use two of its variants: Gradient (Grad) where graidents are used and Gradient sign (GradSign) which is similar to 1.
  Basic Iterative Method (BIM): BIM (Kurakin et al., 2016) is a straight forward extension of FGSM, that applies it multiple times with a smaller step size. Specifically,
  x0adv = x, x N adv = Clipx, {x N−1 adv + α · sign(∇xL(x N−1 adv , y))} (2)
  where x0adv is the clean image and N denotes iteration number.
  Carlini-Wagner attack (CW): CW is an effective optimization-based attack model introduced by Carlini & Wagner (2017). It works by definining an auxilary variable ϑ and minimizes the following objective functions
  min ϑ ‖1 2 (tanh(ϑ) + 1)− x‖+ c.f(1 2 (tanh(ϑ) + 1)), (3)
  where 12 (tanh(ϑ) + 1)− x is the perturbation δ, c is a scalar constant, and f(.) is defined as:
  f(xadv) = max(Z(xadv)y −max{Z(xadv)k : k 6= y},−%)). (4)
  Here, % is to control the adversarial sample’s confidence and Zxadv are the logits values for class k. Projected Gradient Descent (PGD): PGD perturbs the true image x for total number of N steps with smaller step sizes. After each step of perturbation, PGD projects the adversarial example back onto the -ball of normal image x , if it goes beyond the -ball. Specifically,
  xNadv = Π(x N−1 adv + α.sign(∇xL(x N−1 adv , y))), (5)
  where Π is the projection operator, α is step size, and xN denotes adversarial example at the N -th step. We have used `∞ norm as a distance measure. Gaussian Noise For comparison purposes, we also have used Gaussian noise with 0 mean and 0.25 variance.
  2.0.1 ADVERSARIAL TRAINING:
  It has been shown that empirical risk minimization using only clean images for training can decrease the robustness performance of DNNs. A standard approach to achieve the adversarial robustness in DNNs is adversarial training which involves fitting a classifier C on adversarially-perturbed samples along with clean images. We have used PGD based adversarial training which has shown to be effective against many firstorder adversaries (Madry et al., 2017) unlike other methods which overfit for on a single attack.
sections.heading: 2.1 EXPERIMENTAL SETUP
sections.text: We have used two network architectures, Resnet He et al. (2016) with 20,38 and 50 layers and VGG Simonyan & Zisserman (2014) with 11 and 16 layers. We have used CIFAR10 and 100 datasets (Krizhevsky et al., 2009) for all the evaluations. We have used term natural training for training with clean images while adversarial training is done with PGD based method formulated by Madry et al. (2017). We have always used a learning rate of 0.1 except for no normalization scenarios where convergence is not possible with higher learning rates. In such cases, we have used a learning rate of 0.001. We decrease the learning rate 10 times after 120 epochs and trained all the networks for 164 epochs. For robustness evaluations, we have used =0.03/1 for most of the experiments and used 20 epochs for all the iterative attacks. We also have tested the model on different noise levels ranging from 0.003/1 to 0.9/1.
sections.heading: 3 HOW DOES BATCHNORM CAUSES ADVERSARIAL VULNERABILITY
sections.heading: 3.1 HOW BATCHNORM WORKS
sections.text: In this section, we briefly explain the working principle of BatchNorm layer. Broadly speaking, the BatchNorm layer has been introduced to circumvent the issue of internal covariate shift (ICS). Consider a minibatch B of size M , containing samples xm for m = 1, 2, ...,M . BatchNorm normalizes the mini-batch during training, by calculating the mean µβ and variance σ2β as follows:
  µB = 1
  M M∑ i=1 xi ; σB = √√√√ 1 M M∑ i=1 (xi − µB)2 + . (6)
  Activation normalization is then performed as,
  x̂i = xi − µB σB . (7)
  To further compensate for the possible loss of representational ability of network, BatchNorm also learns per-channel linear transformation as: yi = γx̂i + B, (8) for trainable parameters γ and β that represent scale and shift, respectively. These parameters are learnt using the same procedure, such as stochastic gradient descent, as other weights in the network. During training, the model usually maintains the moving averages of mini-batch means and variances(a.k.a. tracking), and during inference, uses these tracked statistics in place of the mini-batch statistics. Formally, tracking (moving average) of mean and variance, for scalar τ are given as,
  µ̂ = (1− τ)µ̂+ τµβ , σ̂2 = (1− τ)σ̂2 + τσ2B (9) For inference, we can write,
  ytest = xtest − µ̂
  σ̂ .γ + β. (10)
sections.heading: 3.2 BATCHNORM AND ADVERSARIAL VULNERABILITY
sections.text: Recently, Galloway et al. (2019) empirically showed that the accelerated training properties and occasionally higher clean test accuracy of employing BatchNorm in network come at the cost of low robustness to adversarial perturbations. While removing the BatchNorm layer may be helpful for robustness, it also means the loss of desirable properties of BatchNorm like very high learning rate, faster convergence, boost in test accuracy, etc. Therefore, it is pertinent to devise a normalization method that is not only robust to the adversarial perturbations but also inherits the desirable properties of the BatchNorm layer.
sections.heading: 3.3 DEVIL IS IN THE TRACKING
sections.text: In this section, we aim to investigate the reasons behind the adversarial vulnerability of the BatchNorm layer on the following two grounds;
  • We note that during training, mini-batch statistics are used to normalize activations as shown in Equations 6 and 7. Moving average of these statistics are also calculated during the training that is called tracking (shown in Equation 9). The tracked mean and variance are used in the inference step (Equation 10). In this way, different values for mean and variance are used during training and inference. We show this in Figure 1 where it is clear that batch statistics at the start of the training are very different from tracked values that are used at the inference.
  • Our second observation is based on the recent work of (Ding et al., 2019; Jacobsen et al., 2019). These works shed light on the link of the distributional shift in input data and robustness. Specifically, Ding et al. (2019) showed that adversarial robustness is highly sensitive to change in the input data distribution and prove that even a semantically-lossless shift on the data distribution could result in drastically different robustness for adversarially trained models.
  Based on the above two observations, we hypothesize that: Different values of first-order statistics (mean and variance) are used in the normalization layer for training and inference. This means different internal
  representations being used at training and inference time which causes drift in input distributions of these layers. Therefore, the tracking part is the main culprit behind the adversarial vulnerability of the BatchNorm layer.
  To prove our hypothesis, we have done extensive experiments. For each experiment, we train a neural network model with three different normalization layers: BatchNorm, BatchNorm without tracking, and no normalization. To prove the generality of our argument, we have used various architectures, depths, and datasets as written in section 2.1. We train these networks on clean images as well as with based adversarial training procedure. For adversarial training, we use PGD attack for perturbation. We choose PGD due to its ability to generalize well for other adversarial attacks.
  Table 1 shows our results on Resnet20. For detailed experimental results on various architectures, depths and dataset with different attacks see Table 4, 5 in appendix. The results clearly show that while the elimination of BatchNorm and training at a very small learning rate can help increase robustness, it also reduces clean data accuracy (with BatchNorm). More importantly, this proves our hypothesis that by removing tracking, we can increase the robustness of a neural network significantly. By using BatchNorm without tracking, we also keep many benefits of BatchNorm. Unfortunately, by eliminating the tracking part of BatchNorm, clean accuracy of a network also reduces as compared to clean accuracy with BatchNorm. We tackle this issue in the next section.
sections.heading: 4 ROBUST NORMALIZATION
sections.text: Although alleviation of ICS was claimed reason for the success of BatchNorm, recently, Bjorck et al. (2018) have shown that BatchNorm works because it avoids activation explosion by repeatedly correcting all activations. For this reason, it is possible to train networks with large learning rates, as activations cannot grow uncontrollably and convergence becomes easier. On a different side, recent work on robustness has shown a connection between the removal of outliers in activations and robustness (Xie et al., 2019; Etmann et al., 2019). Based on these observations, we use min-max rescaling that is often employed in preprocessing. This is also useful in the elimination of outliers since it rescales the data to a specific scale, Minmax normalization is defined as;
  yi = xi −min(xi)
  max(xi)−min(xi) . (11)
  However, we experimentally found this layer to be less effective in terms of convergence. Considering the importance of mean (Salimans & Kingma, 2016), we modify this to;
  yi = xi − µ
  max(xi)−min(xi) . (12)
  We empirically observe the effectiveness of Equation 12 over Equation 11 but the overall performance was still inadequate. During debugging, we found that Equation 12 suppress activations much stronger than BatchNorm. This can also be seen from Popoviciu’s inequality (Popoviciu, 1935),
  4σ2 ≤ (max(xi)−min(xi))2. (13)
  Following Popoviciu’s inequality, we introduce the hyperparameter 0 < p < 1, that reduces the denominator in Equation 12,
  yi = xi − µ
  (max(xi)−min(xi))p (14)
  We experimentally found that p = 0.2 value generalizes well for many networks as well as datasets. We call this normalization Robust Normalization (RobustNorm or RN) due to its robustness properties. We do not use tracking for the RobustNorm. But for comparison purposes, we keep running average of both mean and denominator and use this running average during inference and call this normalization RobustNorm with tracking.
  Table 3 shows the accuracy of RobustNorm on Resnet20 for clean as well as adversarial training with both CIFAR10 and CIFAR100 datasets with 95% confidence interval calculated over 5 random restarts. These results show a better clean accuracy of RobustNorm in both natural and adversarial training scenarios. Apart from this, RobustNorm with tracking also shows better performance compared to BatchNorm with tracking. For adversarial robustness, we have shown Figure 2 with different attacks on CIFAR100 dataset. From the Table 3 and Figure 2, it is clear that RobustNorm keeps its clean accuracy while being more robust. For more results on Resnet38, Resnet50, VGG11, and VGG16 with CIFRAR10 and CIFAR100 datasets and both natural and adversarial training and many attack methods, please have a look at Table 4 and 5.
  Figure3 shows the evolution of validation loss and accuracy for PGD based adversarial training with a confidence interval of 95% on Resnet20 architecture and CIFAR100 dataset. From Figure 8 in the appendix, it can also be seen that the evolution of training loss and accuracy is normal. But validation loss and accuracy for normalizations with tracking is much different for different random restarts. This can probably be explained based on flat and sharp minima attained by different normalizations as can be seen in loss landscape in Figure 9 in the appendix. For further discussion on the loss landscape, please see section B in appendix.
sections.heading: 4.1 RESISTANCE FOR DIFFERENT VALUES OF ADVERSARIAL PERTURBNESS
sections.text: To further understand the performance of RobustNorm under adversarial conditions, we run an experiment where values are increased for the test set. We train networks with BatchNorm, BatchNorm w/o Tracking and RobustNorm with Natural as well as PGD-`∞ based adversarial training and tested them on different values of . The results are shown in Figure 4. As increases, the robustness of neural network decreases but the robustness of neural network with RobustNorm is much higher than BatchNorm while also higher than BatchNorm w/o tracking. To see the effect of an increase in adversarial noise on CIFAR100 dataset, see Figure 6 in the appendix.
sections.heading: 5 IS TRACKING A NECESSARY EVIL?
sections.text: In the previous sections, we have empirically shown the wickedness of tracking in BatchNorm. But there is more to the story. One benefit of tracking that makes it a necessary evil in BatchNorm is its ability to have
  consistent results on small inference time batch sizes. For all the results mentioned in this paper, we have used 100 as inference time batch size. By removing tracking, we lose this ability as shown in Figure 5. Does this make tracking a necessary evil? Based on the observations in section 3, one straight forward solution to this problem could be the use of tracked values during training as well since it will change the issue of different distributions. This, however, causes the model to blow up as argued by Ioffe & Szegedy (2015); Ioffe (2017). To resolve this issue, one possible solution is Batch Renormalization like formulation (Ioffe, 2017) but it requires careful tuning of many hyperparameters which makes its use very difficult. Similarly, other possible solutions such as Arpit et al. (2016); Salimans & Kingma (2016); Wu & He (2018) have their own challenges as well. From these observations, we argue that this problem requires more investigation.
  It is also important to note that RobustNorm’s results are restored by increasing batch size by a small number. Similarly, tracking is also less harmful for RobustNorm and RobustNorm with tracking is still more robust while having all the benefits of BatchNorm.
sections.heading: 6 CONCLUSION
sections.text: Addition of maliciously crafted noise in normal inputs, also called adversarial examples has proven to be deceptive for neural networks. While there are many reasons for this phenomena, recent work has shown BatchNorm to be a cause of this vulnerability as well. In this paper, we have investigated the reasons behind this issue and found that tracking part of BatchNorm causes this adversarial vulnerability. Then, we showed that by eliminating it, we can increase the robustness of a neural network. Afterward, based on the intuitions from the work done for the understanding of BatchNorm, we proposed RobustNorm which has much higher robustness than BatchNorm for both natural as well as adversarial training scenarios. In the end, we have shown how tracking can be a necessary evil and argued that it requires further careful investigation.
sections.heading: A ROBUSTNORM
sections.text: A.1 ROBUSTNORM FOR OTHER ARCHITECTURES
  In this section, we provide more detailed results for our experiments for both CIFAR10 and CIFAR100 datasets. Table 4 and 5 shows detailed results.
  A.2 RESISTANCE TO INCREASE IN ADVERSARIAL NOISE
  In this section, we put results of increasing adversarial noise on CIFAR100 dataset.
  A.3 ANOTHER ASPECT OF ROBUSTNORM
  As we have discussed, ICS hypothesis has been negated by a few recent studies. One of these studies (Santurkar et al., 2018) suggested that based on the results, “ it might be valuable to perform a principled exploration of the design space of normalization schemes as it can lead to better performance.” In this way, we can see RobustNorm with tracking as a new normalization scheme which is based on alternative explanations yet having performance equal to BatchNorm which, in a way, weakens ICS hypothesis. See Figure 7 for a comparison of accuracies over different models for CIFAR10 and CIFAR100 datasets.
sections.heading: B LOSS LANDSCAPE
sections.text: In this section, we discuss possible reasons for the high variation of validation loss for a network with normalization with tracking. As shown in Figure 8, training loss of all the norms converges with similar
  fashion for all the random restarts but validation loss has a lot of variation over these restarts. In other words, the value of training loss is similar among many restarts while the same loss values change drastically for validation. To further understand it, we draw loss landscape of these networks using formulation given by Li et al. (2018b) in Figure 9. From these plots, we observe an interesting behaviour: networks having normalization without tracking(i.e. better robustness) tend to have sharp minima as can be seen in figures 9c, 9d, 9e, 9f while their counterparts have more flat loss landscape i.e. figures 9a, 9b, 9g, 9h. There is a long history of debate on generalization ability of sharp vs flat minima Hochreiter & Schmidhuber (1997); Keskar et al. (2016); Dinh et al. (2017). We think more work in this direction can lead to a better understanding of how BatchNorm causes this vulnerability.
sections.heading: C MORE EXPERIMENTS ON ROBUSTNORM
sections.text: In this section, we discuss some less interesting experiments done to understand the role of power in RobustNorm.
  C.1 EFFECT OF POWER ON ROBUSTNORM
  In this section, we show the effect of changing hyperparameter p for clean as well as robust accuracy. From figure 4 and 11, it can be seen that both robustness to many attacks as well as accuracy changing with the power. So it can be concluded that by tuning hyperparameters, we can get better results.