summaries: In the paper, authors investigate the questions of adversarial robustness through the lens of spatial frequencies. In particular, contrary to a popular misconception, adversarial examples are not always related to high frequency components. Instead, they can encompass a wide range of spatial frequencies and are largely dataset dependent. The paper further studies adversarial training using different frequencies to better understand an accuracy vs robustness tradeoff. Carefully crafted experiments validate their findings and suggest a more effective approach for adversarial training.   This paper presents a frequency-based understanding of adversarial examples in deep neural networks. The main observation is that the adversarial examples are neither in high-frequency or low-frequency components but dataset dependent. The authors also analyse the properties of training robust models with frequency constrains, and propose a frequency-based explanation for accuracy and robustness tradeoff. This paper intends to investigate and analyze the phenomena of adversarial examples through the perspective of Fourier Analyses. It claims three major contributions: 1) they claim that adversarial examples is neither high frequency nor low frequency. 2) they adapt adversarial training with Fourier Analyses 3) they provide a new framework to measure robustness This work explores adversraial robustness from a frequency perspective. While this had been done before, this work challenges the commonly held notions, that adversarial perturbations are mainly a high-frequency phenomenon. The authors provide insight that the frequency properties of adversarial examples are dependant on the underlying training dataset. Additionally, the impact of different frequency properties in adversarial training is explored.
abstractText: Adversarial examples pose a unique challenge for deep learning systems. Despite recent advances in both attacks and defenses, there is still a lack of clarity and consensus in the community about the true nature and underlying properties of adversarial examples. A deep understanding of these examples can provide new insights towards the development of more effective attacks and defenses. Driven by the common misconception that adversarial examples are high-frequency noise, we present a frequency-based understanding of adversarial examples, supported by theoretical and empirical findings. Our analysis shows that adversarial examples are neither in high-frequency nor in low-frequency components, but are simply dataset dependent. Particularly, we highlight the glaring disparities between models trained on CIFAR-10 and ImageNet-derived datasets. Utilizing this framework, we analyze many intriguing properties of training robust models with frequency constraints, and propose a frequency-based explanation for the commonly observed accuracy vs robustness trade-off.
references.authors: Antonio A. Abello Roberto Hirata Zhangyang Wang
references.title: Dissecting the high-frequency bias in convolutional neural networks
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops,
references.authors: N. Ahmed T. Natarajan K.R. Rao
references.title: Discrete cosine transform
references.venue: IEEE Transactions on Computers,
references.authors: Anish Athalye Nicholas Carlini David A. Wagner
references.title: Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples
references.authors: Rémi Bernhard Pierre-Alain Moëllic Martial Mermillod Yannick Bourrier Romain Cohendet Miguel Solinas Marina Reyboz
references.title: Impact of spatial frequency based constraints on adversarial robustness
references.venue: CoRR, abs/2104.12679,
references.authors: Nicholas Carlini David A. Wagner
references.title: Towards evaluating the robustness of neural networks
references.venue: CoRR, abs/1608.04644,
references.authors: Nicholas Carlini David A. Wagner
references.title: Adversarial examples are not easily detected: Bypassing ten detection methods
references.venue: Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security,
references.authors: Josue Ortega Caro Yilong Ju Ryan Pyle Sourav Dey Wieland Brendel Fabio Anselmi Ankit Patel
references.title: Local convolutions cause an implicit bias towards high frequency adversarial examples, 2021
references.authors: Francesco Croce Matthias Hein
references.title: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
references.venue: ArXiv, abs/2003.01690,
references.authors: Nilaksh Das Madhuri Shanbhogue Shang-Tse Chen Fred Hohman L. Chen M. Kounavis Duen Horng Chau
references.title: Keeping the bad guys out: Protecting and vaccinating deep learning with jpeg
references.venue: compression. ArXiv,
references.authors: Yingpeng Deng Lina J. Karam
references.title: Frequency-tuned universal adversarial attacks
references.venue: CoRR, abs/2003.05549,
references.authors: H. Drucker Y. Le Cun
references.title: Double backpropagation increasing generalization performance
references.venue: In IJCNN-91-Seattle International Joint Conference on Neural Networks, volume ii, pp. 145–150
references.authors: Gintare Karolina Dziugaite Zoubin Ghahramani Daniel M. Roy
references.title: A study of the effect of JPG compression on adversarial images
references.venue: CoRR, abs/1608.00853,
references.authors: I. Goodfellow Jonathon Shlens Christian Szegedy
references.title: Explaining and harnessing adversarial examples
references.venue: CoRR, abs/1412.6572,
references.authors: Chuan Guo Jared S. Frank Kilian Q. Weinberger
references.title: Low frequency adversarial perturbation
references.venue: In UAI,
references.authors: Andrew Ilyas Shibani Santurkar D. Tsipras Logan Engstrom Brandon Tran A. Madry
references.title: Adversarial examples are not bugs, they are features
references.venue: NeurIPS,
references.authors: Malhar Jere Sandro Herbig Christine H. Lind F. Koushanfar
references.title: Principal component properties of adversarial samples
references.venue: ArXiv, abs/1912.03406,
references.authors: Malhar Jere Maghav Kumar F. Koushanfar
references.title: A singular value perspective on model robustness
references.venue: ArXiv, abs/2012.03516,
references.authors: J.Z. Kolter Eric Wong
references.title: Provable defenses against adversarial examples via the convex outer adversarial polytope
references.venue: In ICML,
references.authors: Alex Krizhevsky
references.title: Learning multiple layers of features from tiny images
references.venue: Technical report,
references.authors: P. Lorenz Paula Harder Dominik Strassel Margret Keuper Janis Keuper
references.title: Detecting autoattack perturbations in the frequency
references.authors: Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras Adrian Vladu
references.title: Towards deep learning models resistant to adversarial attacks
references.venue: In International Conference on Learning Representations,
references.authors: Guillermo Ortiz-Jimenez Apostolos Modas Seyed-Mohsen Moosavi-Dezfooli Pascal Frossard
references.title: Hold me tight! Influence of discriminative features on deep network boundaries
references.venue: In Advances in Neural Information Processing Systems
references.authors: Nicolas Papernot P. Mcdaniel Xi Wu S. Jha A. Swami
references.title: Distillation as a defense to adversarial perturbations against deep neural networks. 2016
references.venue: IEEE Symposium on Security and Privacy (SP),
references.authors: Ramprasaath R. Selvaraju Abhishek Das Ramakrishna Vedantam Michael Cogswell Devi Parikh Dhruv Batra
references.title: Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization
references.venue: CoRR, abs/1610.02391,
references.authors: A. Shafahi Mahyar Najibi Amin Ghiasi Zheng Xu John P. Dickerson Christoph Studer L. Davis G. Taylor T. Goldstein
references.title: Adversarial training for free
references.venue: NeurIPS,
references.authors: Yash Sharma Gavin Weiguang Ding Marcus A. Brubaker
references.title: On the effectiveness of low frequency
references.venue: perturbations. ArXiv,
references.authors: Jiawei Su Danilo Vasconcellos Vargas K. Sakurai
references.title: One pixel attack for fooling deep neural networks
references.venue: IEEE Transactions on Evolutionary Computation,
references.authors: Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna D. Erhan I. Goodfellow R. Fergus
references.title: Intriguing properties of neural networks
references.venue: CoRR, abs/1312.6199,
references.authors: Florian Tramèr A. Kurakin Nicolas Papernot D. Boneh P. Mcdaniel
references.title: Ensemble adversarial training
references.venue: Attacks and defenses. ArXiv,
references.authors: Yusuke Tsuzuku Issei Sato
references.title: On the structural sensitivity of deep convolutional networks to the directions of fourier basis functions
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: H. Wang Cory Cornelius Brandon Edwards Jason Martin
references.title: Toward few-step adversarial training from a frequency perspective
references.venue: Proceedings of the 1st ACM Workshop on Security and Privacy on Artificial Intelligence,
references.authors: Haofan Wang Mengnan Du Fan Yang Zijian Zhang
references.title: Score-cam: Improved visual explanations via score-weighted class activation
references.venue: mapping. CoRR,
references.authors: Haohan Wang Xindi Wu Zeyi Huang Eric P. Xing
references.title: High-frequency component helps explain the generalization of convolutional neural networks
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020b
references.authors: Weilin Xu David Evans Yanjun Qi
references.title: Feature squeezing: Detecting adversarial examples in deep
references.venue: Recognition (CVPR),
references.authors: Yin Raphael Gontijo Lopes Jonathon Shlens Ekin D. Cubuk Justin Gilmer
references.title: A fourier
references.venue: neural networks. ArXiv,
references.authors: Wang
references.title: We borrow a simple metric the “Occlusion Score
references.authors: Wang
references.title: Dropped Freq: 32 - 48 Dropped Freq: 48 - 63 Figure A.18: ImageNet examples where image is reconstructed after dropping (zeroing) certain frequency bands which implies the importance of the frequency
references.venue: Original Image Dropped Freq:
sections.heading: 1 INTRODUCTION AND BACKGROUND
sections.text: Since the introduction of adversarial examples by Szegedy et al. (2014), there has been a curiosity in the community around the nature and mechanisms of adversarial vulnerability. There exists an ever-growing body of work focused on attacking neural networks starting with the simple FGSM (Goodfellow et al., 2015), followed by the advanced PGD (Madry et al., 2018), a stronger C&W attack (Carlini & Wagner, 2016), the sparser Deep Fool (Su et al., 2019) and recently even a parameter free Auto-Attack (Croce & Hein, 2020). These methods and algorithms are consistently countered by the adversarial defense community, starting with distillation-based methods (Papernot et al., 2016), logit-based approaches (Kannan et al., 2018), then moving on to the simple, yet powerful PGD training (Madry et al., 2018), ensemble-based methods (Tramèr et al., 2018) and various other schemes (Zhang et al., 2019; Xie et al., 2019). Despite the immense progress made by the field, there exist many unanswered questions and ambiguities regarding these methods and adversarial examples themselves. Several works (Athalye et al., 2018; Kolter & Wong, 2018; Croce & Hein, 2020; Carlini & Wagner, 2017) have raised doubts about the efficacy of many methods and have made appeals to the research community to be more vigilant and skeptical with new defenses.
  Meanwhile, there exists a thriving research corpus dedicated to deeply studying and understanding adversarial examples themselves. Ilyas et al. (2019) presented a feature-based analysis of adversarial examples, while Jere et al. (2019) presented preliminary work on PCA-based analysis of adversarial examples, which was followed up with Jere et al. (2020) offering a nuanced view of the same through the lens of SVD. Ortiz-Jimenez et al. (2020) derive insights from the margins of classifiers.
  Given the intriguing nature of adversarial examples, another way of examining them is through the signal processing perspective of frequencies. Tsuzuku & Sato (2019) first proposed a frequency framework by studying the sensitivity of CNN’s for different Fourier bases. Yin et al. (2019) then pursued a related direction where they explored the frequency properties of neural networks with respect to additive noise. Abello et al. (2021) explore how the frequency properties of the image itself affect the model’s outputs and robustness. Caro et al. (2021) studied whether convolution operations themselves have an intrinsic frequency bias. Guo et al. (2019) came up with the first variant of adversarial attacks which target the low frequencies and Sharma et al. (2019) strengthened this line of thought by showing that such attacks had a high success rate against adversarially defended
  models. Deng & Karam (2020) proposed a method of generating adversarial attacks in the frequency domain itself. Complementary to these, there have been efforts by Lorenz et al. (2021) and Wang et al. (2020a) in detecting or mitigating adversarial examples by training in the frequency domain.
  These works also analyzed the nature of adversarial examples under the purview of frequencies and tried to arrive at an explanation for their nature. Wang et al. (2020b) hypothesized how CNNs exploit high frequency components, leading to less robust models, which is also the primary argument for a class of pre-processing based defenses, e.g., those based on JPEG. Wang et al. (2020d) also had arguments in support of this conjecture, based on their analysis on CIFAR-10 (Krizhevsky, 2009). It is confounding that these results are at odds with the successful low frequency adversarial attacks by Sharma et al. (2019) and raises the pertinent question: What is the true nature of adversarial examples in the frequency domain? Our work challenges some pre-existing notions about the nature of adversarial examples in the frequency domain and arrives at a more nuanced understanding that is well rooted in theory and backed by extensive empirical observations spanning multiple datasets. Some of our observations overlap with insights from the concurrent work by Bernhard et al. (2021) and offers additional evidence in this ongoing debate. Based on these, we arrive at a new framework that explains many properties of adversarial examples, through the lens of frequency analysis. We also carry out the first detailed analysis on the behaviour of frequency-constrained adversarial training. Our key contributions can be summarized as follows:
  • We show that adversarial examples are neither high frequency nor low frequency phenomena. It is more nuanced than this dichotomous explanation.
  • We propose variations of adversarial training by coupling it with frequency-space analysis, leading us to some intriguing properties of adversarial examples.
  • We propose a new framework of frequency-based robustness analysis that also helps explain and control the accuracy vsṙobustness trade-off during adversarial training.
  The rest of the paper is organized as follows: we first start off with basic notations and preliminaries. Then we introduce our main findings about adversarial examples in frequency domain and subsequently present a detailed analysis about their properties, complemented by extensive experiments.
sections.heading: 2 PRELIMINARIES
sections.text: We denote a neural network with parameter θ by y = h(x; θ), which takes in an input image x ∈ RH×W (omitting the channel dimension for brevity) and outputs y ∈ RC where C is the number of classes. Let D and D−1 represent the forward Type-II DCT (Discrete Cosine Transform) (Ahmed et al., 1974) and its corresponding inverse. The DCT breaks down the input signal and expresses it as a linear combination of cosine basis functions. Its inverse recovers the input signal from this representation. For a 1-D signal, the kth-freq of x ∈ RN and its corresponding inverse is given by
  D(x)[k] = g[k] = N−1∑ n=0 xnλk cos (2n+ 1)kπ 2N , (1)
  D−1(x) = x[n] = N−1∑ k=0 g[k]λk cos (2n+ 1)kπ 2N , (2)
  where k = {0, 1, . . . , N − 1} and λk =  √
  1 N for k = 0√ 2 N else.
  (3)
  We denote an adversarial attack that is bound by budget by max ||δ||p≤ L(h(x+ δ; θ), y) (4)
  where L is the loss associated with the network and δ is the adversarial noise bounded under a defined Lp norm to be less than perturbation budget . We perform a standard PGD-style update (Madry et al., 2018) to solve this maximization problem via gradient ascent and for an attack bounded by an Lp norm and step size α, the adversarial noise is given by
  δ = argmax ||V ||p≤α
  V T∇xL(h(x; θ), y) (5)
  where V is the direction of steepest normalized descent. Now, to generate an adversarial example that consists of certain frequencies, we restrict its adversarial noise δ to a subspace S defined by S = Span{f1, f2, . . . , fk}, where fi are orthogonal DCT modes and k ≤ N ,
  δf = argmax ||V ||p≤α
  V TD−1(D(∇xL(h(x; θ), y)) M) (6)
  where Mz(X) = { 1 if D(Xz) ∈ S 0 if D(Xz) 6∈ S
  is the mask to select frequencies. (7)
  In our work, we consider the L∞ and L2 norms, solving for which gives us the update steps:
  δf = α · Sgn(D−1(D(∇xL M))) for L∞ and (8) δf = α ·D−1 ( D ( ∇xL M ||∇xL M ||2 )) for L2 (9)
  We refer to this method as DCT-PGD in the rest of the paper. Note that the manual step size selection of standard PGD is not always accurate, leading to discrepancies in robustness measures as illustrated in Lorenz et al. (2021). Hence, we provide our results and observations with a DCT version of Auto-Attack. Unless mentioned otherwise, we utilize the ResNet-18 architecture for all models in our experiments. We use the term adversarial training to refer to the method by Madry et al. (2018) for all models, with the exception for ImageNet models where we use Adversarial training for free method (Shafahi et al., 2019). We utilize L∞ norm with of 4/255 for TinyImageNet and ImageNet datasets and of 8/255 for CIFAR-10 in all our experiments. Exact training details are included in the Appendix A.2. The terms low frequencies refer to frequency bands 0 to 32 and high frequencies refer to frequency bands bands 33 to 63.
sections.heading: 3 WHY DO WE NEED A FREQUENCY PERSPECTIVE?
sections.text: The focus of the community has been mostly on generating adversarial examples which are indistinguishable to humans, but can easily fool models. This notion gave rise to the incorrect assumption that since these perturbations are imperceptible to humans and they generally have to be in higher frequencies. The assumption was solidified when various pre-processing defense methods like Gaussian blur and JPEG showed initial success, further adding to confirmation bias. The fallacy is a classic case of Post Hoc Ergo Propter Hoc, i.e., the outcome of events is influenced by the mere ordering. Most of these experiments were centered only around CIFAR-10 and one can easily observe that the efficacy of such methods are questionable when extended to larger datasets like ImageNet and TinyImageNet (e.g., see (Dziugaite et al., 2016; Das et al., 2017; Xu et al., 2018)). This incorrect assumption has also led to claims about adversarial training shifting the importance of frequencies from the higher to the lower end of the spectrum (Wang et al., 2020c;b). As we show in the subsequent sections, this is not entirely true.
  We contend that this entire framework of investigating adversarial examples (e.g., blocking high frequency components using Gaussian blur pre-processing) is flawed, as one cannot verify the converse setting of blocking low frequency components. This is because low frequency components are inherently tied with labels (Wang et al., 2020b), conflating the two phenomena. Contrary to these, we argue and show that adversarial examples are neither high frequency nor low frequency and are dependent on the dataset.
sections.heading: 4 NATURE OF ADVERSARIAL SAMPLES IN FREQUENCY SPACE
sections.heading: 4.1 NOISE GRADIENTS
sections.text: Measuring the change of output with respect to the input is a fundamental aspect of system design. Whether it is a controls circuit or a mathematical model, the measure dydx gives us valuable information about the working of the model. When the model in question is a black box, like a neural network, the measure is invaluable as often it is our only insight into the inner mechanisms of the model. In the case of a classifier, the measure dydx is a tensor that is the same size as the input, which tells us about the impact of each pixel in input x on the resulting output y. Drucker & Le Cun (1991)
  first applied this concept on neural networks and called them input gradients. Over the recent years, this measure and its variants have found a new home in the model interpretability community (Selvaraju et al., 2016; Wang et al., 2019), where it forms the bedrock for various improvements.
  Taking a cue from this, we propose to measure dydδ or Noise Gradients, which inform us about the regions of noise, which have maximal impact on the output y. In our work, we are more interested in the frequency properties of adversarial examples, and hence take this one step further and propose to measure the DCT of noise gradients, i.e., D ( dy dδ ) orD (∇δY ). In a sense, we are measuring the model’s reaction to different frequency components in the adversarial input. This tensor D (∇δY ) (which has same shape as input) will point us towards the specific frequencies that affect the output y of the model. To analyze the adversarial frequency properties of a given dataset, we calculate the average noise gradients with respect to the model, under both normal training and adversarial training paradigms. Once computed, it will paint a picture about the interplay of adversarial noise and frequencies.
sections.heading: 4.1.1 ANALYSIS OF NOISE GRADIENTS
sections.text: We define the quantity D (∇δY )f as the noise gradient at frequency f . Note that this quantity is useful because it differs from D(δ) by at most a constant multiple, i.e.,
  D(δ) ∝ D (∇δY ) (10)
  Proof. Let x̂ = x+ δ where δ is the adversarial noise, then
  ∇δY = ∇xY = ∇x̂Y and ∇δL = ∇xL = ∇x̂L (11)
  We have (from Appendix A.1), ∇xL ∝ ∇xY (12)
  From the definition of the PGD update step, we have
  δ = α · ∇xL = α · ∇δL (13) for some constant α. Taking DCT of both sides, and by the linearity of the DCT, we have
  D(δ) = D(α · ∇δL) = α ·D(∇δL), and therefore, D(δ) ∝ D(∇δL). (14) Now from equation 11 and equation 12 we get
  D(∇δL) ∝ D(∇δY ) (15) Using this in equation 14, we have
  D(δ) ∝ D(∇δY ) (16)
  We see that the term D(∇δY ) corresponds to the frequencies that are affected by adversarial noise.
sections.heading: 4.1.2 EMPIRICAL OBSERVATION OF NOISE GRADIENTS
sections.text: We compute the average DCT of noise gradients over validation sets of TinyImageNet, CIFAR-10, and ImageNet datasets for models with normal and adversarial training under attack from a PGDbased L∞ adversary. The resulting tensors are visualized in Figure 1a. It shows the path taken by the PGD attack in the frequency domain under different scenarios for different datasets. We see that for normally trained CIFAR-10 models, the DCT of noise gradient activations are towards the higher frequencies and they gradually shift towards lower frequencies once the model is adversarially trained. Whereas for TinyImageNet and ImageNet models, we observe that the activations are already in lower-mid frequencies and adversarial training further concentrates them. These results clearly establish the following:
  • The DCT content of PGD attacks is highly dataset-dependent and we cannot make general arguments regarding frequency nature of adversarial samples just based on training.
  • The notion that adversarial training shifts the model focus from higher to lower frequencies is not entirely true. In many datasets, the model is already biased towards the lower end of the spectrum even before adversarial training.
  • To verify that this phenomenon is attributed to the dataset alone, we also observe similar behaviour across other architectures (Appendix A.5), across different image sizes (Appendix A.6) and for different attacks like L2 (Appendix A.7).
sections.heading: 5 MEASURING IMPORTANCE OF FREQUENCY COMPONENTS
sections.text: To examine the properties and behaviour of adversarial examples in the frequency domain, we also craft various empirical metrics that measure the importance of frequency components under various paradigms.
sections.heading: 5.1 IMPORTANCE BY VULNERABILITY
sections.text: We measure the importance of a frequency component by measuring the attack success rate when an adversarial attack is constrained to frequency f . Essentially, we are quantifying the importance by measuring the expected vulnerability of each frequency. This amounts to measuring the accuracy of h(x+δf ), where δf is the adversarial perturbation that is constrained to frequency f , obtained using the aforementioned DCT-PGD method. A lower accuracy of the model for a particular δf indicates a more important frequency f . In Figure 2, we visualize the accuracy of models with both normal training and adversarial training across different datasets under this setting. We see that only in the case of CIFAR-10, the trends for normal training and adversarial training are reversed, indicating that attacks constrained to higher frequencies are more successful for normal models, while lower frequency attacks are more effective on the adversarially trained models. In TinyImageNet and ImageNet datasets, we see that the overall trend remains same across the two training paradigms with adversarial training improving robustness across the spectrum. To obtain a high level view, we design another set of experiments where instead of attacking individual frequency components, we restrict the attack to frequency ranges (or bands, set of 16 equal divisions of the spectrum). The results of these under DCT-PGD version of Auto-Attack are shown in Figure A.7. In their work, Wang et al. (2020b) had claimed that low frequency perturbations cause visible changes in the image, thus defeating the purpose of imperceptibility clause of adversarial examples. However, we find that for
  larger datasets, such perturbations are imperceptible to a human. Example images have been shown in Appendix (Figure A.20 and A.21).
sections.heading: 5.2 IMPORTANCE DURING TRAINING
sections.text: With the objective of understanding the relative importance of frequency components while training, we formulate an experiment where we train models by masking out (making them zeros) frequency components of the input in a probabilistic manner and then using the trained model for normal inference. Example images when certain frequency bands are dropped is shown in Figure A.18. We train four types of models, where the frequency masking is restricted to four equal frequency bands and the amount of masking/dropping is controlled by a parameter p. This translates to training
  argmin θ L(h(xf̂ ; θ), y) (17) where xf̂ = D −1(M D(x)) (18)
  and Mz = { 1 z ∼ Up ∧ z ∈ [f1, f2..., fk] 0 else is the Mask generated using p (19)
  where xf̂ is the input constrained to a particular frequency band within the range [f1, f2, · · · fk]. While training, we select the frequencies to be dropped using a random uniform distribution U , with the percentage of dropping controlled by parameter p. A value of p = 1 indicates all frequencies in the specified band are set to zero. We train a total of 36 models per dataset, encompassing 9 different drop rates (p values) and 4 frequency bands. The experiment is repeated across datasets and the results are shown in Figure 3. As expected, we observe that a higher drop rate leads to lower accuracy. We also see that across datasets, high drop rates in low frequency band of 0-15 affects the model more. This behaviour is expected as lower frequencies have a strong relation with the labels (Wang et al., 2020b) and their extreme dropping leaves the model with little information to learn from. But if we observe the degree to which it affects the performance, we see disparities between the datasets. For example, the model trained on CIFAR-10 experiences a mere∼2% drop even when 90% of frequencies in the low band (frequencies 0-15) are dropped. Under the same condition, the model on TinyImageNet experiences ∼10% drop and the model on ImageNet experiences a whopping ∼35% drop in accuracy, highlighting the relative importance of these frequency bands. Also, note how very high drop rates in the highest frequency bands (frequencies 48-63) have little to no effect in non CIFAR-10 models.
sections.heading: 6 ADVERSARIAL TRAINING WITH FREQUENCY-BASED PERTURBATIONS
sections.text: Till now, we have analyzed the frequency properties of the model across datasets. In all experiments so far, we merely observed how the model reacts to adversarial perturbations under various frequency constraints. To further understand the properties of robustness in the frequency domain, we propose to train models with adversarial perturbations restricted to these frequency subspaces, a first of its kind. The training follows
  min θ max ||δf ||p≤
  L(h(x+ δf ; θ), y) (20)
  where δf is adversarial noise restricted to a frequency subspace defined by f . To obtain a high-level view of the process, we first train models adversarially with frequencies restricted to four equal
  frequency bands, ranging from low to high. Predictably, the models perform well when adversarial PGD attack is also restricted to the same frequency bands. The resulting robustness heatmap of attacks across the spectrum is shown in first column of Figure 4. For a more fine-grained view of the same, we adversarially train 64 models for each dataset, by perturbing each individual frequency. Then we adversarially attack these models in every frequency to produce a robustness heatmap, shown in the second column of Figure 4. In their work, Yin et al. (2019) had claimed that training with low-frequency perturbations did not help the model to be robust against those frequencies. Their analysis was not based on adversarial perturbations, but their claim was generalized. This effect was not observed in our experiments. We see that the model has good robustness when trained and tested against low-frequency perturbations, across datasets. The diagonals of the robustness heatmaps tell us that models perform well against an adversary constrained to the same frequency used for training. Moreover, we also see that models trained with perturbations restricted to mid/higher frequencies can withstand attacks from a fairly broad range of frequencies compared to models trained with lower frequency perturbations. Now that we have established this new training paradigm, we explore its various nuances and intriguing properties.
sections.heading: 6.1 THE UNEQUAL EPSILON DISTRIBUTION
sections.text: Do all frequencies have the same impact in adversarial training? To answer this question, we modify the construction of adversarial perturbation δ by weighing contributions from different frequency components and manipulating the value of they receive. It follows
  δ = K∑ i=0 ηi · sgn(∇xL)i for L∞ norm (21)
  ηi = K − i (22)
  where K is the number of equal frequency bands (four in our case) and η is a linear scaling parameter. This setting effectively translates to giving more importance to perturbation in one frequency space over the other. We train 2 models: One as described by equation 22, favoring lower frequency bands and then its complement, by reversing η and favoring higher frequency bands. For these ex-
  CIFAR-10
  TinyImageNet
  ImageNet
  periments, we employ Free adversarial training by Shafahi et al. (2019). The plot of PGD and clean accuracy during training are shown in Figure 5. We see that the model in which lower frequencies are favoured acts closest to standard PGD-based adversarial training. This shows that for a model to be robust, it only needs to be adversarially trained in the frequencies that matter most and not the entire spectrum. But at the same time, we see that the model where high frequency perturbations are favoured shows superior clean accuracy in all datasets except CIFAR-10. These results tell us that frequency based perturbations are intricately tied with clean accuracy and robustness of a model. We explore this in detail in the next section.
sections.heading: 6.2 ACCURACY VS ROBUSTNESS: AN ALTERNATIVE PERSPECTIVE
sections.text: Building on top of previous results, we design an experiment to examine the accuracy vs robustness trade-off that is commonplace while training robust models. We introduce a parameter λ that controls the weight given to frequency components in the perturbation during adversarial training. The update step for PGD under L∞-norm now looks like:
  δ = λ · [ α · sgn(∇xLLF) ] + (1− λ) · [ α · sgn(∇xLHF) ] (23)
  where ∇xLLF and ∇xLHF are gradients restricted to low (frequencies 0-31) and high frequencies (frequencies 32-63) respectively. We adversarially train ten different models by varying the value of λ and show their clean and robust accuracy in Figure 6. We see that in the case of TinyImageNet and ImageNet, the clean accuracy decreases when we train with low frequency perturbations, while increasing robustness. In case of CIFAR-10, we see that there is an initial increase in robustness followed by a steep fall. This is because higher frequencies have a significant role in adversarial robustness for this dataset, which is not achieved when λ values are high. We also observe a steep fall in robustness for ImageNet at λ of 0.9. This is because the frequency importance is distributed in the low-mid range for ImageNet (Figure 1a) and very high λ values tend to ignore the 32-48 frequency bands. These results establish that robustness and clean accuracy of an adversarially trained model are dependent on the frequencies we perturb. The λ parameter gives us control over the trade-off, enabling us to be more prudent while designing architectures and training regimes that demand a mix of clean accuracy and robustness.
sections.heading: 7 CONCLUSION
sections.text: In this paper, we analyze adversarial robustness through the perspective of spatial frequencies and show that adversarial examples are not just a high frequency phenomenon. Using both theoretical and empirical results we show that constituent frequencies of adversarial examples are dependent on the dataset. Then we propose and study the properties of adversarial training using specific frequencies, which can be used to understand the accuracy-robustness trade-off. These results can be utilized to train robust models more quickly by focusing on the frequencies that matter most. We hope that our findings will resolve some misconceptions about the frequency content of adversarial examples and aid in creating more robust architectures.
sections.heading: 8 ETHICS
sections.text: Adversarial examples pose a unique challenge to real world deep learning systems. We believe that our analysis will aid in the development of adversarial attacks as well as robust architectures. While we are aware of the potential for malicious uses of both of these applications we find minimal direct ethical concerns with the work in this paper. It is our hope that our work will only provide a deeper understanding of what constitutes an adversarial example and the mechanisms behind adversarial training in order to guide future research in this area.
sections.heading: A APPENDIX
sections.heading: A.1 PROOFS
sections.text: Here are the proofs for some results from above. In equation 11 we mentioned
  ∇δY = ∇xY = ∇x̂Y (24)
  Consider a neural network y = h(x; θ). Let the adversarial sample be x̂ = x + δ, where δ is the additive adversarial noise.
  y = h(x̂) = h(x+ δ) (25) dy
  dx = h(x+ δ)′ · 1 = dy dδ (26)
  dy dx̂ = h(x̂)′ = h(x+ δ)′ hence (27)
  dy dx̂ = dy dδ = dy dx or ∇δY = ∇xY = ∇x̂Y (28)
  In the same section’s equation 12 we also mentioned∇xL ∝ ∇xY .
  Let L = 1
  2 (h(x; θ)− ŷ)2 be the loss. (29)
  dL dx = (h(x; θ)− ŷ) · h(x; θ) ′ (30)
  here h(x; θ) ′ = dy
  dx and (h(x; θ)− ŷ) is a constant (31)
  dL dx = K · dy dx which implies (32)
  ∇xL ∝ ∇xY (33)
sections.heading: A.2 TRAINING DETAILS
sections.text: We utilize ResNet-18 in all our experiments (unless stated otherwise). For ImageNet and TinyImageNet datasets, we train for a total of 100 epochs, with an initial learning rate of 0.1 decayed every 30 epochs, momentum of 0.9 and a weight decay of 5e-4. In Madry adversarial training for the same, we use an value of 4/255. Under adversarial training for free setting, we train both models for 25 epochs with learning rate decayed every 8 epochs and the m (repeat step) set to 4.
  For CIFAR-10, we train the model for total of 350 epochs, starting with a learning rate of 0.1, decayed at 150 and 250 epochs and use the same setting with an of 8/255 for Madry training. In adversarial training for free setting, we train the model for 100 epochs with learning rate decay every 30 epochs and the m value set to 8.
  We utilize the pretrained models provided by PyTorch for ImageNet normal models. All experiments involving ImageNet-based adversarial training were done using Adversarial training for free method, with total epochs of 25 and m value set to 4.
sections.heading: A.3 FREQUENCY RANGE-BASED PERTURBATIONS
sections.text: We revisit the results shown in Figure 2 and show the same in a broader sense by attacking different frequency ranges. The results under DCT-PGD based Auto-Attack are shown in Figure A.7. We can see that the trends which were observed and discussed in earlier sections remain unchanged.
sections.heading: A.4 WHAT DO FREQUENCY ATTACKS TARGET ?
sections.text: A natural question that might arise with respect to DCT-PGD paradigm is how can we be sure that there is proportionate distortion in the frequency space as well. ((Rephrase)) We can visualize this using simple properties of the DCT. Consider the 1-D DCT from above. Since it is a linear transform, we can rewrite it as :
  D(z) =WZ where W is the linear DCT transform on the input tensor Z (34) x̂ = x+ δ in DCT space becomes (35)
  W · x̂ =W · x+W · δ (36) (37)
  The elements ofW represent different standard DCT basis functions, such that the lower frequencies are in upper left corner and the higher frequencies are towards the lower right corner. For any
  element i that also represents a frequency component, we can say that:
  Wi · x̂i =Wi · xi +Wi · δi (38)
  Essentially, we see that in the frequency space, each component of the resulting adversarial example x̂ is linearly distorted by the corresponding frequency component of noise δ.
sections.heading: A.5 DOES MODEL MATTER?
sections.text: We run the experiments across VGG-16 to confirm that the above trends are model agnostic and aren’t just limited to ResNet style architectures. In the results shown in Figure A.9 we see that the trends remain unchanged across datasets.
sections.heading: A.6 DOES IMAGE SIZE MATTER?
sections.text: To confirm that the anomalies of adversarial examples are indeed due the underlying dataset and not just the size, we repeat the experiment by training models where ImageNet and TinyImagenet images are resized to smaller sizes using bicubic filter. The average noise gradients calculated from these models are shown in Figure A.15.
sections.heading: A.7 L2-BASED ADVERSARIAL ATTACKS
sections.text: We repeat the same experiments to calculate noise gradients under the L2 attack. We do not observe any divergent behaviour, compared to L∞ attack. The results across datasets are shown in Figure A.8.
sections.heading: A.8 EXTENDING TO MORE DATASETS
sections.text: We also repeat the experiments across datasets, including non-ImageNet derived datasets like MNIST, Fashion-MNIST and CIFAR-100.The results are shown in Figure A.10.
sections.heading: A.9 EFFECT OF AUTO-ATTACK
sections.text: We calculate and plot the noise gradients for all models under Auto-attack setting. In general, there appears to be no significant difference when compared to results from PGD attack. The results are shown in figure A.11.
  0 20 40 60
  0
  10
  20
  30
  40
  50
  60
  Size = 64
  0 10 20 30
  0
  5
  10
  15
  20
  25
  30
  Size = 32
  0 5 10 15
  0
  2
  4
  6
  8
  10
  12
  14
  Size = 16
  Resized TinyImageNet
  0 100 200
  0
  50
  100
  150
  200
  Size = 224
  0 20 40 60
  0
  20
  40
  60
  Size = 64
  0 10 20 30
  0
  10
  20
  30
  Size = 32
  0 5 10 15
  0
  5
  10
  15
  Size = 16
  Resized ImageNet
  Figure A.15: Effect of resizing the images on TinyImageNet and ImageNet.
  0 20 40 60
  Frequencies
  0.0
  0.2
  0.4
  0.6
  O cc
  lu si
  on S
  co re
  CIFAR-10
  0 20 40 60
  Frequencies
  0.0
  0.1
  0.2
  0.3
  0.4
  0.5
  O cc
  lu si
  on S
  co re
  TinyImageNet
  0 20 40 60
  Frequencies
  0.0
  0.1
  0.2
  0.3
  0.4
  0.5
  O cc
  lu si
  on S
  co re
  ImageNet
  Normal Training PGD Adversarial Training
  Figure A.16: Occlusion Scores averaged over validation, across three datasets. Adversarial training is with L∞ norm with of 8/255 for CIFAR-10 and 4/255 for others.
sections.heading: A.10 CLASS WISE RESULTS
sections.text: We also investigate if there exists different frequency distribution for each class in a dataset. We show these results for CIFAR-10 (Fig A.12), MNIST (Fig A.13) and Fashion-MNIST (Fig A.14) datasets, for both normally trained and adversarially trained models. Apart from subtle differences, we do not see any general shift in the trends and observations.
sections.heading: A.11 OCCLUSION SCORE
sections.text: We borrow a simple metric the “Occlusion Score” from Wang et al. (2020c). Given a network h(x) and an image x, the occlusion score Of (x) for frequency f on class c is defined as:
  Of (x) = |h(x)c − h(xf̂ ) c| (39)
  where xf̂ refers to the input image x with the frequency f̂ removed from the spectrum. A higher score indicates that there is a drop in model accuracy when that particular frequency is removed,
  which implies the importance of the frequency. In their paper, Wang et al. (2020c) show the results of this metric on CIFAR-10 dataset and incorrectly conclude that adversarial training tends to shift attribution scores from higher frequency regions to lower frequency regions. We show that this is not the case by simply extending the experiment to include TinyImageNet and ImageNet datasets From the results shown in A.16, we can clearly see that in non CIFAR datasets, the attribution scores are already skewed towards lower frequencies and the shift after adversarial training happens across all frequencies.
sections.heading: A.12 EXAMPLES OF FREQUENCY-BASED PERTURBATIONS
sections.text: We show example images under different perturbation budgets of L∞ norm, across datasets in Figures A.19, A.20 and A.21. We also show examples of images when certain frequency bands are dropped A.18 and the complementary case of including only specified frequencies A.17.