summaries: This paper proposes a set of tools to probe spectral bias of deep neural networks, that is their tendency to learn low frequency, simpler functions earlier in training, whereas high frequencies are fit later. Authors propose adding noise to the labels through a target function via label smoothing, where the frequency and direction of the target function in image space can be varied. Moreover, they introduce a linear interpolation technique on validation data to probe the smoothness of the learned function along paths connecting natural images. Authors perform extensive experiments to demonstrate how the proposed tools can be used to investigate the spectral effect of training parameters such as model size and different forms of explicit and implicit regularization. This work presents an empirical study of how different training aspects affect the spectral bias of neural networks in practice. To that end, the authors propose to inject label noise of different frequencies to the CIFAR10 dataset and suggest that the time a neural network takes to start overfitting this noise is a good metric of its spectral bias. They also propose to measure the variability of the loss landscape in the linear interpolation path between two images as proxy for spectral bias. Experiments on the effect of model architecture, explicit regularization, and data augmentation suggest that deep neural networks exhibit a strong spectral bias, in practice, which can be modulated by different design factors. This paper proposes two methods for estimating smoothness of the prediction function learned by a deep network :   1/ an extension of Rahaman et al. to multiclass classification, by adding a sine of desired frequency to the target vector, then training on the new target vectors. The "effective noise fitting" is then the difference between the validation loss on clean targets and the validation loss on modified targets.  2/ a measure of change of the logits in l2 norm between 2 training examples, averaged over many examples  Using their methods, they show the effect on the smoothness of the predicted function of:  - varying the number of parameters  - adding explicit regularization (weight decay/mixup)  - distillation The paper focuses on the spectral bias of neural networks, i.e., their tendency to learn first the low-frequency information. The goal of this work is to extend the spectral bias into practical image recognition networks, i.e., beyond the fully-connected nets and the NTK regime that it was previously studied. To that end, the paper conducts a number of studies in both explicit and implicit regularization schemes used in practice, and provides links for the success of model distillation. 
abstractText: Despite their ability to represent highly expressive functions, deep learning models trained with SGD seem to find simple solutions that generalize surprisingly well. Spectral bias – the tendency of neural networks to prioritize learning low frequency functions – is one possible explanation for this phenomenon, but so far spectral bias has primarily been observed in theoretical models and simplified experiments. In this work, we propose methodologies for measuring spectral bias in modern image classification networks. We find that these networks indeed exhibit spectral bias, and that interventions that improve generalization sometimes increase and sometimes decrease the frequencies of the learned function. For example, we experimentally show that larger models learn high frequencies more readily than smaller ones, but many forms of regularization, both explicit and implicit, inhibit the learning of high frequencies. We also explore the connections between function frequency and image frequency and find that spectral bias is sensitive to the low frequencies prevalent in natural images. Our work enables measuring and ultimately influencing the spectral behavior of neural networks used for image classification, and is a step towards understanding why deep models generalize well.
references.authors: Ronen Basri David Jacobs Yoni Kasten Shira Kritchman
references.title: The convergence rate of neural networks for learned functions of different frequencies, 2019
references.authors: Ronen Basri Meirav Galun Amnon Geifman David Jacobs Yoni Kasten Shira Kritchman
references.title: Frequency bias in neural networks for input of non-uniform density, 2020
references.authors: Mikhail Belkin Siyuan Ma Soumik Mandal
references.title: To understand deep learning we need to understand kernel learning
references.venue: In International Conference on Machine Learning,
references.authors: Zac Cranko Simon Kornblith Zhan Shi Richard Nock
references.title: Lipschitz networks and distributional robustness, 2018
references.authors: Ekin D. Cubuk Barret Zoph Dandelion Mane Vijay Vasudevan Quoc V. Le
references.title: Autoaugment: Learning augmentation policies from data, 2019a
references.authors: Ekin D. Cubuk Barret Zoph Jonathon Shlens Quoc V. Le
references.title: Randaugment: Practical automated data augmentation with a reduced search space, 2019b
references.authors: Li Deng
references.title: The mnist database of handwritten digit images for machine learning research
references.venue: IEEE Signal Processing Magazine,
references.authors: Suriya Gunasekar Jason Lee Daniel Soudry Nathan Srebro
references.title: Implicit bias of gradient descent on linear convolutional networks, 2019
references.authors: Suriya Gunasekar Jason Lee Daniel Soudry Nathan Srebro
references.title: Characterizing implicit bias in terms of optimization geometry, 2020
references.authors: Moritz Hardt Benjamin Recht Yoram Singer
references.title: Train faster, generalize better: Stability of stochastic gradient descent, 2016
references.authors: Elad Hoffer Itay Hubara Daniel Soudry
references.title: Train longer, generalize better: closing the generalization gap in large batch training of neural networks, 2018
references.authors: Jason Jo Yoshua Bengio
references.title: Measuring the tendency of cnns to learn surface statistical regularities, 2017
references.authors: Nitish Shirish Keskar Dheevatsa Mudigere Jorge Nocedal Mikhail Smelyanskiy Ping Tak Peter Tang
references.title: On large-batch training for deep learning: Generalization gap and sharp minima, 2017
references.authors: Alex Krizhevsky Vinod Nair Geoffrey Hinton
references.title: Cifar-10 (canadian institute for advanced research)
references.venue: Globally-robust neural networks,
references.authors: Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras Adrian Vladu
references.title: Towards deep learning models resistant to adversarial attacks, 2019
references.authors: Horia Mania John Miller Ludwig Schmidt Moritz Hardt Benjamin Recht
references.title: Model similarity mitigates test set overuse, 2019
references.venue: In International Conference on Learning Representations,
references.authors: Hossein Mobahi Mehrdad Farajtabar Peter L. Bartlett
references.title: Self-distillation amplifies regularization in hilbert space, 2020
references.authors: Preetum Nakkiran Gal Kaplun Dimitris Kalimeris Tristan Yang Benjamin L. Edelman Fred Zhang Boaz Barak
references.title: Sgd on neural networks learns functions of increasing complexity, 2019
references.authors: Behnam Neyshabur Ryota Tomioka Nathan Srebro
references.title: In search of the real inductive bias: On the role of implicit regularization
references.venue: in deep learning,
references.authors: Behnam Neyshabur Srinadh Bhojanapalli David McAllester Nathan Srebro
references.title: Exploring generalization in deep learning, 2017
references.authors: Behnam Neyshabur Zhiyuan Li Srinadh Bhojanapalli Yann LeCun Nathan Srebro
references.title: Towards understanding the role of over-parametrization in generalization of neural networks, 2018
references.authors: Guillermo Ortiz-Jimenez Apostolos Modas Seyed-Mohsen Moosavi-Dezfooli Pascal Frossard
references.title: Hold me tight! influence of discriminative features on deep network boundaries, 2020a
references.authors: Guillermo Ortiz-Jimenez Apostolos Modas Seyed-Mohsen Moosavi-Dezfooli Pascal Frossard
references.title: Neural anisotropy directions, 2020b
references.authors: Samet Oymak Zalan Fabian Mingchen Li Mahdi Soltanolkotabi
references.title: Generalization guarantees for neural networks via harnessing the low-rank structure of the jacobian, 2019
references.authors: Nasim Rahaman Aristide Baratin Devansh Arpit Felix Draxler Min Lin Fred A. Hamprecht Yoshua Bengio Aaron Courville
references.title: On the spectral bias of neural networks, 2019
references.authors: Harshay Shah Kaustav Tamuly Aditi Raghunathan Prateek Jain Praneeth Netrapalli
references.title: The pitfalls of simplicity bias in neural networks, 2020
references.authors: Daniel Soudry Elad Hoffer Mor Shpigel Nacson Suriya Gunasekar Nathan Srebro
references.title: The implicit bias of gradient descent on separable data, 2018
references.authors: Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna Dumitru Erhan Ian Goodfellow Rob Fergus
references.title: Intriguing properties of neural networks
references.authors: Christian Szegedy Vincent Vanhoucke Sergey Ioffe Jonathon Shlens Zbigniew Wojna
references.title: Rethinking the inception architecture for computer
references.authors: DJ Tolhurst Y Tadmor Tang Chao
references.title: Amplitude spectra of natural images
references.venue: Ophthalmic and Physiological Optics,
references.authors: Ashia C Wilson Rebecca Roelofs Mitchell Stern Nathan Srebro Benjamin Recht
references.title: The marginal value of adaptive gradient methods in machine learning
references.venue: arXiv preprint arXiv:1705.08292,
references.authors: Zhi-Qin John Xu Yaoyu Zhang Yanyang Xiao
references.title: Training behavior of deep neural network in frequency domain, 2019
references.authors: Yoshihiro Yamada Masakazu Iwamura Takuya Akiba Koichi Kise
references.title: Shakedrop regularization for deep residual learning
references.venue: IEEE Access,
references.authors: Dong Yin Raphael Gontijo Lopes Jonathon Shlens Ekin D. Cubuk Justin Gilmer
references.title: A fourier perspective on model robustness in computer vision, 2020
references.authors: Sergey Zagoruyko Nikos Komodakis
references.title: Wide residual networks, 2017
references.authors: Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals
references.title: Understanding deep learning requires rethinking generalization, 2017
references.authors: Hongyi Zhang Moustapha Cisse Yann N. Dauphin David Lopez-Paz
references.title: mixup: Beyond empirical risk minimization, 2018
references.authors: Xiao Zhang Haoyi Xiong Dongrui Wu
references.title: Rethink the connections among generalization, memorization and the spectral bias of dnns
sections.heading: 1 INTRODUCTION
sections.text: Two fundamental questions in machine learning are why overparameterized models generalize and how to make them more robust to distribution shift. Resolving both of these questions requires understanding how complex our models should be.
  For instance, it is thought that overparameterized models generalize well because there are implicit regularizers that constrain the complexity of the learned functions. However, the precise nature of these implicit regularizers, and their importance in practice, remains unclear. As for how to achieve robustness, no consensus has emerged regarding function complexity. On the one hand, Cranko et al. (2018) and Leino et al. (2021) argue that Lipschitz smoothness of the learned function offers a guarantee of robustness; on the other, Shah et al. (2020) and Madry et al. (2019) argue that simplicity can be harmful and a robust model must actually be more complex than its non-robust counterpart.
  One window into function complexity is spectral bias – the tendency of neural networks to learn low frequency (simple and smooth) functions early in training, gradually increasing the frequency (complexity) of the learned function as training proceeds. Foundational work in this area has shown theoretical evidence of spectral bias by analyzing convergence rates of neural networks towards functions of different frequencies and showing that they converge to low frequency functions at a faster rate (Basri et al., 2019; Rahaman et al., 2019).
  In practice, spectral bias is difficult to measure: the most direct method involves taking a Fourier transform with respect to the input, which is expensive to compute due to the high dimensionality of images. Early experimental work focused on low-dimensional synthetic data (Basri et al., 2019; 2020) or used proxy measurements of spectral bias by inserting label noise of various frequencies during training (Rahaman et al., 2019). However, the label noise method is limited to binary classification/regression and involves modifying the training data, making it difficult to disentangle how other changes to the training data (such as data augmentation) affect the spectral content of the learned function. Thus, it remains an open question as to whether modern neural networks exhibit spectral bias and what role it plays in generalization.
  In this work, we investigate model complexity through the lens of spectral bias, by introducing experimental methods to study the frequency decomposition of the functions learned by modern image classification networks. We find that neither simplicity nor complexity is purely beneficial or detrimental to performance.
  Contributions. We extend the label noise procedure of Rahaman et al. (2019) to enable measuring spectral bias in multi-class classification and apply this technique to understand the function frequencies present in high-accuracy models on CIFAR-10 (Krizhevsky et al.). We also introduce a second method for measuring the smoothness of a learned function via linear interpolation between test examples. This offers a proxy measurement of spectral bias without the need to modify training labels, allowing us to probe the effects of the training data on the learned function frequencies.
  Using these experimental methods, we find that increasing the model size decreases spectral bias; larger models learn high frequencies faster. However, we observe that more accurate models are not always higher frequency. Several forms of explicit and implicit regularization, including weight decay, increasing the dataset size, and applying Mixup (Zhang et al., 2018) data augmentation, increase spectral bias and produce a lower-frequency learned function. Our experiments show that these common training interventions affect the frequency content of the learned function in different ways, helping us to understand their mechanism and offering a tool to aid design of improved training methods in the future.
  This perspective allows us to shed light on the mechanism behind self-distillation, in which a student model is trained to fit the predictions of a teacher model. Using our linear interpolation methodology, we observe that self-distillation produces a student model whose learned function is smoother than that of its teacher. This suggests that the teacher model acts as a sort of low-pass filter on the target function, perhaps making it more accessible to the spectrally-biased student.
  Finally, we explore the relationship between image frequency and function frequency. By further extending the label noise methodology of Rahaman et al. (2019) to study spectral bias in directions of interest through the input space, we find that models are most sensitive to the low image frequencies common in natural images.
sections.heading: 2 RELATED WORK
sections.text: Implicit bias. A common belief is that some form of implicit bias imposed by the training procedure may account for the generalization ability of overparameterized neural networks, and accordingly, much research has been directed towards understanding these implicit biases (Soudry et al., 2018; Zhang et al., 2017; Keskar et al., 2017; Gunasekar et al., 2019; 2020; Hardt et al., 2016; Hoffer et al., 2018; Belkin et al., 2018; Neyshabur et al., 2018; 2015; 2017; Mania et al., 2019; Gunasekar et al., 2019; Nakkiran et al., 2019). One of the most well-known forms of implicit bias comes from the optimization algorithm itself, which typically guides networks to solutions that generalize well (Soudry et al., 2018; Keskar et al., 2017; Wilson et al., 2017).
  Spectral bias. Spectral bias is a form of implicit bias, with much recent attention including theoretical and experimental approaches (Oymak et al., 2019; Xu et al., 2019; Rahaman et al., 2019; Basri et al., 2019; Zhang et al., 2021); we describe some of these prior results here. Basri et al. (2019) studied spectral bias using a linear model of SGD training dynamics to show that models learn low frequency (simple) functions early in training and then gradually learn higher frequencies as training proceeds, assuming training data that is distributed uniformly on the hypersphere. Basri et al. (2020) extended the analysis to consider nonuniformly spaced training data, and found that learning is faster where samples are denser; if sampling is nonuniform, then during training, the learned function will be higher frequency in regions with denser samples.
  Rahaman et al. (2019) used a more direct analysis to show the same spectral bias toward low frequency functions and also suggested that when training data lie on a manifold (as natural images are believed to), increasing the complexity of the manifold makes it easier for models to fit high frequency functions. Additionally, Rahaman et al. (2019) posited that high frequency components of the learned function are most sensitive to perturbations in the model parameters, connecting back to the idea of flat optimization minima (Neyshabur et al., 2017) or smooth loss landscape Mehmeti-Göpel et al. (2021). They proposed experimental methods to study spectral bias in image classification, but
  focused on a binary subset of the relatively simple MNIST dataset (Deng, 2012), with mean square error loss.
  Our label smoothing experiments are a direct extension of Rahaman et al. (2019) to multiclass classification with the more common cross-entropy loss. Our linear interpolation experiments are somewhat similar to the recently-proposed experimental methods in Zhang et al. (2021), except that we sample along paths between images rather than in regions surrounding each image; this allows for a more global measurement of spectral bias.
  Model sensitivity to image frequency. While we focus on function frequency in this work, prior research aimed to understand model sensitivity to image frequencies. Jo & Bengio (2017) found that CNNs are sensitive to Fourier statistics of the training data, even those irrelevant to human viewers. Ortiz-Jimenez et al. (2020b) studied the image frequency bias induced by using a convolutional architecture, and Ortiz-Jimenez et al. (2020a) argued that models are sensitive primarily to discriminative Fourier directions in the training data. They posit that adversarial training induces large margins in orthogonal directions, thereby improving robustness. Yin et al. (2020) introduced a procedure to measure the sensitivity of trained models to Fourier image perturbations, and showed that adversarial training and Gaussian data augmentation increase robustness to high frequency perturbations but increase sensitivity to low frequency perturbations. Our work adds to this line of research a study of the interplay between two notions of frequency: the function frequency involved in spectral bias, and the image frequencies present in the data.
sections.heading: 3 METHODOLOGY
sections.text: The goal of our work is to measure the complexity of neural network functions through the lens of frequency. In low dimensions, measuring the frequency decomposition of a function is straightforward and tractable: evaluate the function at dense, uniform sampling positions and compute its discrete Fourier transform. However, the function of interest in image classification is unavoidably highdimensional, mapping images (with thousands of pixel values) to object classes. It would be intractable even to collect sufficient samples of this function to compute a discrete Fourier transform, let alone compute the transform itself. Instead, we employ two complementary approaches to measure informative proxies of this frequency decomposition.
sections.heading: 3.1 LABEL SMOOTHING
sections.text: The core idea for measuring function frequency introduced by Rahaman et al. (2019) is to construct a sinusoid over the space of images, and to use that sinusoid as a form of label noise during training. Let D = {(Xi,yi)}ntraini=1 be the training examples and Dvalid = {(Xj ,yj)} nval j=1 be the validation examples, with Xi an image and yi a one-hot class encoding, where ntrain is the number of training examples, nval is the number of validation examples, d is the side length of an image (assumed to be square for simplicity), c is the number of color channels, and M is the number of classes, so Xi ∈ Rd×d×c and yi ∈ RM . To extend this procedure to the multi-class setting, we add noise of various frequencies to an M - dimensional label vector via label smoothing (Szegedy et al., 2015). Let S : Rd×d×c → [0, 1] be a target function that maps an input image Xi to a scalar value between 0 and 1. We apply label smoothing to each label yi, mapping it to ȳi = yi(1−S(Xi)) + 1M S(Xi) to retain a valid probability distribution. We then train from scratch using the original examples Xi and their smoothed labels ȳi. Finally, we evaluate on the validation images Xj , comparing to both their original one-hot labels yj and smoothed labels ȳj = yj(1− S(Xj)) + 1M S(Xj). Our experiments use the CIFAR-10 dataset (Krizhevsky et al.), where ntrain = 50000, nval = 10000, d = 32, c = 3, and M = 10; however, our experimental setup is sufficiently general as to transfer easily to any image classification task.
  This method is introduced in Figure 1. As training proceeds, we see both the training loss (between predictions and smoothed labels ȳi) and noisy validation loss (between predictions and smoothed labels ȳj) decrease. The clean validation loss (between predictions and one-hot labels yj), however, initially decreases but at some point in training plateaus or begins to increase. At this point, the model has begun to learn the target function S. Accordingly, we introduce noise fitting, defined as the difference between the validation loss on one-hot labels and the validation loss on labels smoothed with the same function S that was applied to the training labels:
  noise fitting = clean validation loss− noisy validation loss (1)
  We compare different models by computing the minimum noise fitting achieved throughout training. A model with higher min noise fitting more readily fits the target function S than a model with lower min noise fitting.
  By choosing different functions S, we can probe nuances of spectral bias. Typically (inspired by Rahaman et al. (2019)) we choose a radial wave: S(X) = sin(2πf(‖X‖ − ED‖X‖)), where we can vary the frequency f to understand spectral bias at this global scale. We can also choose more targeted functions S; for instance, if V is a direction of interest through the space of images (i.e. V is an image-shaped vector of unit norm), we can construct S(X) = sin(2πf 〈X,V〉). This allows us to vary both the frequency and direction of the target sinusoid, to understand model sensitivity along different directions through image space.
  However, label smoothing has a few limitations. It requires retraining each model from scratch, with substantial investment of time and computational resources. Because it involves perturbing the training labels, it cannot directly answer questions about the interaction between the training dataset and spectral bias, such as the effects of data augmentation. For the same reason, we cannot use this methodology to study the spectral decomposition of existing trained models.
sections.heading: 3.2 LINEAR INTERPOLATION
sections.text: To complement the label smoothing approach, we also propose a measurement methodology based on linear interpolation between validation images. Although we cannot take dense, regularly-spaced samples that cover the entire input (image) space, we can sample along specific paths through, and thereby glean glimpses into the spectral content of the learned function.
  We consider two types of paths: those between images from the same class, and those between images from different classes. When considering paths between images in the same class, we choose 200 random, distinct pairs of images from each of the 10 CIFAR-10 classes and average over the paths defined by these pairs. When interpolating between classes, we choose 200 random, distinct image pairs from each of the 45 (10 choose 2) pairs of distinct classes, and average over the resulting paths.
  For each pair of images (X0,X1), we vary λ ∈ [0, 1] to trace out a path, where each image in the path is given by Xλ = λX1 + (1− λ)X0. We choose λs so that the distance between adjacent images on the path is constant, which means that each path may produce a different number of samples depending on the total distance between X0 and X1. Before interpolating, we normalize X0 and X1. Figure 2 (Bottom Left) shows an illustrative example of two images and a corresponding interpolation path between them. For our experiments, we typically have 50 to 100 images along the interpolation path.
  Relationship between interpolation and label smoothing experiments. In order to validate our interpolation methodology, we consider the oracle function that maps directly from the interpolated image Xλ to the target label used in our label smoothing experiments. By increasing the frequency of the label smoothing noise, we create oracle functions with higher frequencies. We then verify that our interpolation measurement indeed measures higher frequencies on the higher frequency oracle functions.
  In Figure 2 (Left), we show an example where we vary the label smoothing frequency from f = 0 to f = 0.05. We plot the value of λ along the interpolating path on the x axis, and the norm of the difference between the model output on the interpolated image ŷλ and the model output on the original image ŷ0 on the y axis. The norm of the prediction difference allows us to visualize how smooth the function is along the interpolating path; smoother prediction difference norm indicates a lower frequency function.
  In order to quantify the frequency of the function along the interpolating path, we compute the per-class discrete Fourier transform (DFT). Letting {Xλt}Tt=1 denote the set of interpolated images, the DFT coefficient for frequency f is given by Ŷf [m] = ∑T t=1 ŷλt [m] exp(−i2πft), where i is the imaginary number and m ∈ {1, . . . ,M} denotes the class. In order to determine how the magnitudes of the DFT coefficients Ŷf are distributed between low and high frequencies, we quantify how much of the spectrum is high frequency by choosing a frequency threshold and computing the fraction of the total coefficient magnitudes above the threshold. In Figure 2 (Right) we show this summary metric averaged over all within-class interpolating paths for the oracle functions.
  Both sides of Figure 2 show that our interpolation methodology indeed measures the oracle functions that we constructed to be high frequency as high frequency. In Figure 2 (Left), the higher frequency oracle functions have less smooth prediction norm differences, and in Figure 2 (Right), they have a higher fraction of DFT coefficient magnitudes in the high frequencies.
sections.heading: 3.3 DATA AND MODELS
sections.text: Our experiments use the CIFAR-10 (Krizhevsky et al.) dataset of low-resolution natural images from ten classes, a mixture of animals and objects. Images are pixel-wise normalized by the mean and standard deviation of the training images. We consider six different convolutional neural networks that have achieved high accuracy on this task: two wide-resnets (Zagoruyko & Komodakis, 2017) of different sizes (wrn 32 and wrn 160), three shake-shake regularized networks (Gastaldi, 2017) of different sizes (shake shake 32, shake shake 96, and shake shake 112), and pyramid net, which uses the even stronger shake-drop regularization (Yamada et al., 2019). Our implementations are based
  on Cubuk et al. (2019a); we use the same optimizer (stochastic gradient descent with momentum) and cosine learning rate schedule. We train without data augmentation (to ensure all models are trained on exactly the same examples), except for experiments that explicitly vary data augmentation. Without data augmentation, the test accuracies of our models are: 89.4% (wrn 32), 90.2% (wrn 160), 92.2% (shake shake 32), 93.5% (shake shake 96), 93.6% (shake shake 112), and 95.8% (pyramid net).
sections.heading: 4 RESULTS
sections.text: Throughout this section we show representative examples from our experiments to highlight our main findings. Full results (on all six models we tested) are included in section A.
sections.heading: 4.1 SPECTRAL BIAS AND MODEL ARCHITECTURE
sections.text: We begin by applying our label smoothing methodology to find that modern image classification CNNs exhibit spectral bias, learning low frequency target functions early in training and learning higher frequency functions as training proceeds. Note that Figure 3 (Left) shows spectral bias over a small but illustrative range of frequencies (the full range of frequencies we tested was 0 to 0.1); target functions of sufficiently low frequency are fit almost immediately and target functions of sufficiently high frequency are never learned during the 250 epochs of training we tested.
  Although all models we tested exhibit spectral bias, we found that the precise nature of the bias depends on the choice of model. For example, with all else fixed, increasing the width of a model decreases its spectral bias, enabling it more readily fit target frequency functions. This trend is evident in Figure 3 for the wide-resnet family (Center) and the shake-shake family (Right). Results of our label smoothing experiment on all six models we tested are included in section A.2.
sections.heading: 4.2 SENSITIVITY TO NATURAL IMAGE DIRECTIONS
sections.text: Although label smoothing with a radial wave offers a convenient global picture of spectral bias, by replacing the radial wave with other target functions we can use the same methodology to test more localized aspects of spectral bias. We take a step in this direction by considering the family of target label smoothing functions S(X; k) = sin(2πf 〈X,Fk〉), where Fk is a diagonal Fourier basis image with frequency k and the same dimensions as X, visualized in Figure 4 (Bottom).
  We consider (a subset of) Fourier basis images because the Fourier spectra statistics of natural images are well studied (see, e.g. Tolhurst et al. (1992)): natural images tend to be composed of Fourier basis images with amplitude proportional to their inverse spatial frequency. Indeed, in Figure 4 we find that shake shake 96 is more sensitive to label smoothing in low image frequency directions.
  This finding is consistent with theoretical predictions of Basri et al. (2020) that models learn faster in regions of higher density of training examples: since low image frequencies are more common
  in natural images, the effective sampling density enjoyed by a label smoothing target is higher in these directions. It is also possible that this bias is inherent to the convolutional model architecture regardless of the data distribution, as discussed in Ortiz-Jimenez et al. (2020b), or that a combination of both effects is at play. Determining the precise cause of this image frequency finding is an interesting direction for further study.
sections.heading: 4.3 SPECTRAL BIAS AND REGULARIZATION
sections.text: We also found a relationship between regularization and spectral bias, which is explored in Figure 5 and Figure 6. Some forms of regularization, like weight decay and dropout, are amenable to study by either of our methodologies; Figure 5 demonstrates agreement between these methods in the context of varying weight decay. Further results showing agreement between these two measurement methodologies are presented in section A.4.
  In total we consider 4 types of (explicit and implicit) regularization: weight decay, training set size, data augmentation (Cubuk et al., 2019b;a; Zhang et al., 2018), and the strength of Mixup augmentation (Zhang et al., 2018). The latter three involve changes to the training data and are therefore only directly studied via our interpolation methodology, which we use in Figure 6.
  Weight Decay. Weight decay is a penalty on the square norm of the model parameters; a stronger penalty encourages a model to learn smaller-norm weights. Figure 5 shows that increasing the weight decay also encourages a model to learn high frequencies slower (Left) and to ultimately be lower-frequency (Right). This result is consistent with the expectation that smaller-norm weights yield a learned function with a smaller Lipschitz constant and thus smoother changes in output (as discussed e.g. in Szegedy et al. (2014)). This experiment also demonstrates the nuanced relationship between the bandwidth or smoothness of the learned function and its accuracy; shake shake 32 trained with weight decay 0.0005 or 0.001 achieves higher test accuracy compared to the same architecture trained with either more or less weight decay. Neither complexity nor smoothness is, in itself, purely beneficial or detrimental: the trick is to balance them appropriately for the dataset.
  Training Set Size. As shown in Figure 6 (Left) on wrn 160, we find that increasing the number of training examples produces a lower-frequency learned function. Note that all models in Figure 6 (Left) were trained for the same number of epochs and with the same batch size, so the models trained with more examples were also trained for more gradient steps (since each epoch contained more batches). The models trained with more examples therefore had ample opportunity to fit the high frequencies present in their counterparts trained with fewer examples, and yet did not. This result supports the idea that additional examples serve as an implicit regularization by forcing the model away from more complex solutions that would predict the added examples incorrectly.
  Data Augmentation. Data augmentation is a common strategy to increase the effective training set size without the expense of actually collecting additional examples. In Figure 6 (Center) we consider the effects of RandAugment (Cubuk et al., 2019b), AutoAugment (Cubuk et al., 2019a), and Mixup (Zhang et al., 2018) data augmentation strategies, each of which tends to improve the final accuracy of the trained model (with RandAugment producing the most substantial benefits in our experiments). RandAugment and AutoAugment generate new images by applying geometric and lighting transformations, whereas Mixup produces new images by linearly interpolating between pairs of existing images. We find that training with RandAugment or AutoAugment tends to make the learned function slightly less smooth between examples of the same class (Top). Mixup (at strength 0.1) makes the learned function substantially smoother, both within-class (Top) and between-class (Bottom). This experiment reinforces the nuanced relationship between function complexity and performance; a model can be improved by the addition of helpful complexity (as in RandAugment and AutoAugment, which encode robust transformation priors) or by the removal of unnecessary complexity (as in Mixup, which encodes a simple linear prior in unsupervised input regions).
  Mixup Strength. Mixup augmentation (Zhang et al., 2018) perturbs each batch of training data as follows: each example is matched with a partner based on a random permutation of the batch, and each example is then perturbed towards its partner by an interpolation amount λ drawn from a symmetric beta distribution (with the same λ used for all examples in the batch). We refer to the parameter of this beta distribution as the Mixup strength, as it controls the degree to which the augmented images tend to lie close to an original training image or close to the average of two training images. A parameter
  of 0 corresponds to no augmentation (λ = 0 or λ = 1, always using the original images), a parameter of 1 corresponds to the uniform distribution over λ ∈ [0, 1], and a parameter of ∞ corresponds to λ = 0.5, the exact midpoint between a pair of training images. Figure 6 (Center) uses Mixup strength 0.1 (one of the values recommended by Zhang et al. (2018)). Figure 6 (Right) shows that, as expected, increasing the strength of the Mixup augmentation generally makes the learned function lower-frequency, but that Mixup that is too strong can yield high frequencies. Interestingly, the impact of Mixup strength on the learned frequencies is different within versus between classes, suggesting that perhaps tuning this parameter separately for these two cases may prove beneficial.
sections.heading: 4.4 SELF-DISTILLATION
sections.text: In a sense another form of regularization, we finally consider the effect of self-distillation on the smoothness of the learned function. In self-distillation, a ‘teacher’ model is trained with some form of strong regularization, in our case a combination of weight decay and early stopping based on training loss. A ‘student’ model with the same architecture is then trained from scratch to fit the pseudolabels produced by the teacher, instead of the original one-hot training labels (alternatively, an interpolation between the pseudolabels and the original labels may be used). Prior research (Furlanello et al., 2018) found that this procedure can train student models that outperform both their teachers and a baseline (the same architecture trained to completion on the one-hot labels).
  Prior research has also sought to understand the mechanism behind knowledge distillation and self-distillation in particular. Mobahi et al. (2020) finds that self-distillation acts as a regularizer by limiting the basis functions available to the student to learn. We complement their theoretical work with our interpolation experimental methodology in Figure 7, where indeed we find that the student model learns a lowerfrequency function than its teacher, when both have the same training loss. We conjecture that this effect is analogous to low-pass prefiltering common in digital signal processing: a highfrequency target function that we cannot adequately sample is first smoothed (in this case by being approximated by a regularized teacher) and then it can be modeled via samples without further loss in fidelity. Without this prefiltering, our samples are inadequate to capture the complexity of the target function, so we reconstruct an imperfect version corrupted by aliasing.
sections.heading: 5 CONCLUSIONS
sections.text: In this paper, we introduced two methods to measure spectral bias in modern image classification neural networks, and applied these methods towards the central question:
  What kinds of function frequencies are needed for modern neural networks to generalize?
  Specifically, we applied these methods to examine the impact of a variety of training choices on the learned frequencies. Among training choices that improve validation accuracy, using a larger model, training longer, and using RandAugment (Cubuk et al., 2019b) or AutoAugment (Cubuk et al., 2019a) tend to increase the frequency of the learned function. However, regularizing with modest weight decay, training with more examples, using modest Mixup augmentation (Zhang et al., 2018), and performing self-distillation all improve validation accuracy while decreasing the frequency of the learned function. Our findings support the common wisdom that, while some complexity is necessary to fit the data, unnecessary complexity can harm generalization. Our experimental methods offer a valuable window into precisely what kind of function complexity we should strive for.
sections.heading: REPRODUCIBILITY STATEMENT
sections.text: An anonymized copy of our code is included in the supplement. Our work uses the publicly-available CIFAR-10 dataset (Krizhevsky et al.).
sections.heading: ETHICS STATEMENT
sections.text: Since our work is foundational rather than applied in nature, we do not anticipate any direct ethical concerns. However, as with any foundational research in machine learning, our work has the potential to improve the performance and controllability of machine learning models, particularly in the domain of image classification. These improvements in turn may find use in ethical (e.g. medical diagnostics, autonomous vehicles) or ethically questionable (e.g. surveillance) applications.
  As for direct ethical impacts of our work, we acknowledge the energy costs of training and evaluating the models used in our experiments. One of our key contributions is proposing a method (via linear interpolation) to measure the frequency content of a learned function without retraining, which should enable future experimental research in spectral bias to be less energy-intensive.
sections.heading: A APPENDIX
sections.heading: A.1 LINEAR INTERPOLATION: METHODOLOGICAL DETAILS
sections.text: For each sampled path, we compute the discrete Fourier transform (DFT) of the prediction function separately for each of the M class predictions, take the (real) magnitude of the resulting (complex) DFT coefficients, and average them among the M classes. We then average these coefficient magnitudes among the many paths (all within-class paths and, separately, all between-class paths) and compute the fraction of this averaged DFT magnitude that is allocated to the high frequency components. We use a simple threshold that considers the lowest 10% of frequency components (i.e. frequencies between 0 and 0.05) “low” and the remaining (between 0.05 and 0.5) “high”, but qualitatively our results are not sensitive to this particular threshold. The highest frequency we can measure is 0.5, the Nyquist rate corresponding to our sampling distance of 1; if a particular maximum frequency is desired for measurement all that must be changed to achieve it is the sampling distance along the interpolating paths.
sections.heading: A.2 LABEL SMOOTHING
sections.text: A.3 SENSITIVITY TO NATURAL IMAGE DIRECTIONS
sections.heading: A.4 AGREEMENT BETWEEN LABEL SMOOTHING AND LINEAR INTERPOLATION
sections.text: A.4.1 VARYING THE FREQUENCY OF THE RADIAL WAVE LABEL SMOOTHING
  When the frequency of the radial wave used for label smoothing increases, models take more time to fit the smoothing noise. Figure 3 (Left) and Figure 8 show this using label smoothing; the corresponding interpolation experiment is shown in Figure 10. The target frequencies between 0.035 and 0.04 are close enough that, were the model to fit each perfectly, the interpolation curves would be nearly visually indistinguishable, as we can tell from Figure 2 (Left). However, the model is actually smoother when trying to fit higher frequency label smoothing noise, because it fits this noise less well (in addition to fitting it later in training). We can see this effect, for instance, by noting that wrn 32, wrn 160, and shake shake 32 fail to fit frequency 0.04, both in Figure 8 and Figure 10.
  It is also worth noting that, in Figure 10 and repeatedly across our interpolation experiments, the learned function is smoother (lower-frequency) within-class and less smooth (higher-frequency) between-class. This is to be expected since the model must change predictions somewhere along the path between examples from different classes.
sections.heading: A.4.2 LEARNING LOW FREQUENCIES FIRST
sections.text: The label smoothing experiment presented in Figure 3 (Left) and Figure 8 shows that low frequency target functions are learned earlier than higher frequency targets; we can also confirm this finding
  using interpolation with model checkpoints saved at different epochs throughout training (without any label smoothing). We see that indeed in the early stages of training low frequencies predominate, with higher frequencies entering as training proceeds. Surprisingly, we also see that towards the end of training low frequencies again predominate; the reason for this is an interesting question for further research.
sections.heading: A.4.3 WEIGHT DECAY
sections.text: Figure 5 shows on shake shake 32 via both label noise and linear interpolation that increasing the weight decay results in a smoother (lower frequency) learned function. Figure 12 shows the same result across all six models we tested.
  (a) shake shake 32 (b) shake shake 96 (c) shake shake 112
  (d) wrn 32 (e) wrn 160 (f) pyramid net
  (A) Radial wave label smoothing
  (b) shake shake 32 (c) shake shake 96 (d) shake shake 112
  (e) wrn 32 (f) wrn 160 (g) pyramid net
  (B) Min noise fitting summaries of label smoothing. Note that for pyramid net, min noise fitting does not fully capture the results of noise fitting; the “dips” are clearly separated across epochs, with higher weight decay inducing delayed dips, even if they achieve similar minimum noise fitting values.
sections.heading: A.5 TRAINING SET SIZE
sections.text: Figure 6 (Left) shows the regularization (frequency reduction) effect of increasing dataset size for wrn 160. Figure 13 shows the same effect on all six models we tested.
sections.heading: A.6 DATA AUGMENTATION
sections.text: Figure 6 (Center) uses linear interpolation to study the effect of common data augmentation procedures on the learned frequencies for wrn 160. Figure 14 shows the same experiment on all six models we tested.
sections.heading: A.6.1 MIXUP STRENGTH
sections.text: Figure 6 (Right) uses linear interpolation to show that training with Mixup data augmentation (Zhang et al., 2018) causes wrn 160 to learn a lower-frequency function, but too much Mixup can produce a higher-frequency function. Figure 15 repeats the experiment on all six models we tested.
sections.heading: A.7 SELF-DISTILLATION
sections.text: Figure 7 uses linear interpolation to show that self-distillation with shake shake 96 produces a student model that is lower-frequency than its teacher, even though both are trained to the same training loss and the student has higher validation accuracy than the teacher. Figure 16 shows the same result across all six models we tested.