## Interpreting Global Perturbation Robustness of Image Models using Axiomatic Spectral Importance Decomposition

Róisín Luo (Jiaolin Luo)

j.luo2@universityofgalway.ie

James McDermott

james.mcdermott@universityofgalway.ie

Colm O'Riordan

colm.oriordan@universityofgalway.ie

SFI Centre for Research Training in Artificial Intelligence School of Computer Science, University of Galway Galway, H91 TK33, Ireland

Reviewed on OpenReview: https: // openreview. net/ forum? id= uQYomAuo7M

## Abstract

Perturbation robustness evaluates the vulnerabilities of models, arising from a variety of perturbations, such as data corruptions and adversarial attacks. Understanding the mechanisms of perturbation robustness is critical for global interpretability. We present a model-agnostic, global mechanistic interpretability method to interpret the perturbation robustness of image models. This research is motivated by two key aspects. First, previous global interpretability works, in tandem with robustness benchmarks, e.g. mean corruption error (mCE), are not designed to directly interpret the mechanisms of perturbation robustness within image models. Second, we notice that the spectral signal-to-noise ratios (SNR) of perturbed natural images exponentially decay over the frequency. This power-law-like decay implies that: Low-frequency signals are generally more robust than high-frequency signals - yet high classification accuracy can not be achieved by low-frequency signals alone. By applying Shapley value theory, our method axiomatically quantifies the predictive powers of robust features and non-robust features within an information theory framework. Our method, dubbed as I-ASIDE ( I mage A xiomatic S pectral I mportance D ecomposition E xplanation), provides a unique insight into model robustness mechanisms. We conduct extensive experiments over a variety of vision models pre-trained on ImageNet, including both convolutional neural networks ( e.g. AlexNet , VGG GoogLeNet/Inception-v1 , , Inception-v3 , ResNet , SqueezeNet , RegNet , MnasNet , MobileNet , EfficientNet , etc.) and vision transformers ( e.g. ViT , Swin Transformer , and MaxViT ), to show that I-ASIDE can not only measure the perturbation robustness but also provide interpretations of its mechanisms.

## 1 Introduction

Image modeling with deep neural networks has achieved great success (Li et al., 2021; Khan et al., 2022; Han et al., 2022). Yet, deep neural networks are known to be vulnerable to perturbations. For example, the perturbations may arise from corruptions and adversarial attacks (Goodfellow et al., 2014; Hendrycks &amp; Dietterich, 2019; Szegedy et al., 2013), etc. Perturbation robustness, henceforth referred to as robustness, characterizes a crucial intrinsic property of models (Hendrycks &amp; Dietterich, 2019; Bai et al., 2021; Goodfellow et al., 2014; Silva &amp; Najafirad, 2020).

Robustness mechanisms refer to the mechanisms which lead to robustness in models. The study of robustness mechanisms aims to answer the question ' why some models are more robust than others ' (Lipton, 2018; Zhang et al., 2021; Bereska &amp; Gavves, 2024). The causes within this question can arise from multifarious

The 2D spectrum is mapped into 1D radial spectrum and further divided into M bands which are denoted from I 0 to I M ´ 1 .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-d5cce9e1-89b7-42dd-a3eb-a241a8165522.jpg)

Figure 1: Power-law-like energy spectral density (ESD) distribution of natural images over the frequency. The signal spectrum is divided into M bands (from I 0 to I M ´ 1 ). Each spectral band is a robustness band.

aspects, including architecture designs (Zhou et al., 2022), data annotations, training methods (Pang et al., 2020), inferences, etc. For example, noisy labels can arise from data annotations (Frénay &amp; Verleysen, 2013; Wei et al., 2021); adversarial attacks often take place in inferences. This research provides a unified view regarding the mechanisms of image model robustness on spectra. We only focus on global interpretation of robustness mechanisms of image models. Our method is primarily motivated by two key aspects, as detailed below.

Despite the substantial advances in previous global interpretability works (Covert et al., 2020; Kolek et al., 2022), in tandem with the wide adoption of empirical robustness benchmarks (Hendrycks &amp; Dietterich, 2019; Zheng et al., 2016; Zhang et al., 2021), these methods are not designed to provide global interpretations regarding model robustness. For example, SAGE (Covert et al., 2020), a global interpretability work, can attribute the decisive pixel features in decision-makings. Yet, the importance of decisive pixel features fails to interpret the global robustness. Although the robustness of image models can be quantified by mean corruption errors (mCE) (Hendrycks &amp; Dietterich, 2019) or the distances in feature spaces between clean and perturbed image pairs (Zheng et al., 2016; Zhang et al., 2021), these scalar metrics often fall short in interpreting the underlying 'why' question. This gap prompts us to provide global mechanistic interpretations of perturbation robustness.

Motivation also arises from the robustness characterization of the spectral signals within natural images. Images can be represented as spectral signals (Körner, 2022; Sherman &amp; Butler, 2007). The signal-to-noise ratios (SNR) (Sherman &amp; Butler, 2007) of spectral signals can be used to characterize the signal robustness with respect to perturbations. Our later empirical study, as illustrated in Figure 2, suggests that the spectral SNRs of perturbed natural images decay over the frequency with a power-law-like distribution. We refer to the spectral SNR as the ratio of the point-wise energy spectral density (ESD) (Stoica et al., 2005) of spectral signal to noise ( i.e. perturbation):

$$S N R ( r ) \coloneqq \frac { E S D _ { r } ( x ) } { E S D _ { r } ( \Delta x ) } \quad \quad \quad$$

where r is a radial frequency point, x is an image, ∆ x is the perturbation, ESD r p¨q gives the point-wise energy density at r . The ESD r p¨q is defined as:

$$E S D _ { r } ( x ) \coloneqq \frac { 1 } { | L _ { r } | } \cdot \sum _ { ( u, v ) \in L _ { r$$

where L r denotes a circle as illustrated in Figure 1, | L r | denotes the circumference of L r , and F p x q denotes the 2D Discrete Fourier Transform (DFT) of x . Readers can further refer to more details in Appendix B.1.

Why do the spectral SNRs of some corruptions and adversarial attacks exhibit a power-law-like decay over the frequency? We surmise that the ESDs of many perturbations are often not power-law-like, while the ESDs of natural images are power-law-like empirically, as shown in Figure 1. For example, the spatial perturbation drawn from N p 0 , σ 2 q ( i.e. white noise) has a constant ESD: ESD r p ∆ x q ' σ 2 . In Figure 2, we characterize the spectral SNR distributions of perturbed images. We set the energy of perturbations to 10% of the energy of the clean image for a fair comparison. The perturbation sources include corruptions (Hendrycks &amp; Dietterich, 2019) and adversarial attacks (Szegedy et al., 2013; Tsipras et al., 2018). We notice that the SNR distributions are also power-law-like over the frequency. We refer to spectral signals as spectral features or simply as features if without ambiguity. This power-law-like SNR decay suggests an empirical feature robustness prior: Low-frequency features are robust features (RFs) while high-frequency features are non-robust features (NRFs) . The experiments in Figure 3 demonstrate that models trained with low-frequency signals exhibit higher robustness compared to the models trained with high-frequency signals. This also echoes with our earlier claim 'low-frequency signals are generally more robust than high-frequency signals - yet high classification accuracy can not be achieved by low-frequency signals alone'.

Contributions . By applying the Shapley value theory framework (Shapley, 1997; Roth, 1988; Hart, 1989), we are able to assess the predictive powers (Zheng &amp; Agresti, 2000) of RFs and NRFs. Further leveraging a specifically designed characteristic function of the Shapley value theory framework, the predictive powers are assessed on the basis of their information gains. I-ASIDE uses information theory within the Shapley value theory framework, for interpreting robustness mechanisms, as detailed in Section 3. We claim our major contributions as:

- · We propose a model-agnostic, global interpretability method, for interpreting robustness mechanisms of image models, through the lens of the predictive powers of robust features and non-robust features;
- • We analyze the robustness mechanisms of image models within information theory on spectra;
- · We showcase a case study that I-ASIDE has the potential to interpret how supervision noise levels affect model robustness.

## 2 Notations

Þ

Dataset and annotation . We use a tuple x X Y , y to denote an image classification dataset, where X is the image set and Y is the label set. We use | Y | to denote the number of classes ( i.e. the cardinality of set Y ). The annotation task of image classification datasets is to assign each image with a discrete class probability distribution. We use P y x p | q to denote the ground-truth probability that an image x is assigned as a class y . We use P x p q to denote the probability of x in set X . We use P y p q to denote the probability of y in set Y . In class-balanced datasets, P y p q ' 1 | Y | .

Image classifier . The primary task of an image classifier is to predict the probability distributions over discrete classes for given images. We use Q y x θ p | ; q : p x, y q Ñ r 0 1 , s to denote a classifier in the form of conditional probability. The Q predicts the probability that an image x is of class y . The θ are the parameters. For brevity, we ignore the parameter θ . For example, we denote Q y x θ p | ; q as Q y x p | q .

## 3 Method

High-level overview . We apply Shapley value theory for axiomatically assigning credits to spectral bands. Within this framework, the specially devised characteristic function measures the information gains of spectral bands. I-ASIDE interprets robustness mechanisms using this axiomatic framework with the information theory.

Problem formulation . Quantifying the predictive powers of features can be viewed as a value decomposition problem. In this research, the value is the information quantities that the features contribute to decisions.

Figure 2: Spectral SNR characterization with multiple corruptions and adversarial attacks. The corruptions include: white noise, Poisson noise, Salt-and-pepper noise, and Gaussian blur. The adversarial attacks include: FGSM (Goodfellow et al., 2014), PGD (Madry et al., 2017), SparseFool (Modas et al., 2019) and Pixel (Pomponi et al., 2022). We set the perturbation energy to 10% of the energy of the reference image. The results are shown in decibels (dB) for better visualization. The dB below zero indicates that the perturbations overwhelm the spectral features.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-724f01aa-b21f-4144-ad3c-95c29342b4bb.jpg)

Specifically, the value is in the form of the log-likelihood expectation of predictions ( i.e. the negative crossentropy loss). The value decomposition aims to assign each robustness band a predictive power such that: (1) The sums of the predictive powers are equal to the value , and (2) the assigned predictive powers should reflect their importance in the decisions. In the coalitional game theory, this decomposition scheme is known as an axiomatic fairness division problem (Roth, 1988; Hart, 1989; Winter, 2002; Klamler, 2010; Han &amp; Poor, 2009). The fairness division must satisfy four axioms: efficiency , symmetry , linearity and dummy player (Roth, 1988). We refer to the axiomatic fairness division as axiomatic decomposition . Of the scheme, the axioms guarantee uniqueness and fairness (Aumann &amp; Maschler, 1985; Yaari &amp; Bar-Hillel, 1984; Aumann &amp; Dombb, 2015; Hart, 1989; Roth, 1988). The property fairness refers to the principle ' equal treatment of equals ' (Yokote et al., 2019; Navarro, 2019). Shapley value theory is the unique solution satisfying the above axioms. However, the theory merely provides an abstract framework. To employ, we have to instantiate two abstracts: (1) players and coalitions, and (2) characteristic function.

Abstract (1): players and coalitions . The spectral bands are dubbed as spectral players . A subset of the spectral player set is dubbed as a spectral coalition . The details are as shown in Section 3.1. The M spectral players can forge 2 M spectral coalitions. We represent the presences and absences of the spectral players as the pass-bands and stop-bands in a multi-band-pass digital signal filter (Oppenheim, 1978; Roberts &amp; Mullis, 1987; Pei &amp; Tseng, 1998), as shown in Section 3.2.

Abstract (2): characteristic function . The characteristic function is designed to measure the contributions that the coalitions contribute in decisions. The contributions of the 2 M coalitions are then combined to compute their marginal contributions in decisions. We specially design the characteristic function, as shown in Section 3.3 and Appendix B.3, to measure the information gains. Figure 4 shows the framework of applying the Shapley value theory. Figure 5 shows the block diagram of the spectral coalition filtering. Figure 6 shows an example of 2 M spectral coalitions.

|                                       | training frequency range   | training frequency range   | training frequency range   |
|---------------------------------------|----------------------------|----------------------------|----------------------------|
|                                       | full-freq r 0 , 1 s        | low-freq r 0 , 0 . 3 s     | high-freq r 0 . 3 , 1 s    |
| train acc.                            | 98 . 05%                   | 97.13 %                    | 96 . 18%                   |
| val. acc. w/ clean generalization gap | 96 . 10%                   | 98.80 %                    | 56 . 70%                   |
|                                       | ´ 1 . 95                   | ` 1 . 67                   | ´ 39 . 48 Ó                |
| val. acc. w/ Gaussian                 | Ó 92 . 60%                 | Ò 98.00 %                  | 26 . 70%                   |
| generalization gap                    | 5 . 45                     | 0 . 87                     | 69 . 48                    |

Figure 3: Understanding the role of spectral signals. We train a resnet18 on three datasets derived from STL10 (Coates et al., 2011): r 0 1 , s contains full-frequency signals; r 0 0 3 , . s only contains low-frequency signals with a cut-off frequency by 0 3; and . r 0 3 . , 1 s only contains the high-frequency signal with a cut-off frequency by 0 3. . In the derived datasets, the filtered high or low frequency signals are randomly replaced by the high or low frequency signals sampled from the full-frequency train set. We visualize the embeddings in the left figure with respect to clean samples and perturbed samples with Gaussian noise ( σ ' 0 1). . The samples are from the validation set. The accuracy on both the train and validation sets is provided in the right table. We also provide generalization gaps, measured by the differences between validation accuracy and train accuracy ( i.e. acc val ´ acc train ). We have noted in that: (1) both high-frequency and low-frequency signals contain sufficient discriminative information to achieve high training accuracy; (2) high-frequency signals alone are not robust signals because they fail to generalize well from train to validation; (3) overall, low-frequency signals are more robust signals because models trained alone with them generalize better and exhibit higher robustness. We summarize these empirical observations into Assumption 3.8.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-b432067b-6b84-4d26-b3de-4c91352d8193.jpg)

We organize the implementation details of instantiating the aforementioned two abstracts from three aspects: (1) formulating a spectral coalitional game, (2) the implementation of spectral coalitions and (3) the design of the characteristic function.

## 3.1 Spectral coalitional game

Spectral player . We use I i (where i P r M s : ' t 0 1 , , ¨ ¨ ¨ , M ´ u 1 ) to denote the i -th spectral player. The I 0 contains the most robust features and the I M ´ 1 contains the most non-robust features. The M spectral players constitute a player set I : ' t I i u M ´ 1 i ' 0 . Figure 15 in Appendix C.1 shows two partition schemes to partition spectral bands ( /lscript 8 and /lscript 2 ). We empirically choose /lscript 8 .

Þ

Spectral coalition . A subset r I Ď I is referred to as the spectral coalition . The player set I is often referred to as the grand coalition .

r Shapley value . A spectral coalitional game p I , v q is defined on a spectral player set I equipped with a characteristic function v . The weighted marginal contribution of a spectral player I i over all possible coalitions is referred to as the Shapley value of the spectral player I i . We use ψ i p I , v q to represent the Shapley value of

Characteristic function . A characteristic function v p r I q : r I Ñ R measures the contribution for a given coalition and satisfies v pHq ' 0. In this research, the contribution of r I is measured in the form of the log-likelihood expectation of the predictions by the Q , in which the input images only contain the signals present in the r I . We show that this design of v theoretically measures how much information the Q uses from the features in the I for decisions.

Þ

P

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-389909e4-8dcc-4976-9696-199038885248.jpg)

X

Figure 4: Framework of applying Shapley value theory. Spectral coalition filtering creates spectral coalitions over X . Each coalition contains a unique combination of spectral signals, in which some spectral bands are present and others are absent. The coalitions are fed into a classifier Q . For each coalition, Q outputs the predictions. The characteristic function v then uses the predictions to estimate the contributions of the features present in the coalitions. The results from v are combined to compute the marginal contributions of spectral bands i.e. the spectral importance distribution of Q .

the player I i . The Shapley value ψ i p I , v q is uniquely given by:

$$\psi _ { i } ( \mathcal { I }, v ) = \sum _ { \tilde { I } \subseteq \mathcal { I } \overline { \mathcal {$$

where 1 M ` M ´ 1 | ˜ I | ˘ ´ 1 gives the weight of the player I i presenting in the coalition ˜ . I

Spectral importance distribution (SID) . We use Ψ p v q : ' p ψ i q i Pr M s P R M to denote the collection of ψ i p I , v q over all players. We min-max normalize Ψ p v q by taking Ψ ¯ p v q ' Ψ p v q´ minΨ p v q || Ψ p v q´ minΨ p v q|| 1 . The reason for normalizing the SIDs is that we want to scalarize the SIDs for numerical comparisons. The ¯ Ψ p v q is referred to as spectral importance distribution . Figure 7 shows examples of the spectral importance distributions of trained and un-trained models.

Þ

Spectral robustness score (SRS) . We can also summarize spectral importance distributions into scalar values. We refer to the summarized scalar values as spectral robustness scores . We use S v p q : v Ñr 0 1 , s to denote the summarizing function.

## 3.2 Spectral coalition filtering

We represent the presences and absences of the spectral players through the signal pass-bands and stop-bands using a multi-band-pass digital signal filtering (Oppenheim, 1978; Pei &amp; Tseng, 1998; Steiglitz, 2020), as shown in Figure 5. For example, the example spectral coalition t I 0 , I 2 u signifies the signals present only in I 0 and I 2 . With the spectral coalition filtering, we are able to evaluate the contributions of the combinations of various spectral features. Figure 6 shows an example of 2 M spectral coalitions.

Þ

$$\text{mm-wise} \ e n n e u a s. \\ \mathbb { T } ( \widetilde { \mathcal { I } } ) ( m, n ) = \begin{$$

To implement the presences and absences of spectral signals, we define a mask map ❚ p r I q : r I Ñt 0 1 , u M N ˆ on 2D spectrum, where r I is a spectral coalition and M ˆ N denotes 2D image dimensions. The mask map is point-wisely defined as:

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-fc26cf75-01ec-49c6-9e5b-ca8ab2a16131.jpg)

p

q

Figure 5: Spectral coalition filtering. In this example, the mask map ❚ p r I q ( i.e. transfer function) only allows to pass the signals present in the spectral coalition t I 0 , I 2 u . The M is 4 and the absences are assigned to zeros. The images after spectral coalition filtering ( x ' r I ) are then fed into frozen classifiers to assess the contributions of spectral coalitions. The binary operation notation ' denotes coalition filtering given by Definition 3.1. The notation d denotes Hadamard product.

where p m,n q P r M s ˆ r N s . In the digital signal processing literature, this mask map is known as a transfer function (Steiglitz, 2020). In the 2D mask map, the frequency points are '1' in pass-bands (presences) and '0' in stop-bands (absences). The process of spectral coalition filtering with the mask map for a given spectral coalition I is shown as in Figure 5.

r We apply the transfer function ❚ p r I q on the spectra of images with element-wise product ( i.e. Hadamard product (Horn, 1990; Horadam, 2012)). Let F be the Discrete Fourier transform (DFT) operator and F ´ 1 be the inverse DFT (IDFT) operator (Tan &amp; Jiang, 2018). Readers can further refer to Appendix B.1.

Definition 3.1 (Spectral coalition filtering) . We define a binary operator ' ' ' to represent the signal filtering by:

$$x \rtimes \widetilde { \mathcal { I } } \coloneqq \mathcal { F } ^ { - 1 } \left [ \underbrace { \mathcal { F } ( x$$

Remark 3.2 (Absence baseline) . Formally, in the context of attribution analysis, the term 'baseline' defines the absence assignments of players (Sundararajan et al., 2017; Shrikumar et al., 2017; Binder et al., 2016). For example, if we use 'zeros' to represent the absence of players, the 'zeros' are dubbed as the 'baseline' in the attribution analysis. We have discussed multiple baselines in Appendix B.2.

where ' d ' denotes Hadamard product ( i.e. element-wise product), ✶ P R M N ˆ denotes an all-ones matrix and b P C M N ˆ represents the assignments of the absences of spectral players. In the context of attribution analysis, b is often referred as the baseline. In our implementation, we empirically set b ' 0 .

Definition 3.3 (Spectral coalition filtering over set) . Accordingly, we define the filtering over a set X as:

$$\mathcal { X } \rtimes \widetilde { \mathcal { I } } \coloneqq \{ x \rtimes \widetilde { \mathcal { I } } | x \in \mathcal { X } \}.$$

## 3.3 Characteristic function design

The characteristic function is needed in order to measure the contributions of the features in r I . We define the characteristic function as the gains of the negative cross-entropy loss values between feature presences and absences.

Figure 6: An example of a complete 2 M spectral coalitions. This example shows 16 spectral coalitions with M ' 4. Each coalition provides various information relevant to decisions. Each image is a coalition. Each coalition contains a unique spectral signal combination. We use binary code to index these coalitions. The '1' in the i -th position indicates the presence of the i -th player. For example, 1001 indicates the presences of two spectral players ( I 0 and I 3 ) in the coalition.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-228b1d1b-734b-4b05-92a9-adce511aa54d.jpg)

Definition 3.4 (Characteristic function) . The characteristic function v p I q : I Ñ R is defined as:

Þ

where the constant term C : ' E x ∼ X ř y P Y P y x p | q ¨ log Q y x p | ' Hq is used to fulfil v pHq ' 0 . We refer to the C as the Dummy player constant.

$$\text{fint} \ 3. 4 \ ( \text{Characteristic function} ). \ The characteristic function v ( \widetilde { \mathcal { I } } ) \, \cdot \, \widetilde { \mathcal { I } } \mapsto \mathbb { R } \, \text{is defined as} \colon \\ v ( \widetilde { \mathcal { I } } ) \colon = \underset { x \sim \mathcal { X } } { \mathbb { E } } \sum _ { y \in \mathcal { Y } } \left \{ \underbrace { P ( y | x ) \cdot \log Q ( y | x \rtimes \widetilde { \mathcal { I } } ) } { \text{Feature presence} } - \underbrace { P ( y | x ) \cdot \log Q ( y | x \rtimes \widetilde { \mathcal { I } } ) } { \text{Absence baseline} } \right \} \\ = \underset { x \sim \mathcal { X } } { \mathbb { E } } \sum _ { y \in \mathcal { Y } } P ( y | x ) \cdot \log Q ( y | x \rtimes \widetilde { \mathcal { I } } ) - C \\ \ast e \text{ the constant term} \ C \, \colon = \underset { x \sim \mathcal { X } } { \mathbb { E } } \sum _ { y \in \mathcal { Y } } P ( y | x ) \cdot \log Q ( y | x \rtimes \widetilde { \mathcal { I } } ) \, \text{is used to fulfill } v ( \mathcal { \mathcal { I } } ) = 0. \ W e r \, \text{fer to the } C \\. \ \sim \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim}\ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim} \ \text{$\sim}$$

Remark 3.5. If the labels are one-hot, then Equation 7 is simplified into:

and C : ' E x,y ∼ x X Y , y log Q y x p | ' Hq .

$$v ( \widetilde { \mathcal { I } } ) \coloneqq \underset { x, y \sim \langle \mathcal { X }, \mathcal { Y } \rangle } { \mathbb { E } } \log Q ( y | x \rtimes \widetilde { \mathcal { I } } ) - C \\ Q ( y | x \rtimes \emptyset ).$$

Linking to information theory . The relationship between information theory and the characteristic function in the form of negative log-likelihood of Bayes classifiers has been discussed in the literature (Covert et al., 2020; Aas et al., 2021; Lundberg &amp; Lee, 2017). Following on from their discussions, we show that the v in Equation 7 profoundly links to information theory in terms of spectral signals. The maximal information of features relevant to labels in r I is the mutual information I p X ' r I Y , q . A classifier Q can merely utilize a proportion of the information. Theorem 3.6 states an information quantity identity regarding the I p X ' r I Y , q and the v . The term D KL r P || Q s measures the point-wise ( i.e. for an image x ) KL-divergence between the predictions and ground-truth labels. On the basis of the information quantity identity, the v can be interpreted as the information gains between the presences and absences of the features, which the Q utilizes in decisions from the r I . By enumerating all coalitions, the information gains are then combined to compute the marginal information gains of features in decisions via Equation 3.

Theorem 3.6 (Spectral coalition information identity) . The information quantity relationship is given as:

$$\mathbb { I } ( \mathcal { X } \rtimes \widetilde { \mathcal { I } }, \mathcal { Y } ) \equiv \mathbb { E } _ { x \in \mathcal { X } \rtimes \widetilde { \mathcal { I } } } D _ { K L } [ P ( y | x ) | | Q ( y | | x ) ] + H ( \mathcal { Y } ) + v ( \widetilde { \mathcal { I } } ) + C \quad \quad ( 9 ) \\ \intertext { v is defined in Equation } \gamma \text{ and } H ( \mathcal { Y } ) \text{ is the Shannon entropy of the label set } \mathcal { Y }. \text{ The proof is provided}$$

where v is defined in Equation 7 and H p Y q is the Shannon entropy of the label set Y . The proof is provided in Appendix B.3.

Figure 7: Spectral importance distributions (SIDs) of trained models and un-trained models. The experimental models are pre-trained on ImageNet . We also include the models with random weights as a control marked by the blue box. We have noticed that: (1) The spectral importance distributions of trained models exhibit non-uniformity, and (2) the spectral importance distributions of un-trained models exhibit uniformity. We summarize these empirical observations as Assumption 3.7.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-68302bda-6802-4db0-848d-1d2ddc357f15.jpg)

## 3.4 Spectral robustness score (SRS)

Although we are firstly interested in using the spectral importance distributions (SID) for robustness interpretations, they can also be summarized into scalar scores for purposes such as numerical comparisons, and later correlation studies.

Assumption 3.7 (Spectral uniformity assumption of random decisions) . The second row in Figure 7 shows the SIDs from various models with randomized weights. We randomize the model weights with Kaiming initialization (He et al., 2015). The measured SIDs exhibit spectral uniformity. This suggests: Un-trained models do not have spectral preferences . We refer to 'the models with randomized parameters' as random decisions. Therefore, we assume that the SIDs of random decisions are uniform: M .

✶ Assumption 3.8 (Robustness prior) . We assume: Higher utilization of robust features in decisions implies robust models . This is further substantiated by the experiments in Figure 3. To reflect this robustness prior, we empirically design a series β : ' p β , β 0 1 , ¨ ¨ ¨ , β M ´ 1 q T where β P p 0 1 , q as the summing weights of SIDs. Empirically, we choose β ' 0 75 . because this choice achieves the best correlation with model robustness.

Summarizing with weighted sum . Let Ψ p v q be the measured spectral importance distribution (SID). Set ¯ Ψ p v q ' Ψ p v q´ minΨ p v q || Ψ p v q´ minΨ p v q|| 1 with min-max normalization. The weighted sum of the Ψ p v q with the weights β is given by:

$$\left | \beta ^ { T } \bar { \Psi } ( v ) - \beta ^ { T } \frac { 1 } { M } \right | & ( 1 0 ) \\ \text{m decision baseline. } \text{ Let } S ( v ) \, \colon v \mapsto [ 0, 1 ] \, & \text{be the normalized result in}$$

Þ

$$\i e \, S ( v ) & \text{ is given by} \colon \\ S ( v ) & \colon = \frac { \left | \beta ^ { T } \bar { \Psi } ( v ) - \beta ^ { T } \frac { 1 } { M } \right | } { \sup _ { \bar { \Psi } } \left | \beta ^ { T } \bar { \Psi } ( v ) - \beta ^ { T } \frac { 1 } { M } \right | } = \left | \frac { \bar { \beta } ^ { T } \bar { \Psi } ( v ) - \eta } { 1 - \eta } \right | \\ \omega, \bar { \beta } & = \frac { \beta } { \| \beta \| _ { 2 } } \text{ and } \eta = \frac { 1 } { M } \| \frac { \| \beta \| _ { 1 } } { \beta \| _ { 2 } }. \ \text{ Readers can refer to Appendix C.2 for the simplification}$$

where β T ✶ M is served as a random decision baseline. Let S v p q : v Ñ r 0 1 , s be the normalized result in Equation 10. The S v p q is given by:

where β P p 0 1 , , q ¯ β ' β || β || 2 and η ' 1 M || β || 1 || β || 2 . Readers can refer to Appendix C.2 for the simplification deduction.

## 4 Experiments

We design experiments to show the dual functionality of I-ASIDE , which can not only measure robustness and but also interpret robustness. We organize the experiments in three categories:

Figure 8: The spectral robustness scores (SRS), measured with I-ASIDE, correlate to the mean corruption errors (mCE) in the literature (Hendrycks &amp; Dietterich, 2019).

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-094cf358-14f9-4720-bb96-af570a617507.jpg)

- • Section 4.1: Correlation to a variety of robustness metrics;
- • Section 4.2: Studying architectural robustness;
- • Section 4.3: A case study interpreting how supervision noise levels affect model robustness.

Section 4.1 shows that the scores obtained with I-ASIDE correlate with the model robustness scores measured with other methods. Section 4.2 and Section 4.3 show that I-ASIDE is able to interpret robustness by examining SIDs.

Reproducibility . We choose M ' 8 and 200 samples to conduct experiments. We choose 200 samples because the experiments in Appendix C.3. The experiment shows that: A small amount of examples are sufficiently representative for spectral signals .

## 4.1 Correlation to robustness metrics

Definition 4.1 ( Mean prediction error . ) In our experiments, we measure model perturbation robustness with mean prediction errors (mPE) besides the mean corruption errors (mCE). Let x be some clean image and x ˚ be the perturbed image. For a classifier Q , we define the mean prediction error (mPE) as:

$$\Delta \mathcal { P } \coloneqq \mathbb { E } _ { x, y \sim \langle \mathcal { X }, \mathcal { Y } \rangle } | Q ( y | x ) - Q ( y | x ^ { * } ) |.$$

We demonstrate that I-ASIDE is able to measure model robustness. The experiments are broken down into three aspects: (1) correlation to mCE scores, (2) correlation to adversarial robustness, and (3) correlation to corruption robustness.

Correlation to mCE scores . Figure 8 shows the correlation between spectral robustness scores (SRS) and the mean corruption errors (mCE). The mCE scores are taken from the literature (Hendrycks &amp; Dietterich, 2018). The mCE scores are measured on a corrupted ImageNet which is known as ImageNet-C in the literature (Hendrycks &amp; Dietterich, 2019). The ImageNet-C includes 75 common visual corruptions with five levels of severity in each corruption. This correlation suggests that the results measured with I-ASIDE correlate with the results measured with robustness metric mCE.

)

Mean prediction error (

10

10

10

10

10

10

1.0

1.7

2.3

3.0

3.6

4.3

regnet\_y\_400mfresnet50

vgg16

mobilenet\_v2

resnet18

googlenet efficientnet\_v2\_s

vit\_b\_16

0.28

0.26

0.30

0.32

Spectral robustness score (Low-robust

0.34

0.36

High-robust)

(a) FGSM (

eps

'

0 1)

.

0.26

regnet\_y\_400mf resnet50

vgg16

mobilenet\_v2

resnet18

googlenet efficientnet\_v2\_s

vit\_b\_16

0.28

0.30

0.32

Spectral robustness score (Low-robust

(b) FGSM (

eps

'

0 2)

.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-2a988040-39dc-47e9-96bb-dcc299ea38de.jpg)

0.26

regnet\_y\_400mf resnet50

vgg16

mobilenet\_v2

resnet18

googlenet efficientnet\_v2\_s

vit\_b\_16

0.28

0.30

0.32

Spectral robustness score (Low-robust

0.34

0.36

High-robust)

eps

(d) PGD (

'

0 2)

.

Figure 9: The spectral robustness scores (SRS), measured with I-ASIDE, correlate with the mean prediction errors (mPE) in adversarial attacks. The circle sizes in (b) are proportional to the SRS.

Correlation to adversarial robustness . Figure 9 shows the correlation between the correlation between spectral robustness scores (SRS) and the mean prediction errors (mPE) of the adversarial attacks with FGSM and PGD. We vary the eps from 0 1 to 0 2. . . The results show that our scores correlate with the mean prediction errors in various eps settings. This suggests that the results measured by our method correlate with adversarial robustness.

Correlation to corruption robustness . Figure 10 shows the correlation between the correlation between spectral robustness scores (SRS) and the mean prediction errors (mPE) of the corruptions with white noise and Gaussian blurring. We vary the σ of white noise from 0 1 to 0 2. . . We vary the Gaussian blurring kernel sizes from 3 ˆ 3 to 7 ˆ 7. The results show that our scores correlate with the mean prediction errors in all cases. This suggests that the results measured by our method can reflect the corruption robustness.

0.34

0.36

High-robust)

)

Mean prediction error (

)

Mean prediction error (

10

10

10

10

10

10

10

10

10

10

10

10

1.1

1.7

2.2

2.8

3.3

3.8

1.1

1.7

2.2

2.8

3.3

3.8

Figure 10: The spectral robustness scores (SRS), measured with I-ASIDE, correlate to the mean prediction errors (mPE) in corruptions. The circle sizes in (b) are proportional to the SRS.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-d4dfb234-fc75-4e4e-a3dc-2fc213665b90.jpg)

## 4.2 Studying architectural robustness

I-ASIDE is able to answer questions such as:

- • Does model parameter size play a role in robustness?
- • Are vision transformers more robust than convolutional neural networks (ConvNets)?

Does model parameter size play a role in robustness ? Figure 11 (a) shows parameter counts do not correlate with model robustness. Thus, the tendency of a model to use robust features is not determined by parameter counts alone . We would like to carry out further investigation in future work.

(a) Spectral robustness score (SRS) comparison

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-b23ab306-00be-45e1-b980-68beb6e4c8d6.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-4ec3f1cc-a3f5-44e4-8c8d-10a85f274e47.jpg)

- (b) Spectral importance distribution (SID) t-SNE projection

Figure 11: How do architectural elements affect robustness? The left figure is to answer: 'Does model parameter size play a role on robustness?'. The right figure, a t-SNE projection of SIDs, is to answer: 'Are vision transformers more robust than convolutional neural networks?'. For the two questions, the answers suggested by using I-ASIDE are yes. But the story is more complicated, we have provided a brief discussion regarding this case study within Section 4.2. We also have noted that efficientnet surprisingly exhibits comparable perturbation robustness as the architectures in Transformer family. All experimental models are pre-trained on ImageNet . The circle sizes in (b) are proportional to SRS.

Are vision transformers more robust than ConvNets ? Figure 11 (b) shows a t-SNE projection of the spectral importance distributions of a variety of models. The results show that vision transformers form a cluster ( swin\_b , maxvit\_t and vit\_b\_16 ) and outperform ConvNets in terms of robustness. This results correlate with the recent robustness research in the literature (Paul &amp; Chen, 2022; Zhou et al., 2022; Shao et al., 2021). The interpretation is that: Vision transformers tend to use more robust features than ConvNets .

Discussion . Vision transformers generally outperform ConvNets; nevertheless, state-of-the-art ConvNets, e.g. efficientnet (Tan &amp; Le, 2019), can achieve comparable robustness performance ( e.g. by error rates on benchmark datasets) (Li &amp; Xu, 2023). The literature (Devaguptapu et al., 2021) affirms that efficientnet is more robust than most ConvNets. But, why efficientnet is unique? The efficientnet introduces an innovative concept in that the network sizes can be controlled by scaling the width, depth, and resolution with a compound coefficient (Tan &amp; Le, 2019). The base architecture is then searched with neural architecture searching (NAS) (Ren et al., 2021) instead of hand-crafted design. The NAS optimization objective is to maximize the network accuracy subject to arbitrary image resolutions. The searching implicitly encourages that the network structure of efficientnet uses more robust features. This is because: The low-frequency signals in various resolutions are robust signals while high-frequency signals are not. The second column in Figure 7 shows the SID of efficientnet pre-trained on ImageNet . The SID shows that efficientnet\_v2\_s uses more robust features than alexnet and resnet18 .

Figure 12: How do models respond to label noise? Our results show that models trained with higher label noise levels tend to use spectral signals uniformly, i.e. without a preference for robust (low-frequency) features.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-f456a0c0-2b03-4dd5-9bd1-989b5d014f49.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-9bda06c3-5bdb-46f0-ae23-e51fd6469cc7.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-6131a51a-5210-4098-b6de-d733fbb3e831.jpg)

## 4.3 Interpreting how supervision noise levels affect model robustness

The previous robustness benchmarks with mean corruption errors (mCE) are not able to answer the longstanding question: ' How and why label noise levels affect robustness? '. We demonstrate that I-ASIDE is able to answer this question.

Learning with noisy labels . Supervision signals refer to the prior knowledge provided by labels (Sucholutsky et al., 2023; Zhang et al., 2020; Shorten &amp; Khoshgoftaar, 2019; Xiao et al., 2020). There is a substantial line of previous research on the question of 'how supervision noise affects robustness' (Gou et al., 2021; Frénay &amp; Verleysen, 2013; Lukasik et al., 2020; Rolnick et al., 2017). This question is not completely answered yet. For example, Flatow &amp; Penner add uniform label noise into CIFAR-10 and study its impact on model robustness (Flatow &amp; Penner, 2017). Their results show that classification test accuracy decreases as the training label noise level increases. However, empirical studies like this are not able to answer the underlying 'why' question.

Noisy-label dataset . We derive noisy-label datasets from a clean Caltech101 . We randomly assign a proportion of labels with a uniform distribution over label classes to create a noisy-label dataset. We refer to the randomly assigned proportion as supervision noise level. We vary the noise level from 0 2 to 1 0 to derive . . five training datasets.

Experiment . We train three models ( googlenet , resnet18 and mobilenet\_v2 ) over the clean and the five noisy-label datasets for 120 epochs respectively. We then measure their SIDs. The results are visualized in Figure 12 with heat maps. The results show that there is a pattern across the above three models in that: The SIDs are more uniform with higher supervision noise levels . The interpretation regarding the learning dynamics with the presence of label noise is that: Models tend to use more non-robust features in the presence of higher label noise within training set .

## 5 Related work

We further conduct a literature investigation from three research lines: (1) global interpretability, (2) model robustness, and (3) frequency-domain research. This literature study shows that I-ASIDE provides unique insights in these research lines.

Global interpretability . Global interpretability summarizes the decision behaviours of models from a holistic view. In contrast, local interpretability merely provides explanations on the basis of instances (Sundararajan et al., 2017; Smilkov et al., 2017; Linardatos et al., 2020; Selvaraju et al., 2017; Arrieta et al., 2020; Zhou et al., 2016; Ribeiro et al., 2016; Lundberg &amp; Lee, 2017; Lakkaraju et al., 2019; Guidotti et al., 2018; Bach et al., 2015; Montavon et al., 2019; Shrikumar et al., 2017). There are four major research lines in image models: (1) feature visualization, (2) network dissection, (3) concept-based method, and (4) feature importance.

Feature visualization seeks the ideal inputs for specific neurons or classes by maximizing activations (Olah et al., 2017; Nguyen et al., 2019; Zeiler et al., 2010; Simonyan et al., 2013; Nguyen et al., 2016a;b). This method provides intuitions regarding the question: 'What inputs maximize the activations of specific neurons or classes?' . Network dissection aims to connect the functions of network units ( e.g. channels or layers ) with specific concepts ( e.g. eyes or ears ) (Bau et al., 2017). Concept-based methods understand the decisions by answering the question 'how do models use a set of given concepts in decisions?' (Kim et al., 2018; Ghorbani et al., 2019; Koh et al., 2020; Chen et al., 2020). For example, TCAV explains model decisions by evaluating the importance of a given set of concepts ( e.g. the textures dotted , striped and zigzagged ) for a given class ( e.g. the class zebra ) (Kim et al., 2018).

Global input feature importance analysis, often by using Shapley value theory framework, attempts to answer the question: 'How do input features contribute to predictions?' (Altmann et al., 2010; Greenwell et al., 2018; Lundberg &amp; Lee, 2017; Ribeiro et al., 2016; Simonyan et al., 2013; Sundararajan et al., 2017; Covert et al., 2020). However, there are few works falling in the scope of global interpretability with feature importance analysis. A related work, SAGE, applying the Shapley value theory framework, globally assigns spatial input features with importance values for interpreting spatial feature contributions (Covert et al., 2020).

Although the aforementioned global interpretability methods provide insights into understanding decisions inside black-box models, they do not provide interpretations regarding robustness mechanisms. Our work fundamentally differs from them in that: We provide interpretations regarding robustness mechanisms. We attempt to answer the fundamental question: 'Why some models are more robust than others?'.

Model robustness . Model robustness refers to the prediction sensitivity of models to perturbations. The perturbations can perturb in spaces such as the input space and the parameter space (Hendrycks &amp; Dietterich, 2019; Drenkow et al., 2021). In this research, we focus on the perturbations within the input space. The perturbations can stem from sources such as adversarial attacks (Szegedy et al., 2013; Goodfellow et al., 2014), corruptions (Hendrycks &amp; Dietterich, 2019), outliers (Hendrycks et al., 2018a; Pang et al., 2021) and supervision signal noise (Hendrycks et al., 2018b).

Model robustness is often assessed using scalar metrics (Hendrycks &amp; Dietterich, 2019; Krizhevsky et al., 2012; Laugros et al., 2019; Taori et al., 2020). For example, robustness can be measured by the distances between clean and perturbed pairs in feature spaces (Zheng et al., 2016). Hendrycks &amp; Dietterich benchmark the corruption robustness with mean corruption errors (mCE) over a set of corrupted datasets like ImageNet-C (Hendrycks &amp; Dietterich, 2019), using AlexNet (Hendrycks &amp; Dietterich, 2019) as a normalization baseline.

Despite their widespread adoption in previous literature, these scalar metrics lack the ability to provide detailed insights into the robustness mechanisms. Our work not only serves as a robustness metric but also offers mechanistic interpretations, answering the 'why' question behind model robustness. This dual functionality distinguishes our approach, providing a deeper understanding of the mechanisms.

Frequency-domain research . Neural networks are non-linear parameterized signal processing filters. Investigating how neural networks respond to input signals in the frequency-domain can provide a unique insight into understanding its functions. Xu et al. delve into the learning dynamics of neural networks in the frequency-domain (Xu et al., 2019a;b). They present their findings as 'F-Principle'. Their work suggests that the learning behaviors of neural networks exhibit spectral non-uniformity: Neural networks fit low-frequency components first, then high-frequency components.

In a related study, Tsuzuku &amp; Sato show that convolutional neural networks have spectral non-uniformity with respect to Fourier bases (Tsuzuku &amp; Sato, 2019). Later, Wang et al. connect model generalization behaviors and image spectrum (Wang et al., 2020). They argue that: (1) The supervision signals provided by humans use more low-frequency signals in images and (2) models trained on it tend to use more low-frequency signals. Our showcase experiment in Figure 12 provides the interpretations regarding their empirical findings.

In the interpretability research line within the frequency-domain, Kolek et al. propose 'CartoonX' based on the rate-distortion explanation (RDE) framework (Macdonald et al., 2019; Heiß et al., 2020). The RDE framework identifies decision-critical features by partially obfuscating the features. They refer to 'the prediction errors between clean inputs and the partially obfuscated inputs' as distortions. CartoonX pinpoints the decision-critical features within wavelet domain to answer the query: 'What features are crucial for

decisions?' (Kolek et al., 2022). Our work differs from CartoonX in that: (1) Our method aims to interpret model robustness mechanisms while CartoonX does not, (2) our method is a global interpretability method while CartoonX is a local interpretability method, (3) our method analyzes within an information theory framework while CartoonX uses RDE framework, and (4) our method uses Fourier bases while CartoonX uses wavelet bases.

## 6 Limitations

I-ASIDE provides a unique insight into the perturbation robustness mechanisms. Yet, our method has two major limitations: (1) The spectral perspective can merely reflect one aspect of the holistic view of model robustness, and (2) the SID resolutions are low.

Limitation (1) . For example, carefully crafted malicious adversarial perturbations on low-frequency components can fool neural networks (Luo et al., 2022; Liu et al., 2023; Maiya et al., 2021). Luo et al. demonstrate that attacking low-frequency signals can fool neural networks, resulting in attacks which are imperceptible to humans. This further implies the complexity of this research topic.

Limitation (2) . The computation cost is imposed by O p 2 M q . Fortunately, we do not need high SID resolution to analyze the model robustness problem. For example, a choice with M ' 8 is sufficient to interpret robustness mechanisms (as we have shown) while the computational cost remains reasonable.

## 7 Conclusions

On the solid ground provided by information theory and coalitional game theory, we present an axiomatic method to interpret model robustness mechanisms, by leveraging the power-law-like decay of SNRs over the frequency. Our method addresses the limitation that scalar metrics fail to interpret robustness mechanisms. We carry out extensive experiments over a variety of architectures. The SIDs, when scalarized, can largely reproduce the results found with previous methods, but addresses their failures to answer the underlying 'why' questions. Our method goes beyond them with the dual functionality in that: I-ASIDE can not only measure the robustness but also interpret its mechanisms. Our work provides a unique insight into the robustness mechanisms of image classifiers.

## Acknowledgments

This publication has emanated from research conducted with the financial support of Science Foundation Ireland under grant number 18/CRT/6223. For the purpose of Open Access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. We also thank reviewers for their constructive comments which can significantly improve our research quality. We thank the support from the ICHEC (Irish Centre for High-End Computing). We also thank the help from Prof. Dr. Michael Madden from University of Galway, Ireland.

## References

Kjersti Aas, Martin Jullum, and Anders Løland. Explaining individual predictions when features are dependent: More accurate approximations to shapley values. Artificial Intelligence , 298:103502, 2021.

André Altmann, Laura Toloşi, Oliver Sander, and Thomas Lengauer. Permutation importance: a corrected feature importance measure. Bioinformatics , 26(10):1340-1347, 2010.

Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion , 58:82-115, 2020.

Robert J Aumann and Michael Maschler. Game theoretic analysis of a bankruptcy problem from the talmud. Journal of economic theory , 36(2):195-213, 1985.

| Yonatan Aumann and Yair Dombb. The efficiency of fair division with connected pieces. ACM Transactions on Economics and Computation (TEAC) , 3(4):1-16, 2015.                                                                                                                                                                                                                                                       |
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one , 10(7):e0130140, 2015.                                                                                                                                                             |
| Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. Recent advances in adversarial training for adversarial robustness. arXiv preprint arXiv:2102.01356 , 2021.                                                                                                                                                                                                                                                 |
| David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 6541-6549, 2017.                                                                                                                                                    |
| Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety-a review. arXiv preprint arXiv:2404.14082 , 2024.                                                                                                                                                                                                                                                                                 |
| Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In Artificial Neural Networks and Machine Learning-ICANN 2016: 25th International Conference on Artificial Neural Networks, Barcelona, Spain, September 6-9, 2016, Proceedings, Part II 25 , pp. 63-71. Springer, 2016. |
| Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence , 2(12):772-782, 2020.                                                                                                                                                                                                                                                                   |
| Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.                                                                                                                                    |
| Ian Covert, Scott M Lundberg, and Su-In Lee. Understanding global feature contributions with additive importance measures. Advances in Neural Information Processing Systems , 33:17212-17223, 2020.                                                                                                                                                                                                                |
| Chaitanya Devaguptapu, Devansh Agarwal, Gaurav Mittal, Pulkit Gopalani, and Vineeth N Balasubramanian. On adversarial robustness: A neural architecture search perspective. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 152-161, 2021.                                                                                                                                         |
| Nathan Drenkow, Numair Sani, Ilya Shpitser, and Mathias Unberath. A systematic review of robustness in deep learning for computer vision: Mind the gap? arXiv preprint arXiv:2112.00639 , 2021.                                                                                                                                                                                                                     |
| David Flatow and Daniel Penner. On the robustness of convnets to training on noisy labels. Technical report, Stanford University , 2017.                                                                                                                                                                                                                                                                            |
| Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems , 25(5):845-869, 2013.                                                                                                                                                                                                                                       |
| Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in Neural Information Processing Systems , 32, 2019.                                                                                                                                                                                                                                              |
| Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014.                                                                                                                                                                                                                                                                    |
| Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision , 129(6):1789-1819, 2021.                                                                                                                                                                                                                                                 |
| Brandon M Greenwell, Bradley C Boehmke, and Andrew J McCarthy. A simple and effective model-based variable importance measure. arXiv preprint arXiv:1805.04755 , 2018.                                                                                                                                                                                                                                              |
| Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco Turini, and Fosca Giannotti. Local rule-based explanations of black box decision systems. arXiv preprint arXiv:1805.10820 , 2018.                                                                                                                                                                                                      |

| Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence , 45(1):87-110, 2022.                                                                                                                                  |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Zhu Han and H Vincent Poor. Coalition games with cooperative transmission: a cure for the curse of boundary nodes in selfish packet-forwarding wireless networks. IEEE transactions on communications , 57 (1):203-213, 2009.                                                        |
| Sergiu Hart. Shapley value. In Game theory , pp. 210-216. Springer, 1989.                                                                                                                                                                                                            |
| Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pp. 1026-1034, 2015.                                   |
| Cosmas Heiß, Ron Levie, Cinjon Resnick, Gitta Kutyniok, and Joan Bruna. In-distribution interpretability for challenging modalities. arXiv preprint arXiv:2007.00758 , 2020.                                                                                                         |
| Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 , 2019.                                                                                                                         |
| Dan Hendrycks and Thomas G Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. arXiv preprint arXiv:1807.01697 , 2018.                                                                                                                  |
| Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606 , 2018a.                                                                                                                                         |
| Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. Advances in neural information processing systems , 31, 2018b.                                                                        |
| Kathy J Horadam. Hadamard matrices and their applications . Princeton university press, 2012.                                                                                                                                                                                        |
| Roger A Horn. The hadamard product. In Proc. Symp. Appl. Math , volume 40, pp. 87-169, 1990.                                                                                                                                                                                         |
| Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR) , 54(10s):1-41, 2022.                                                                                            |
| Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning , pp. 2668-2677. PMLR, 2018. |
| Christian Klamler. Fair division. Handbook of group decision and negotiation , pp. 183-202, 2010.                                                                                                                                                                                    |
| Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning , pp. 5338-5348. PMLR, 2020.                                                                        |
| Stefan Kolek, Duc Anh Nguyen, Ron Levie, Joan Bruna, and Gitta Kutyniok. Cartoon explanations of image classifiers. In European Conference on Computer Vision , pp. 443-458. Springer, 2022.                                                                                         |
| Thomas William Körner. Fourier analysis . Cambridge university press, 2022.                                                                                                                                                                                                          |
| Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25, 2012.                                                                                               |
| Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. Faithful and customizable explanations of black box models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society , pp. 131-138, 2019.                                                          |
| Alfred Laugros, Alice Caplier, and Matthieu Ospici. Are adversarial robustness and common perturbation robustness independant attributes? In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops , pp. 0-0, 2019.                                      |

| Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 7558-7568, 2023.                                                                                                                                                               |
|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Zewen Li, Fan Liu, Wenjie Yang, Shouheng Peng, and Jun Zhou. A survey of convolutional neural networks: analysis, applications, and prospects. IEEE transactions on neural networks and learning systems , 33(12): 6999-7019, 2021.                                    |
| Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. Explainable AI: A review of machine learning interpretability methods. Entropy , 23(1):18, 2020.                                                                                               |
| Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue , 16(3):31-57, 2018.                                                                                                |
| Jiyuan Liu, Bingyi Lu, Mingkang Xiong, Tao Zhang, and Huilin Xiong. Low frequency sparse adversarial attack. Computers & Security , 132:103379, 2023.                                                                                                                  |
| Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning , pp. 6448-6458. PMLR, 2020.                                                                          |
| Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems , 30, 2017.                                                                                                                    |
| Cheng Luo, Qinliang Lin, Weicheng Xie, Bizhu Wu, Jinheng Xie, and Linlin Shen. Frequency-driven imperceptible adversarial attack on semantic similarity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 15315-15324, 2022. |
| Jan Macdonald, Stephan Wäldchen, Sascha Hauch, and Gitta Kutyniok. A rate-distortion framework for explaining neural network decisions. arXiv preprint arXiv:1905.11092 , 2019.                                                                                        |
| Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017.                                                                       |
| Shishira R Maiya, Max Ehrlich, Vatsal Agarwal, Ser-Nam Lim, Tom Goldstein, and Abhinav Shrivastava. A frequency perspective of adversarial robustness. arXiv preprint arXiv:2111.00861 , 2021.                                                                         |
| Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Sparsefool: a few pixels make a big difference. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 9087-9096, 2019.                                       |
| Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojciech Samek, and Klaus-Robert Müller. Layer-wise relevance propagation: an overview. Explainable AI: interpreting, explaining and visualizing deep learning , pp. 193-209, 2019.                         |
| Florian Navarro. Necessary players, myerson fairness and the equal treatment of equals. Annals of Operations Research , 280:111-119, 2019.                                                                                                                             |
| Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems , 29, 2016a.                              |
| Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. arXiv preprint arXiv:1602.03616 , 2016b.                                                    |
| Anh Nguyen, Jason Yosinski, and Jeff Clune. Understanding neural networks via feature visualization: A survey. Explainable AI: interpreting, explaining and visualizing deep learning , pp. 55-76, 2019.                                                               |
| Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill , 2(11):e7, 2017.                                                                                                                                                               |
| Alan V Oppenheim. Applications of digital signal processing. Englewood Cliffs , 1978.                                                                                                                                                                                  |

| Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM computing surveys (CSUR) , 54(2):1-38, 2021.                                                                                                                       |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Bag of tricks for adversarial training. arXiv preprint arXiv:2010.00467 , 2020.                                                                                                                                                |
| Sayak Paul and Pin-Yu Chen. Vision transformers are robust learners. In Proceedings of the AAAI conference on Artificial Intelligence , volume 36, pp. 2071-2081, 2022.                                                                                                                    |
| Soo-Chang Pei and Chien-Cheng Tseng. A comb filter design using fractional-sample delay. IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing , 45(5):649-653, 1998.                                                                                         |
| Jary Pomponi, Simone Scardapane, and Aurelio Uncini. Pixle: a fast and effective black-box attack based on rearranging pixels. In 2022 International Joint Conference on Neural Networks (IJCNN) , pp. 1-7. IEEE, 2022.                                                                    |
| Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR) , 54(4):1-34, 2021.                                                             |
| Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pp. 1135-1144, 2016.                              |
| Richard A Roberts and Clifford T Mullis. Digital signal processing . Addison-Wesley Longman Publishing Co., Inc., 1987.                                                                                                                                                                    |
| David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning is robust to massive label noise. arXiv preprint arXiv:1705.10694 , 2017.                                                                                                                                       |
| Alvin E Roth. The Shapley value: essays in honor of Lloyd S. Shapley . Cambridge University Press, 1988.                                                                                                                                                                                   |
| Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pp. 618-626, 2017. |
| Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Yu Chen, and Cho-Jui Hsieh. On the adversarial robustness of vision transformers. arXiv preprint arXiv:2103.15670 , 2021.                                                                                                                        |
| Lloyd S Shapley. A value for n-person games. Classics in game theory , 69, 1997.                                                                                                                                                                                                           |
| Charles H Sherman and John L Butler. Transducers and arrays for underwater sound , volume 4. Springer, 2007.                                                                                                                                                                               |
| Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data , 6(1):1-48, 2019.                                                                                                                                                     |
| Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning , pp. 3145-3153. PMLR, 2017.                                                                               |
| Samuel Henrique Silva and Peyman Najafirad. Opportunities and challenges in deep learning adversarial robustness: A survey. arXiv preprint arXiv:2007.00753 , 2020.                                                                                                                        |
| Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 , 2013.                                                                                                |
| Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 , 2017.                                                                                                                       |
| Kenneth Steiglitz. Digital Signal Processing Primer . Courier Dover Publications, 2020.                                                                                                                                                                                                    |

| Spectral analysis of signals Saddle River, NJ, 2005.                                                                                                                                                                                                                                             |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Ilia Sucholutsky, Ruairidh M Battleday, Katherine M Collins, Raja Marjieh, Joshua Peterson, Pulkit Singh, Umang Bhatt, Nori Jacoby, Adrian Weller, and Thomas L Griffiths. On the informativeness of supervision signals. In Uncertainty in Artificial Intelligence , pp. 2036-2046. PMLR, 2023. |
| Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning , pp. 3319-3328. PMLR, 2017.                                                                                                                             |
| Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013.                                                                                                 |
| Lizhe Tan and Jean Jiang. Digital signal processing: fundamentals and applications . Academic Press, 2018.                                                                                                                                                                                       |
| Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pp. 6105-6114. PMLR, 2019.                                                                                                                 |
| Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems , 33:18583-18599, 2020.                                          |
| Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint arXiv:1805.12152 , 2018.                                                                                                                    |
| Yusuke Tsuzuku and Issei Sato. On the structural sensitivity of deep convolutional networks to the directions of fourier basis functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 51-60, 2019.                                                |
| Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 8684-8694, 2020.                                      |
| Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. arXiv preprint arXiv:2110.12088 , 2021.                                                                                            |
| Eyal Winter. The shapley value. Handbook of game theory with economic applications , 3:2025-2054, 2002.                                                                                                                                                                                          |
| Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. arXiv preprint arXiv:2008.05659 , 2020.                                                                                                                                    |
| Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv:1901.06523 , 2019a.                                                                                                           |
| Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. In International Conference on Neural Information Processing , pp. 264-274. Springer, 2019b.                                                                                       |
| Menahem E Yaari and Maya Bar-Hillel. On dividing justly. Social choice and welfare , 1:1-24, 1984.                                                                                                                                                                                               |
| Koji Yokote, Takumi Kongo, and Yukihiko Funaki. Relationally equal treatment of equals and affine combinations of values for tu games. Social Choice and Welfare , 53:197-212, 2019.                                                                                                             |
| Matthew D Zeiler, Dilip Krishnan, Graham WTaylor, and Rob Fergus. Deconvolutional networks. In 2010 IEEE Computer Society Conference on computer vision and pattern recognition , pp. 2528-2535. IEEE, 2010.                                                                                     |
| Yu Zhang, Peter Tiňo, Aleš Leonardis, and Ke Tang. A survey on neural network interpretability. IEEE Transactions on Emerging Topics in Computational Intelligence , 2021.                                                                                                                       |

Zhenyu Zhang, Xiaobo Shu, Bowen Yu, Tingwen Liu, Jiapeng Zhao, Quangang Li, and Li Guo. Distilling knowledge from well-informed soft labels for neural relation extraction. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pp. 9620-9627, 2020.

Beiyao Zheng and Alan Agresti. Summarizing the predictive power of a generalized linear model. Statistics in medicine , 19(13):1771-1781, 2000.

Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In Proceedings of the ieee conference on computer vision and pattern recognition , pp. 4480-4488, 2016.

Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 2921-2929, 2016.

Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. In International Conference on Machine Learning , pp. 27378-27394. PMLR, 2022.

## A Appendix

## B Fairness division axioms

Symmetry axiom : Let r I P 2 I be some spectral player coalition. For @ I i , I j P I ^ I i , I j R r I , the statement v p r I Yt I i uq ' v p r I Yt I j uq implies ψ i p I , v q ' ψ j p I , v q . This axiom restates the statement ' equal treatment of equals ' principle mathematically. This axiom states that the 'names' of players should have no effect on the 'treatments' by the characteristic function in coalition games (Roth, 1988).

Linearity axiom : Let u and v be two characteristic functions. Let p I , u q and p I , v q be two coalition games. Let p u ` v qp r I q : ' u p r I q ` v p r I q where r I P 2 I . The divisions of the new coalition game p I , u ` v q should satisfy: ψ i p I , u ` q ' v ψ i p I , u q ` ψ i p I , v q . This axiom is also known as ' additivity axiom' and guarantees the uniqueness of the solution of dividing payoffs among players (Roth, 1988).

Efficiency axiom : This axiom states that the sum of the divisions of all players must be summed to the worth of the player set (the grand coalition): M ´ 1 ψ i p I , v q ' v p I q .

Dummy player axiom : A dummy player (null player) I ˚ is the player who has no contribution such that: ψ ˚ p I , v q ' 0 and v p I Yt I ˚ uq ' v p I q for @ I ˚ R I ^ I ˚ Ď I .

$$\ u e s _ { M - 1 } \\ \sum _ { i = 0 } ^ { \ } \\ \ u l l _ { \L }$$

r r r Remark B.1. In the literature (Roth, 1988), the efficiency axiom and the dummy player axiom are also combined and relabeled as carrier axiom.

## B.1 Spectral signal-to-noise ratio (SNR)

Discrete Fourier Transform . The notion 'frequency' measures how 'fast' the outputs can change with respect to inputs. High frequency implies that small variations in inputs can cause large changes in outputs. In terms of images, the 'inputs' are the pixel spatial locations while the 'outputs' are the pixel values.

Þ

Let x : p i, j q ÞÑ R be some 2D image with dimension M ˆ N which sends every location p i, j q to some real pixel value where p i, j q P r M s ˆ r N s . Let F : R 2 Ñ C 2 be some DFT functional operator. The DFT of x is given by:

$$\mathcal { F } ( x ) ( u, v ) = \sum _ { j = 0 } ^ { N - 1 } \sum _ { i = 0 } ^ { M - 1 } x ( i, j ) e ^ { - i 2 \pi ( \frac { 1 } { M } i + \frac { 1 } { N } j ) }. \\ \int _ { \cdots \cdots \cdots } ^ { N } \int _ { \mathcal { C } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } \mathcal { D } }.$$

Point-wise energy spectral density (ESD) . The ESD measures the energy quantity at a frequency. To simplify discussions, we use radial frequency , which is defined as the radius r with respect to zero frequency point ( i.e. the frequency center). The energy is defined as the square of the frequency magnitude according to Parseval's Power Theorem.

Let L r be a circle with radius r on the spectrum of image x , as illustrated in Figure 1. The r is referred to as radial frequency. The point-wise ESD function is given by:

$$\cdot \quad & \quad E S D _ { r } ( x ) \colon = \frac { 1 } { | L _ { r } | } \cdot \sum _ { ( u, v ) \in L _ { r } } | \mathcal { F } ( x ) ( u, v ) | ^ { 2 } \\ \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \end{array}$$

where p u, v q is the spatial frequency point and | L r | is the circumference of L r .

Spectral signal-to-noise ratio (SNR) . The SNR can quantify signal robustness. We define the spectral SNR at radius frequency r as:

$$S N R ( r ) \coloneqq \frac { E S D _ { r } ( x$$

where ∆ x is some perturbation. We have characterized the SNRs of some corroptions and adversarial attacks in Figure 2.

## B.2 Absence assignment scheme

There exist multiple choices for the assignments of the absences of spectral layers in coalition filtering design: (1) Assigning to constant zeros (Zeroing), (2) assigning to complex Gaussian noise (Complex Gaussian) and (3) assigning to the corresponding frequency components randomly sampled from other images at the same dataset (Replacement).

Zeroing . The b in Equation 5 is set to zeros.

Complex Gaussian . The b in Equation 5 is sampled from a i.i.d. complex Gaussian distribution: N p µ, σ 2 2 q ` i N p µ, σ 2 2 q .

Replacement . The b in Equation 5 is set to: b ' F p x ˚ q (where x ˚ ∼ X is a randomly sampled image from some set X ).

In our implementation, we simply choose 'zeroing': b ' 0 . Figure 13 shows the filtered image examples by using the above three strategies and also show the examples of measured spectral importance distributions. Empirically, the three strategies have rather similar performance. In this research, we do not unfold the discussions regarding the masking strategy choices.

(a) Zeroing

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-c878b7ae-c0b1-4a83-bebe-593f0ca9e4e0.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-9ecf1bed-6c49-447d-8e13-0c416434fd62.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-98810b39-ef65-44eb-8636-9a0f6fe70508.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-979ee4b5-3de5-4dc0-9c3f-1a2bc10da1c9.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-8bf647d1-e70e-41f0-b47e-43c66b746294.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-9768f993-9a8a-414d-8ca7-5fc019fe5f74.jpg)

(b) Gaussian

Complex

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-970068c2-1d6f-4910-9b0d-2bdf3b18fcf5.jpg)

(g) resnet18 w/ Zeroing

(c) Replacement

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-ef444d5d-47cf-47d2-b208-7ab1c7dac0c8.jpg)

(d) Zeroing

(e) Gaussian

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-324cd064-e7a3-47b1-acec-95dfdbfa5ca7.jpg)

Complex

(f) Replacement

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-e7de02e0-30ef-4368-b01c-2ca8404cabe0.jpg)

(h) resnet18 w/ Complex Gaussian

(i) resnet18 w/ Replacement

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-b979c11e-1947-46df-9947-eb72b90a3d35.jpg)

(j) efficientnet\_v2\_s w/ Zeroing

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-9886dbb5-745c-46b7-a16f-49817fa5ad96.jpg)

(k) efficientnet\_v2\_s w/ Complex Gaussian

(l) efficientnet\_v2\_s w/ Replacement

Figure 13: Three absence assignment strategies: (1) Assigning the spectral absences with constant zeros (Zeroing), (2) assigning the spevtral absences with Gaussian noise (Complex Gaussian) and (3) randomly sampling spectral components from the same image datasets (Replacement). The standard complex Gaussian distribution is given by: N p 0 , 1 2 q ` i N p 0 , 1 2 q . The figures (a), (b) and (c) show the coalition filtering results with the spectral coalition: t I 0 u . The figures (d), (e) and (f) show the coalition filtering results with the spectral coalition: t I 1 , I 2 , I 3 , I 4 , I 5 , I 6 , I 7 u . The figures (g) to (l) show the examples of the measured spectral importance distributions of a resnet18 and a efficientnet\_v2\_s (both are pre-trained on ImageNet ) with the three assignment strategies.

## B.3 Proof for Spectral Coalition Information Identity Theorem

Proof for Spectral Coalition Information Identity. Suppose the probability measures P x p q , P x, y p q , P y x p | q , and Q y x p | q are absolutely continuous with respect to x on domain X ' r I .

$$\begin{array} {$$

$$\mathbb { I } ( \mathcal { X } \rtimes \tilde { \math$$

$$& \underset { x \ltimes \tilde { \mathcal { I } } } {$$

$$\prod _ { \chi \in \tilde { \mathcal { X } } } ^$$

$$& \underset { x \ltimes \tilde { \mathcal { I } } } { J } \underset { y \in \mathcal { Y } } { \overleftarrow { y \in \mathcal { Y } } } \theta \overset { \bullet \, \cdots } { \bullet \, \cdots } \theta \ \mathcal { M } ( y | x ) \ P ( y ) \ \mathcal { M } \overset { \bullet \, \cdots } { \bullet \, \cdots } \theta \ \mathcal { M } ^ { \prime } \\ = & \int _ { x \ltimes \tilde { \mathcal { I } } } P ( x$$

$$& \overleftarrow { y \in \mathcal { Y } } \bigwedge _ { \chi \rtimes \tilde { \mathcal { I } } } ^ { J } \stackrel { \cdot \cdot \cdot \cdot } { \cdot \cdot \cdot } \int _ { \real } ^ { \cdot \cdot \cdot \cdot } P ( x, y ) \cdot \log Q ( y | x ) d x \\ & + \int _ { \real } \sum _ { \chi \rtimes \tilde { \mathcal { I } } } P ( x, y ) \cdot \log Q ( y | x ) d x \\ & \quad \quad \quad$$

$$& \underset { x \in \mathcal { X } \lll \widetilde { \mathcal { I } } } { \underset { \overbrace { x \in \widetilde { \mathcal { X } } \lll \widetilde { \mathcal { I } } } } { \underset { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace { \overbrace {$$

$$& \cdot \bigotimes _ { \mathcal { X } \ll \widetilde { \mathcal { I } } } \bigoplus _ { y \in \mathcal { Y } } ^ { \bigoplus \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cdot \nu \cd$$

$$= \underset { x \in \mathcal { X } \times \widetilde { \mathcal { I } } } { E } \underset { \widetilde { \mathcal { I } } } { \underset { \widetilde { \mathcal { I } } } { \underbrace { K L } ( P ( y | x ) | | Q ( y | | x ) ) } } \underset { \widetilde { \mathcal { I } } } { \text{point-wise} } } + H ( \mathcal { Y } ) + v ( \widetilde { \mathcal { I } } ) + C \\ H ( \mathcal { Y } ) \text{ is the Shannon entropy of the label set } \$$

where H p Y q is the Shannon entropy of the label set Y .

## C Information quantity relationship in spectral coalitions

Figure 14: Information quantity relationship. This shows the theoretical information quantity relationship between what the characteristic function v measures and the mutual information ■ p X ' r I Y , q . For a given coalition r I , a dataset x X Y , y and a classifier Q , the v measures how much information the classifier Q utilizes in decisions. The measured results are then used to compute the marginal contributions of features.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-0b54d2cb-fd4d-43d6-99e8-8406614abdb0.jpg)

## C.1 Partitioning spectrum with /lscript 8 ball over /lscript 2 ball

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-224376ec-f150-4040-9d35-c54cf0bf75d6.jpg)

Spectral players with /lscript 8 partition.

The pixel density of I M ´ 1 with /lscript 2 ball.

The pixel density of I M ´ 1 with /lscript 8 ball.

Figure 15: Two spectral band partitioning schemes. This shows the motivation we choose /lscript 8 ball over /lscript 2 ball in partitioning the frequency domain into the M bands (i.e., M 'spectral players') over 2D Fourier spectrum. The frequency data density of the spectral players with /lscript 8 remains a constant. However, the frequency data density of the spectral players with /lscript 2 is not a constant since some frequency components do not present. This motives us to empirically choose /lscript 8 metric to form spectral players in implementation.

## C.2 Normalizing summarized SIDs

We normalize the above result and set:

$$S ( v ) \coloneqq \frac { \left | \beta ^ { T } \bar { \Psi } ( v ) - \frac { \| \beta \| _ { 1 } } { M } \right | } { \sup \left | \beta ^ { T } \bar { \Psi } ( v ) - \frac { \| \beta \| _ { 1 } } { M } \right | } \\ | \alpha T \bar { \tau } _ { \tilde { \tau } } / \dots \quad \| \beta \| _ { 1 } |$$

$$= \frac { \Phi ^ { T } \bar { \Psi } ( v ) - \frac { | | \beta | | _ { 1 } } { M } \Big | } { \left | | | \beta | | _ { 2 } - \frac { | | \beta | | _ { 1 } } { M } \Big | } \\ | \bar { \partial } T \bar { \partial } \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Psi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi\Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phis \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \ Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi \Phi$$

$$= \frac { \ell ^ { \prime } \lambda ^ { \prime } \lambda ^ { \prime } \lambda ^ { \prime } } { \left | \beta ^ { T } \bar { \Psi } ( v ) - \frac { \| \beta \| _ { 1 } } { M } \right | } \\ \sup \left | | | \beta | | _ { 2 } \cdot | | \bar { \Psi } ( v ) | | _ { 2 } - \frac { \| \beta \| _ { 1 } } { M } \right | } \\ | \alpha T \bar { \Psi } \lambda \lambda _ { \dots } \quad \| \beta \| _ { 1 } |$$

$$= \left | \frac { \bar { \beta } ^ { T } \bar { \Psi } ( v ) - \frac { 1 } { M } \frac { \| \beta \| _ { 1 } } { \| \beta \| _ { 2 } } } { 1 - \frac { 1 } { M } \frac { \| \beta \| _ { 1 } } { \| \beta \| _ { 2 } } } \right |.$$

where ¯ β ' β || β || 2 and sup ∣ ∣ ∣ β T ¯ Ψ p v q ´ || β || 1 M ∣ ∣ ∣ is derived by:

$$\sup \left | \beta ^ { T } \bar { \Psi } ( v ) - \frac { | | \beta | | _ { 1 } } { M } \right | = \left | \sup \beta ^ { T } \bar { \Psi } ( v ) - \frac { | | \beta | | _ { 1 } } { M } \right | \, \quad \, \\ \dots \dots \.$$

$$\ell = \ell \left | \sup | | \beta | | _ { 2 } \cdot | | \bar { \Psi } ( v ) | | _ { 2 } - \frac { \ell } { M } \right | \ s. t. \, | | \bar { \Psi } ( v ) | | _ { 1 } = 1 \quad \quad ( 3 0 )$$

$$= \stackrel { \cdot } { \left | } | | \beta | | _ { 2 } - \frac { | | \beta | | _ { 1 } } { M } \right | \quad \text{since} \ | | \bar { \Psi } ( v ) | | _ { 2 } ^ { 2 } \leqslant | | \bar { \Psi } ( v ) | | _ { 1 } ^ { 2 }.$$

$$S ( v ) & = \left | \frac { \bar { \beta } ^ { T } \bar { \Psi } ( v ) - \eta } { 1 - \eta } \right |. & ( 3 2 ) \\ & \quad \ \, \cap \, \varepsilon$$

Q.E.D.

Set η ' 1 M || β || 1 || β || 2 :

## C.3 How much samples are sufficient?

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-f0851ed3-eac8-486b-98e3-af735708ecbb.jpg)

(a)

resnet18 @CIFAR10

Figure 16: Convergence of relative estimation errors converge with respect to the numbers of samples K . The errors are measured by: 1 M || Ψ p ` q i 1 p v q ´ Ψ p q i p v q|| 1 where Ψ p q i p v q denotes the i -th measured spectral importance distribution with respect to characteristic function v . The experiments are conducted on CIFAR10 , CIFAR100 and ImageNet with resnet18 .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-88517347-7612-4d62-8729-2a9d464ab765.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01139v2-figure-d8903695-4220-4333-bf22-faefd6f64973.jpg)

Error bound analysis . Let K be the number of the samples of some baseline dataset. Let:

$$\Delta v ( \tilde { \mathcal { I } }, \mathcal { I } _ { i } ) \coloneqq v ( \tilde { \mathcal { I } } \cup \{ \mathcal { I } _ { i } \} ) - v ( \tilde { \mathcal { I } } )$$

and and

$$\Delta v ( \mathcal { I } _ { i } ) \coloneqq ( \Delta v ( \tilde { \mathcal { I } }, \mathcal { I } _ { i } ) ) _ { \tilde { \mathcal { I } } \subseteq \mathcal { I } }$$

$$W \coloneqq \left ( \frac { 1 } { M } \binom { M - 1 } { | \tilde { \mathcal { I } } | } ^ { - 1 } \right ) _ { \tilde { \mathcal { I } } \subset \mathcal { I } }.$$

Hence:

$$\psi _ { i } ( \mathcal { I }, v ) = W ^ { T } \Delta v ( \mathcal { I } _ { i } )$$

where || W || 1 ' 1 since W is a probability distribution. Let ¯ , ∆¯ ψ i v p I i q and ∆¯ ˜ v p I I , i q be estimations with K samples using Monte Carlo sampling. The error bound with /lscript 1 norm is given by:

$$\epsilon \stackrel { \text{def} } { = } \sup _ { i } | | \bar { \psi } _ { i } ( \mathcal { I }, v ) - \psi _ { i } ( \mathcal { I }, v ) | | _ { 1 } = \sup _ { i } | | W ^ { T } \Delta \bar { v } ( \mathcal { I } _ { i } ) - W ^ { T } \Delta v ( \mathcal { I } _ { i } ) | | _ { 1 } \\.$$

$$& \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \stackrel { \sim } { \stackrel { \stackrel { \sim } { \stackrel { \sim } { \stackrel { \stackrel { \sim } { \stackrel { \stackrel { \stackrel { \sim } { \stackrel { \stackrel { \sim } { \stackrel { \stackrel { \stackrel { \stackrel { \stack { \stackrel { \stack { \stackrel { \stack { \stackrel { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack { \stack \stack { \stack \stack { \stack \stack { \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack { \stack { \stack \stack { \stack { \stack \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack \stack \stack { \stack \stack { \stack \stack { \stack \stack { \stack \stack \stack \stack { \stack \stack \stack { \stack \stack \stack \stack { \stack \stack \stack { \stack \stack { \stack \stack \stack \stack { \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \ \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \text \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \Stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \text \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \text \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \stack \text \stack \stack \stack \text \stack \stack \stack \text \stack \stack \stack \text \stack \text \stack \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \text \endlisp} } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } ) } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } _ } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } { } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } \ } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } ; } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } }, } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } }$$

$$\lambda & \stackrel { \psi \psi } { i } r _ { i } \psi _ { i } \psi _ { i }, \psi _ { i } \psi _ { i }, \psi _ { i } \psi _ { i } \psi _ { i } \cdots \stackrel { \psi \psi } { i } } \cdots \stackrel { \psi \psi _ { i } } \cdots \stackrel { \psi \psi _ { i } } \cdots \stackrel { \psi \psi _ { i } } \cdots \stackrel { \psi \psi _ { i } } \cdots \\ & \leqslant \sup _ { i } \| W \| _ { 1 } \cdot \| \Delta \bar { v } ( \mathcal { I } _ { i } ) - \Delta v ( \mathcal { I } _ { i } ) \| _ { \infty } \quad \text{(Hölder$'s inequality)} \\ & = \sup _ { j } \| \sum \quad \left ( \Delta \bar { v } ( \tilde { \mathcal { I } }, \mathcal { I } _ { i } ) - \Delta v ( \tilde { \mathcal { I } }, \mathcal { I } _ { i } ) \right ) \| _ { \infty }$$

$$\begin{array} { c } \underset { i } { \sup } 2 ^ { M - 1 } \cdot \sup _ { \tilde { I } } \| \Delta \bar { v } ( \tilde { \mathcal { I } }, \mathcal { I } _ { i } ) - \Delta v ( \tilde { \mathcal { I } }, \mathcal { I } _ { i } ) \| _ { \infty } \\ \dots \cdot \end{array}$$

$$\begin{matrix} & & i & & \mathcal { I } \\ & & \leqslant 2 ^ { M - 1 } \cdot \left \{ \frac { V a r ( \Delta \bar { v } ) } { K } \right \} ^ { \frac { 1 } { 2 } } & & & \\ & & & & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \ \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \	 & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad &\quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad \\ \quad & \leqslant & 2 ^ { M - 1 } \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \ \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \dot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot
 \end{matrix}$$

$$= \sup _ { i } 2 ^ { M - 1 } \cdot \sup _ { \tilde { \mathcal { I } } } \| \Delta \tilde { v } ( \tilde { \mathcal { I } }, \mathcal { I } _ { i } ) - \Delta v ( \tilde { \mathcal { I } }, \mathcal { I } _ { i } ) \| _ { 1 } \\.$$

where V ar p ∆¯ v q gives the upper bound of the variance of ∆¯ v p ˜ I I , i q .
Róisín Luo James McDermott Colm O'Riordan
Interpreting Global Perturbation Robustness of Image Models using Axiomatic Spectral Importance Decomposition
Perturbation robustness evaluates the vulnerabilities of models, arising from
a variety of perturbations, such as data corruptions and adversarial attacks.
Understanding the mechanisms of perturbation robustness is critical for global
interpretability. We present a model-agnostic, global mechanistic
interpretability method to interpret the perturbation robustness of image
models. This research is motivated by two key aspects. First, previous global
interpretability works, in tandem with robustness benchmarks, e.g. mean
corruption error (mCE), are not designed to directly interpret the mechanisms
of perturbation robustness within image models. Second, we notice that the
spectral signal-to-noise ratios (SNR) of perturbed natural images exponentially
decay over the frequency. This power-law-like decay implies that: Low-frequency
signals are generally more robust than high-frequency signals -- yet high
classification accuracy can not be achieved by low-frequency signals alone. By
applying Shapley value theory, our method axiomatically quantifies the
predictive powers of robust features and non-robust features within an
information theory framework. Our method, dubbed as \textbf{I-ASIDE}
(\textbf{I}mage \textbf{A}xiomatic \textbf{S}pectral \textbf{I}mportance
\textbf{D}ecomposition \textbf{E}xplanation), provides a unique insight into
model robustness mechanisms. We conduct extensive experiments over a variety of
vision models pre-trained on ImageNet to show that \textbf{I-ASIDE} can not
only \textbf{measure} the perturbation robustness but also \textbf{provide
interpretations} of its mechanisms.