## Resilience and Security of Deep Neural Networks Against Intentional and Unintentional Perturbations: Survey and Research Challenges

SAZZAD SAYYED ‡∗ , MILIN ZHANG ∗‡ , SHAHRIAR RIFAT ∗‡ , ANANTHRAM SWAMI † , MICHAEL DE LUCIA † , and FRANCESCO RESTUCCIA ‡ ,

‡ Institute for the Wireless Internet of Things, Northeastern University, United States and

- † DEVCOM Army Research Laboratory, United States

In order to deploy deep neural networks (DNNs) in high-stakes scenarios, it is imperative that DNNs provide inference robust to external perturbations - both intentional and unintentional. Although the resilience of DNNs to intentional and unintentional perturbations has been widely investigated, a unified vision of these inherently intertwined problem domains is still missing. In this work, we fill this gap by providing a survey of the state of the art and highlighting the similarities of the proposed approaches. We also analyze the research challenges that need to be addressed to deploy resilient and secure DNNs. As there has not been any such survey connecting the resilience of DNNs to intentional and unintentional perturbations, we believe this work can help advance the frontier in both domains by enabling the exchange of ideas between the two communities.

## ACMReference Format:

Sazzad Sayyed, Milin Zhang, Shahriar Rifat, Ananthram Swami, Michael De Lucia, and Francesco Restuccia. 2024. Resilience and Security of Deep Neural Networks Against Intentional and Unintentional Perturbations: Survey and Research Challenges. ACM Comput. Surv. 1, 1 (August 2024), 33 pages. https://doi.org/XXXXXXX.XXXXXXX

## 1 INTRODUCTION

Thanks to their ability of performing critical tasks such as object detection [56], language translation [4], image classification [15], and efficient pose estimation [106], DNNs have become essential in our everyday life [14]. For a comprehensive survey on DNNs, we refer the reader to [130]. Breakthroughs in the form of large language models such as GPT-4 [124], universal segmentation models such as Segment Anything [74] and diffusion models such as Stable Diffusion [134] have advanced the frontier of artificial intelligence (AI) and captured the interest of ordinary citizens in using AI in their day-to-day lives.

The unprecedented benefits of DNNs come with their own set of social and ethical challenges, mainly in the form of privacy, safety and security issues [39]. For example, it has been shown that a DNN is extremely sensitive to intentional perturbations where changing a few pixels in the input can lead to misclassifications [11, 49, 115, 146]. In addition,

∗ These authors contributed equally to this research.

Authors' Contact Information: Sazzad Sayyed, sayyed.a@northeastern.edu; Milin Zhang, zhang.mil@northeastern.edu; Shahriar Rifat, rifat.s@ northeastern.edu; Ananthram Swami, ananthram.swami.civ@army.mil; Michael De Lucia, michael.j.delucia2.civ@army.mil; Francesco Restuccia, f.restuccia@northeastern.edu,

‡ Institute for the Wireless Internet of Things, Northeastern University, Boston, Massachussetts, United States and

† DEVCOM Army Research Laboratory, United States.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

© 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.

Manuscript submitted to ACM

DNNs are vulnerable to unintentional perturbations due to natural phenomena, for example, frost, rain, shot noise, impulse noise, defocus blur, motion blur and zoom blur, as well as natural shifts in distribution of labels, which is also known as semantic shift . To ensure AI systems based on DNNs can be deployed in real-world systems, it is imperative to ensure resilience and security from both intentional and unintentional perturbations. Achieving this goal can be ensured, for example, by guaranteeing that the output of a DNN will be inherently robust and accurate, or by implementing a rejection scheme that detects inputs on which the prediction is likely to be incorrect due to perturbation or semantic shift, so that the DNN can be adapted to provide a correct prediction for such inputs under specific constraints. This makes the study of resilience and security of DNNs a timely yet extremely complex issue.

## 1.1 Motivation and Novel Contributions

Although intentional and unintentional perturbations share several critical aspects, a unified vision has so far been elusive. Since its inception in 2014 [150], the study of resilient DNN design has been fragmented into separate domains. Some approaches have been studying intentional perturbations - also known as adversarial machine learning - while some approaches have been proposed to guarantee robustness against Out-of-Distribution (OOD) samples. Since a DNN needs to be resilient to both types of perturbation, it is important to study the connection between these domains. [136] discusses the relation between OOD detection and anomaly detection, open-set recognition, and novel set recognition. The survey in [2] focuses on the detection of adversarial inputs and bench-marking some of the detection approaches. To the best of our knowledge, literature lacks surveys focusing on the connection between the detection of adversarial input and OOD input. As such, in this paper we discuss the literature from the perspective of resilience of DNNs encompassing detection of both adversarial and OOD. Specifically, the main contributions of this paper can be summarized as follows:

- · We categorize and discuss the seminal, significant, and recent work in the domain of OOD detection (i.e., unintentional interference) and adversarial sample detection (i.e., intentional interference);
- · We investigate the commonalities among intentional and unintentional perturbation detection and the corresponding defense strategies, while remarking the strengths and weaknesses of these approaches. We believe these two communities can benefit from this study as they can discover common approaches, similarities in these two fields, and adopt new perspective from the other community.
- · We conclude the paper by pointing out some open questions and research directions regarding ensuring the resilience of DNNs in real-world inference systems.

## 2 VULNERABILITIES OF DEEP NEURAL NETWORKS: BACKGROUND AND TAXONOMY

We define the term "vulnerability" as any action that causes the DNN to perform in not its intended manner as compared to when the action is absent. Under the scope of this survey, we consider actions where external perturbations are introduced to the input samples during inference. In this section, we provide background on different approaches to introduce these perturbations intentionally. Next, we illustrate different scenarios that occur when external perturbations are added to the input samples in a natural fashion.

## 2.1 Intentional Perturbation: Adversarial Attack

We consider a target DNN 𝑓 ( 𝑥 ) = 𝑦 , where 𝑥 ∈ X and 𝑦 ∈ Y represent input and output samples respectively, with X Y , respectively representing input and output space. The objective of an adversary is to find a perturbation 𝛿 that misleads the target DNN. Based on their intent or objectives, adversarial attacks can be categorized as untargeted and targeted . The former aim at causing an incorrect classification without specifying a particular target class, i.e.,

$$f ( x + \delta ) \neq y, \quad \delta \leq \epsilon.$$

Fig. 1. Taxonomy of intentional perturbations.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00193v2-figure-8a1b44d7-c5af-4a8e-9305-21461b1df49a.jpg)

Equation 1 describes the untargeted attack scenario where 𝜖 is a constraint determining the magnitude of the perturbation. On the other hand, the targeted attack aims at inducing a specific incorrect classification toward a specific class ˆ 𝑦 ≠ 𝑦 ,

$$f ( x + \delta ) = \hat { y }, \quad \delta \leq \epsilon.$$

Based on the attacker's knowledge of the target DNN, attacks can be classified as white-box [6, 18, 49, 79, 105], black-box [3, 71, 98, 126, 127], and gray-box [81, 158]. In the white-box setting, the attacker has full knowledge of the targeted DNN architecture, parameters, and training data, which enables highly-effective perturbations. A common strategy in white-box attacks involves formulating the attack as an optimization problem, facilitating the utilization of gradient descent to identify adversarial examples. Consequently, these techniques are often referred to as gradient-based attacks. For example, Fast Gradient Sign Method (FGSM) [49] uses a single step in the direction of gradient of the loss function L(·) with respect to the input to obtain the necessary perturbation to evade correct detection of the the input sample as shown in equation 3. Basic Iterative Method (BIM) [79] improved the approach by searching iteratively. Its variant Projected Gradient Descent (PGD) [105] with random initialization and multiple restarts is considered as a baseline approach to assess adversarial robustness in literature.

$$x _ { a d v } = x + \epsilon \cdot s g n ( \nabla _ { x } \mathcal { L } ( \theta, x, y ) ),$$

Carlini-Wagner (CW) [18] demonstrated the importance of the loss function and optimizer. Furthermore, they proposed a surrogate loss given by Equation 4,

$$\mathcal { L } ( z, y ) = \max _ { z _ { i } \neq y } z _ { i } - z _ { y }.$$

which denotes the margin loss where z is the output vector, 𝑧 𝑖 is the score of 𝑖 -th class of the target DNN and 𝑦 is the ground-truth label of the sample. Multi-Target (MT) [50] enhances the gradient attack by a surrogate loss function designed for multiple targeted classes 𝜏 ∈ 𝑇 ,

$$\mathcal { L } ( z, y ) = \sum _ { \tau \in T } z _ { \tau } - z _ { y }, \quad y \notin T.$$

Equation 5 denotes the multi-targeted loss, which aims at optimizing adversarial examples across multiple targeted classes and thus bypass the local minima associated with a single class. As a result, it demonstrates strong capability to evade defense algorithms. Among other notable work Expectation Over Transformation (EOT) [6] employs input augmentation, Guided Adversarial Margin Attack (GAMA) [144] utilizes guidance from function mapping of unperturbed

samples, Auto Attack (AA) [27] uses ensemble of attacks, Auto Conjugate Gradient (ACG) [172] applies conjugate directions to guide the gradient descent, and MIFPE [179] minimizes effect of floating point error on the gradient step.

On the other hand, black-box attacks have only limited knowledge about the victim model. As a result, black-box attacks leverage indirect information such as model transferability and output score to generate effective adversarial examples. Black-box attacks encompass a diverse array of strategies, including transfer-based [34, 94, 168], and query-based [3, 21, 71] attacks, among others. In the transfer attack scenario, attackers perform gradient attacks on a group of surrogate models by assuming that adversarial samples misleading one DNN model are also likely to mislead others. Therefore, transfer attacks consists of two parts: surrogate model training and transferable adversarial sample generation. Model Ensemble (ME) [98] first proposed to use averaged prediction from ensemble of models to improve transferability. Common Weakness Attack (CWA) [20] targets the common weaknesses of multiple surrogate models based on cosine similarity and smoothness. Other notable approaches are self-ensemble (Ghost Network Ensemble (GNE) [91], Transfer-based Self-Ensemble Attack (T-SEA) [67]), input transformations ([168], [94], Admix [162], Spectrum Sanity Attack (SSA) [100], Structure Invariant Transformation (SIT) [163], Path-Augmented Method (PAM) [186]), gradient calibration (Momentum Iterative Method (MIM) [33], Nesterov Iterative Method (NIM) [94], Linear Back-Propagation (LinBP) [54], Penalizing Gradient Norm (PGN) [46], Translation Invariant Method (TIM) [34], Trasferable Attack based on Intergreted Gradient (TAIG) [70], Variance Tuning (VT) [161], TAIG [70],Gradient Relevant Attack (GRA) [196]), and surrogate model training (Knowledge Extraction Generative Network (KEGN) [176], Data-Free Model Extraction (DFME) [183], Data-free Substitute Training (DaST) [195]).

In query-based attacks, attackers can only access the input and corresponding DNN output ( i.e. , probability scores or hard labels). Adversarial samples are generated based on an iterative search by sending multiple quires to the target model. The approaches of query based attacks follow one of two streams score based and decision based scenario. [21] first proposes zeroth order optimization approach and applies gradient estimation to find perturbations without substitute model training. Natural Evolution Strategy (NES) [71], N-attack [92], AdvFlow [113], and NP-attack [9] try to approximate the gradient information. As gradient estimation demands a large number of queries, SimBA [53], Square Attack [3], PPBA [87], and BABIES[156] utilize random search to find the optimal perturbation for attacking the model.

Gray-box adversarial attacks, also known as partial knowledge attacks, are a class of adversarial machine learning techniques where the attacker has limited knowledge about the target model. Unlike black-box attacks, where the attacker has no knowledge of the model's architecture or parameters, and white-box attacks, where the attacker has full access to the model, gray-box attacks assume the attacker has some information-such as knowledge of the model's architecture but not its exact parameters, or access to some but not all training data. [127] demonstrated that adversarial examples crafted for a substitute model (one that approximates the target model) can effectively transfer to the target model, indicating the potency of gray-box attacks . Another study by [155] explored ensemble methods to enhance the robustness of models against gray-box attacks by using multiple substitute models to generate adversarial examples, showing that such attacks can still circumvent defenses designed for black-box or white-box scenarios .

Based on the imperceptibility of perturbations, adversarial attacks can be classified as 𝐿 𝑝 norm constrained [49, 79, 105], sparse [26, 32, 112], and unrestricted [12, 139, 166] attacks. In most of the literature, the threat model is bounded in 𝑙 2 or 𝑙 ∞ norm. In other word, the constraint in Eqn. 1 and Eqn. 2 are refined as | | 𝛿 | | 𝑝 ≤ 𝜖 , where 𝑝 = 2 or 𝑝 = ∞ . Sparse attacks investigate a more difficult problem where attackers can only perturb a small number of pixel of each input. Unrestricted attacks leverage small digital transformations of inputs ( i.e. , rotation, scaling, brightness) or generate noise that contains semantic difference in physical world ( i.e. , shadows, raindrops, laser beams) to attack victim models.

Based on the specificity of perturbations, adversarial attacks can be classified as instance-aware [49, 79, 105] and universal attack [89, 114, 116, 182]. Instance-aware attack aims to generate adversarial examples tailored to each specific instance. This approach considers the unique characteristics of each input when crafting perturbations to maximize their effectiveness. Universal adversarial perturbation, on the other hand, is a perturbation that can be applied universally to a wide range of samples, regardless of their specific characteristics. Unlike instance-aware perturbations, universal perturbations are designed to have a broad impact across different instances, potentially affecting various types of inputs. They are crafted to exploit vulnerabilities in machine learning models consistently across diverse datasets or input distributions.

## 2.2 Unintentional Perturbation: Determining Out-of-Distribution Samples

Figure 2 shows a taxonomy of existing work in unintentional perturbation. To study the resilience of DNN to unintentional perturbation, existing work makes assumption regarding the nature of the perturbation. If the perturbation has no resemblance to the distribution of the training data, it is extremely challenging to adapt the DNN without additional knowledge and/or labeled samples. In this case, perturbation detection can be considered as a viable option.

Fig. 2. Taxonomy of unintentional perturbation.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00193v2-figure-adbf1bdd-a36b-41a4-a982-f03a7288bc47.jpg)

We introduce some notation and define X and Y as the random variables respectively representing input and labels. D ( 𝑆 𝑋 ) and D ( 𝑆 𝑌 ) is used to denote the marginal distribution of input samples and the labels in the domain from where the training dataset is sampled. The joint distribution of input and label D ( 𝑆 𝑋,𝑌 ) is consequently known as training distribution , source distribution or in-domain distribution . Similarly, D ( 𝑇 𝑋 ) and D ( 𝑇 𝑌 ) are used to denote the marginal distribution of input and label in the target domain while their joint distribution D ( 𝑇 𝑋,𝑌 ) is termed as the target distribution , or out-of-distribution . We define the input space X ∈ R 𝑑 and the corresponding label space Y , where 𝑑 indicates the dimension of the input. The dimension of Y varies depending on the task, e.g., for classification tasks, Y = [ 𝐶 ] ≜ { 1 , ..., 𝐶 } , where C denotes the number of classes, and Y ∈ R 𝑑 for semantic segmentation tasks. We define a DNN 𝑓 𝜃 𝑆 : X ↦→ Y composed of a feature extractor 𝑔 𝜃 𝑆 : X ↦→ Z and a predictor ℎ 𝜃 𝑆 : Z ↦→ Y such that 𝑓 = ℎ ◦ 𝑔 , where Z is the intermediate representation space and Z is the corresponding random variable.

Out of distribution (OOD) samples can be defined as any input drawn from any distribution different from training distribution. The term distribution shift encompasses four types of shifts in the literature as summarized in Figure 2. The first is non-semantic or covariate shift (e.g. noise, blur, frost and different types of corruptions, change in geographical location, brightness etc.) and the second is semantic shift or label set shift , which corresponds to the emergence of new semantic category or class label. The third is prior shift , and corresponds to a shift in the distribution of the classes observed. For example, an object detection algorithm will be more likely to encounter 'animal' objects in a rural area than in an urban area. The fourth is the generalized label shift , where the prior shift and the covariate shift happens simultaneously. We formally define the four types of distribution shifts below.

Definition of Covariate Shift : The covariate shift is a change in the distribution D( 𝑋 ) which preserves their label distribution D( 𝑌 ) . In other words, D ( 𝑆 𝑌 𝑋 | ) = D ( 𝑇 𝑌 𝑋 | ) . Notice that the target feature distribution D ( 𝑇 𝑋 ) is assumed to be known. Since the covariate shift does not change the semantic content of the input, it is also named nonsemantic shift . Direct perturbation, either intentional or unintentional, may cause such shift. Among the unintentional perturbations, we have common corruptions such as Gaussian blur, Gaussian noise, motion blur, defocus blur, frost, fog and rain, among others [59]. This also includes shift in input distribution due to a change in geographical location, operating hardware (e.g., different camera or sensors) or viewpoint [58]. Due to the change in the distribution D(X) , the distribution D(X Y) , changes. This brings the problem of dataset shift [128, 131]. Additionally, we can consider intentionally perturbed inputs a.k.a. adversarial inputs [18, 105, 115] as covariate shifts, since the perturbation only changes the input distribution. This highlights the connection between unintentional and intentional perturbations. Definition of Semantic Shift: Any kind of shift that changes the semantic content and as a result the marginal distribution of label D(Y) of the dataset is a semantic shift or label shift. A semantic shift affects both the image space and the label space as the distribution of input D 𝑋 is shifted from the source distribution and new labels are introduced in the label space. The detection of such shifts can be considered as encompassing the tasks of novel class recognition and open set recognition . In the open set recognition problem [157], the DNN is presented with sample from classes which were not present in training data representing a shift in the label distribution. One-class novel class recognition can be thought of as an extreme version of open set recognition where the DNN is presented with a single class during training and is required to detect new incoming classes during testing or inference.

Definition of Prior Shift: Prior shift refers to a scenario where the marginal distribution of labels in source and target domain are different, i.e., D ( 𝑆 𝑌 ) ≠ D ( 𝑇 𝑌 ) . On the other hand, the class conditional distribution of data given labels are assumed to be same, i.e., D ( 𝑆 𝑋 𝑌 | ) = D ( 𝑇 𝑋 𝑌 | ) . Prior shift might affect domain adaptation processes, also known as Test Time Adaptation (TTA) in literature [48, 180]. Although the marginal label distribution D ( 𝑆 𝑌 ) is uniform in most of the cases, in TTA the data is observed as a small batch of samples at a particular time. Hence, we might observe dominance of certain labels based on the current scenario. This is a very common example of prior shift that might happen which is also termed as ' correlated label distribution ' in TTA literature.

Definition on Generalized Label Shift. Generalized label shift is a more challenging data drift that was first introduced in [151]. It occurs when both covariate shift D ( 𝑆 𝑋 ) ≠ D ( 𝑇 𝑋 ) and prior shift D ( 𝑆 𝑌 ) ≠ D ( 𝑇 𝑌 ) occurs simultaneously. Moreover, it is assumed that there exists some feature representation 𝑍 = 𝑔 𝑋 ( ) for which the conditional distributions based on labels both on source and target domain are equal, i.e., D ( 𝑆 𝑍 𝑌 | ) = D ( 𝑇 𝑍 𝑌 | ) . This situation might arise when a DNN continuously experiences different data corruptions while also being adapted with a skewed label distribution.

## 3 A UNIFIED VISION FOR INTENTIONAL AND UNINTENTIONAL PERTURBATION

There are two main approaches to ensure resilience of DNNs. The first is by training the DNN to be inherently resilient. Examples include adversarial training [72, 73, 185] to avoid intentional perturbations, and training with different augmentation schemes [58, 62] for unintentional perturbations. The second way is to augment the DNN with another binary classifier that differentiates between perturbed and unperturbed samples. Ideally, the auxiliary classifier would be able to differentiate between unperturbed and perturbed inputs for both intentional and unintentional perturbation. In reality, the detection of intentional and unintentional perturbation have developed as two separate domains. In the following, we highlight the connection among the methods for detection of intentional and unintentional perturbations. As such, we have categorized the approaches from the perspective of input space, latent (i.e., model) space, and logit (i.e., output) space. Figure 3 overviews our categorization, which is detailed below. Manuscript submitted to ACM

Fig. 3. Overview of existing approaches for DNN resilience.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00193v2-figure-6b46b034-ce94-49d2-845c-6eec4c1e377f.jpg)

Input Space: Prior work has attempted to model the input distribution to detect intentional and unintentional perturbations. For example, [188] proposed to model adversarially perturbed inputs with a diffusion model, which is also known as generative modeling . The authors propose the expected perturbation score which is the expected value of the gradient of the distribution of the output of the diffusion process at a given time, and show that the distribution of the expected perturbation score is different for the unperturbed and intentionally perturbed samples. For unintentional perturbation detection, the authors in [138] propose a technique to learn the input distribution using generative models and corrects for the bias for complexity of the input samples.

Model (or Latent) Space: Most of the existing work either adapts the model space or create scores based on the latent representations. The works in this space can befurther categorized into four types: (i) augmentation, (ii) regularization, (iii) model adaptation, and (iv) scoring functions. Some prior work has used OOD samples available in the wild to train the DNN to distinguish between the In-Distribution (ID) and OOD samples by regularizing the loss function. The first work in this direction was [61], which was later improved in [160, 197] by improving the sampling of the OOD data as well as the regularization methodology. Prior work in OOD detection adapts the model to better differentiate between the ID and OOD inputs. For example, [31, 35, 148] shape the model features during test-time for better detection of the OOD inputs. The work in [102] discretizes the output of a layer of the model into binary or quaternary codes and detects intentional perturbation by observing deviations from the codes produced by unperturbed samples. In addition, the authors in [30] focus on intentional perturbation detection and design a partial Bayesian network by learning the likelihood of the weights for the last few layers. This network is then used to measure the uncertainty associated with the input. A vast body of work in OOD crafts different scoring functions based on the latent representation of the DNN. For example [1, 83, 123, 149] craft scores using intermediate layer activations, while [83] also uses the score for detection of intentionally perturbed inputs.

Output (or Logit) Space: By output space, we indicate either the logit layer of a DNN or its softmax probability distribution. The work in such output space has mostly focused on model adaptation and scoring functions. To the best of our knowledge, [164] is the only work we have found that focuses on the output space during training time. This work normalizes the output logits to mitigate the overconfidence issue of DNNs. Conversely, [135] examines

Manuscript submitted to ACM

logits to detect the intentionally perturbed inputs and potentially correct the prediction. The work in [60] first used the maximum softmax probability to detect OOD samples, while [57, 65, 93, 97] studied the detection of OOD using score-functions derived from the output of a DNN.

## 4 DETECTION OF INTENTIONAL PERTURBATION

Based on the classification in Section 3, we describe existing work in perturbation detection as well as the related defense strategies. The detection of intentional perturbation - also known as adversarial perturbations in literature - is mainly based on the hypothesis that adversarial samples reside in a manifold different than that of the unperturbed inputs [42, 145]. The key objective behind adversarial sample detection is to accurately detect and minimize the influence of adversarial samples on DNNs. The general framework for this detection is: (a) characterizing the adversarial perturbation; (b) formulating a score function based on the characterization; and (c) deciding on a threshold to accept or reject a sample as adversarially perturbed. In this section, we first review some seminal work in this field and then provide key insights into recent work. We summarize all the reviewed work in Table 4.1.

## 4.1 Use of Intermediate Representations

One of the earliest approaches utilize the features from intermediate layers of a DNN to detect adversarial samples. The authors of [109] train an adversary detector that receives inputs from the intermediate feature representations of a classifier. This detector aims to distinguish between samples from the original dataset and adversarial examples. The author considers two different scenarios: (i) a static scenario, where adversaries have access to the classification network and its gradients only; and (ii) a dynamic scenario, where adversaries have access to both the classification and the proposed detector network along with its gradients. The adversary detector network is trained in a supervised manner using training images and their corresponding adversarially perturbed images. The authors showed that a static detector cannot perform well against a dynamic adversary.

To tackle dynamic adversaries, the author proposes a dynamic adversary training method inspired by the approach in [49], where for each mini-batch the adversarial samples are computed on the fly. For each mini-batch, a dynamic adversary modifies a data point with a parameter 𝜎 selected randomly from the range [0, 1], as it depends on the gradient of the detector which evolves over time. By training the detector, one implicitly makes it resilient to dynamic adversaries for various values of 𝜎 . Although it shows excellent results, it does not show robustness against random noise. One key takeaway from this work is that supervised training with specific attack strategy is unlikely to hold against adaptive attackers and a robust characterization of adversarial perturbation is required to be utilized by the detector so that it can generalize across different attack mechanisms instead of overfitting to a single type of attack.

The detector sub-network proposed in [109] remains vulnerable to the adversarial samples that are not used during the detector training process. To address this issue, [102] proposes a new approach called SafetyNet that relies on the hypothesis that adversarial attacks work by producing different patterns of activation in late-stage Rectified Linear Units (ReLUs) to those produced by natural examples. As a result, it focuses on discrete codes produced by the quantization of each ReLU function at later stages of the classification network. SafetyNet consists of the original classifier along with an adversary detector that examines the internal state of the later layers in the original classifier.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00193v2-figure-16f7fdea-d796-4fce-ad12-ea4c8625d6ec.jpg)

| Rate, L : Code                                                          |                                    | Code                                                     | Code                                           |                                                   | Code                                            |                                   |                         | Code                                    | Code                                 | Code                      | Code                                | Code                                                                      |                                       |                        |                                                          |
|-------------------------------------------------------------------------|------------------------------------|----------------------------------------------------------|------------------------------------------------|---------------------------------------------------|-------------------------------------------------|-----------------------------------|-------------------------|-----------------------------------------|--------------------------------------|---------------------------|-------------------------------------|---------------------------------------------------------------------------|---------------------------------------|------------------------|----------------------------------------------------------|
| Equal Error Metrics                                                     | C                                  | A                                                        | C, E                                           | C                                                 | A, B                                            | A, E                              |                         | F, C                                    | A                                    | C, G                      | A, C, L                             | A, B                                                                      | A, D                                  | K, M, C                | C                                                        |
| Accuracy, I : Robust Average Precision, J : Recovery Rate , K Attack(s) | FGSM, BIM, DeepFool                | FGSM, BIM, Jacobian-based Saliency Map Attack (JSMA), CW | FGSM, DeepFool, Transferable attack (TA), BIM, | FGSM, JSMA, Support Vector Machine Attack (SVMA), | Decision Tree Attack (DTA), FGSM, BIM, JSMA, CW |                                   | FGSM, DeepFool          | FGSM, BIM, CW, Patching Attack, Glasses | FGSM, BIM, DeepFool, CW              | PGD, CW,                  | FGSM, JSMA, DeepFool, CW, Black-Box | FGSM, JSMA, DeepFool, CW, PGD, Elastic-net Attack to DNNs (EAD), Adaptive | FGSM, CW, JSMA                        | BIM, JSMA              | FGSM, BIM, PGD, CW, Fully Decision Based Boundary Attack |
| Reclassification Accuracy, H : Robust Dataset(s)                        | CIFAR10[78], 10-Class Imagenet[28] | MNIST[29], CIFAR10[78],                                  | SVHN[120] CIFAR10[78], Imagenet[28]            | MNIST[29], DREBIN[5],                             | MicroRNA[141]                                   | CIFAR10[78], MNIST[29], SVHN[120] | MNIST[29], F-MNIST[167] | VGGFace[17], LFW[66], CelebA[16]        | CIFAR10[78], CIFAR100[78], SVHN[120] | CIFAR10[78], Imagenet[28] | CIFAR10[78], MNIST[120]             | CIFAR10[78], CIFAR100[78], SVHN[120]                                      | MNIST[120], CIFAR10[78], CIFAR100[78] | Voxceleb1 [119]        | MNIST[120], FashinMNIST[167],                            |
| E : TPR, F : FPR, G : Detection Cost Function Task(s)                   | Image Classification               | Image Classification                                     | Image Classification                           | Image Classification                              |                                                 | Image Classification              | Image Classification    | Facial recognition                      | Image Classification                 | Image Classification      | Image Classification                | Image Classification                                                      | Image Classification                  | Speaker identification | Image Classification                                     |
| M : Year                                                                | 2017                               | 2017                                                     | 2017                                           | 2017                                              |                                                 | 2018                              | 2018                    | 2018                                    | 2018                                 | 2019                      | 2019                                | 2020                                                                      | 2020                                  | 2020                   | 2020                                                     |
| Success Rate, D : TPR(FPR@x), Label Change Rate, Work                   | Metzen et al. [109]                | Feinman et al. [42]                                      | Lu et al. [102]                                | Grosse et al.                                     | [52] et al. [104]                               | Ma                                | Zheng et al. [193]      | Tao et al. [152]                        | Lee et al. [83]                      | Roth et al.               | [135] Wang et al. [159]             | Cohen et al.                                                              | [25] Yang et al.                      | [173] Li et al. [90]   | Shumailov et al. [142]                                   |

Manuscript submitted to ACM

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00193v2-figure-c052cfa6-c490-4193-87e2-43ce7d162b29.jpg)

| Code       | Code                                                                   |                                                   |                        |                               | Code                                             | Code                                                              | Code                                  |                                     |                                                                                              |                  |                               |
|------------|------------------------------------------------------------------------|---------------------------------------------------|------------------------|-------------------------------|--------------------------------------------------|-------------------------------------------------------------------|---------------------------------------|-------------------------------------|----------------------------------------------------------------------------------------------|------------------|-------------------------------|
| Metrics    | A                                                                      | C                                                 | C                      | H                             | A, D                                             |                                                                   | A, E, F                               | A                                   | H, I, J                                                                                      |                  | C                             |
| Attack(s)  | FGSM, BIM, PGD, MIM, CW, Diversed Input Method (DIM), TIM              | Patch-PGD, Adversarial Patch, Adaptive Attack     | FGSM, PGD, CW          | FGSM, BIM, CW, JSMA, DeepFool | FGSM, BIM, PGD, CW, DeepFool, Hopskipjump (HOP), | Square Attack (SA), Spatial Transformation Attack (STA), Adaptive | FGSM, BIM, PGD, DeepFool, Auto Attack | FGSM, PGD, BIM, CW, Auto Attack, TA | Adversarial Patch, Localized and Visible Adversarial Noise (LaVAN), YOLO, Naturalistic Patch |                  | FGSM, BIM, DeepFool, JSMA, CW |
| Dataset(s) | Imagenet[28], LFW[66], CPLFW[191], CALFW[192], CFP[137], VGGFace2[17], | AgeDB-30[118], COCO[95] CIFAR10[78], Imagenet[28] | VGGFace2[17], LFW[66], | MNIST[29], CIFAR10[78],       | Imagenet[28]                                     | CIFAR10[78], CIFAR100[78]                                         | CIFAR10[78]                           | CIFAR10[78], Imagenet[28]           | Classification, Imagenet[28],Pascal VOC 07[40], INRIA[154],                                  | CASIA[143]       | CIFAR10[78]                   |
| Task(s)    | Image Classification, Face Recognition, Ob- ject Detection             | Image Classification                              | Facial Recognition     | Image Classification          |                                                  | Image Classification                                              | Image Classification                  | Image Classification                | Image                                                                                        | Object Detection | Image Classification          |
| Year       | 2021                                                                   | 2021                                              | 2021                   | 2018                          |                                                  | 2023                                                              | 2023                                  | 2023                                | 2023                                                                                         |                  | 2023                          |
| Work       | Deng et al. [30]                                                       | Li et al. [84]                                    | Mekala et al. [108]    | Xu et al. [125]               |                                                  | Picot et al. [129]                                                | Chyou et al. [23]                     | Zhang et al. [188]                  | Tarchoun et al.                                                                              | [153]            | Sun et al.                    |

Manuscript submitted to ACM

(a) Two simple "far away" 2D manifolds.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00193v2-figure-8e656c5f-7f59-41ab-b862-3fc059b6ae82.jpg)

(b) One manifold having a "pocket".

(c) Two simple "nearby" 2D manifolds.

Fig. 4. (a): The adversarial sample 𝑥 ∗ is generated by shifting away from the 'negative' submanifold and crossing the decision boundary (black dashed line), but it remains distant from the 'positive' submanifold. (b): the 'positive' submanifold has a 'pocket' and the adversarial sample 𝑥 ∗ lies in the pocket. (c): The adversarial sample 𝑥 ∗ is close to both the decision boundary and both submanifolds [42].

The detector in SafetyNet employs a Radial Basis Function Support Vector Machine (RBF-SVM) to discern adversarial examples based on binary or quaternary codes representing activation patterns. For the code denoted by 𝑏 , RBF-SVM uses the following formula to classify the samples:

$$f ( b ) = \sum _ { i } ^ { N } \alpha _ { i } y _ { i } \exp ( - \left \| \, b - b _ { i } \, \right \| ^ { 2 } / 2 \sigma ^ { 2 } ) + c$$

The detector hardly generates any gradient unless the attacking code, denoted as 𝑏 , closely resembles the code for a positive example 𝑏 𝑖 when the variance of the kernel, 𝜎 is very small. SafetyNet compels the attacker to solve a hard optimization problem that is discrete in nature. This work employs the idea of including non-differentiable operations so that the adversary cannot obtain the gradients necessary to compute adversarial perturbations.

## 4.2 Manifold-based detectors

In [42], the key intuition is that the adversarial samples lie on a different manifold than unperturbed samples. The authors argue that if after perturbation the data 𝑥 is transformed into 𝑥 ∗ , it can leave the manifold 𝑐 𝑥 in favor of sub-manifold 𝑐 𝑥 ∗ in one of the following three ways illustrated in Figure 4a:

- · 𝑥 ∗ is distant from the submanifold of 𝑐 𝑥 ∗ but closer to the classification boundary between 𝑐 𝑥 and 𝑐 𝑥 ∗ ;
- · 𝑥 ∗ lies closer to 𝑐 𝑥 ∗ submanifold but is still outside. On the other hand, 𝑥 ∗ is distant from the classification boundary that separates the classes 𝑐 𝑥 and 𝑐 𝑥 ∗ . As shown in Figure 4b, here one of the submanifolds has a pocket.
- · 𝑥 ∗ is close to the submanifold 𝑐 𝑥 ∗ , but is still outside. In addition, 𝑥 ∗ is close to the classification boundary which separates the classes 𝑐 𝑥 and 𝑐 𝑥 ∗ .

The authors estimate the manifold with Kernel Density (KD) estimation. They do so with the output of the last hidden layer based on the hypothesis presented in [45], which states that the deeper layers of a DNN offer more linear and 'unwrapped' manifolds compared to the input space. Given an input point 𝑥 and s set of training points 𝑋 𝑙 having label 𝑙 , the Kernel Density Estimator (KDE) ˆ can be obtained as 𝑓 ˆ 𝑓 = 1 | 𝑋 𝑙 | ˝ 𝑥 𝑖 ∈ 𝑋 𝑙 𝑘 𝑥 , 𝑥 ( 𝑖 ) , where 𝑘 ( ., . ) is the kernel function. The latter offers an indication of the distance between 𝑥 and the submanifold for 𝑙 . For the point 𝑥 , if the last hidden layer activation map is 𝜙 𝑥 ( ) , then the density estimate with predicted class 𝑙 is ˆ 𝑘 𝑥, 𝑋 ( 𝑙 ) = ˝ 𝑥 𝑖 ∈ 𝑋 𝑙 𝑘 𝜎 ( 𝜙 𝑥 ,𝜙 ( ) ( 𝑥 𝑖 )) where 𝜎 is a tunable bandwidth. While this approach exhibits effective performance in the detection of adversarial samples

Manuscript submitted to ACM

that are located far away from the 𝑐 𝑥 ∗ submanifold, it performs poorly in scenarios where adversarial samples 𝑥 ∗ lie in proximity to the 𝑐 𝑥 ∗ submanifold. As a result, in addition to the KDE, the author proposes Bayesian Uncertainty (BU) to identify low-confidence regions within the input space. The BU shows considerably different distributions for normal and adversarial samples - supporting the intuition of different manifolds for adversarial examples. The drawback of their method is that the uncertainty estimate hinges on the use of "dropout" which limits its application. Also, the performance is poor on more challenging datasets and stronger attacks.

## 4.3 Local Intrinsic Dimensionality (LID)

The objective of LID [104] is to characterize the specific regions where adversarial examples may be located. Specifically, this work shows that the KD approach adopted in [42] - which was based on the assumption that the adversarial subspaces are low probability regions - fails to detect some forms of adversarial attack. As an alternative, the authors propose LID to characterize the adversarial subspace. LID represents the dimension of the data submanifold local to the data point 𝑥 under consideration. In connection to the classical expansion models, treating the probability mass as a proxy for volume may provide information about the dimensional structure of the data. LID considers the Cumulative Distance Function (CDF) of the number of data points encountered 𝐹 ( 𝑑 ) , where 𝑑 is a realization of the random variable 𝐷 , i.e., the distance from data point 𝑥 to other data points. The LID of 𝑥 at a distance 𝑑 can be defined as:

$$L I D _ { F } ( d ) \preceq \lim _ { \epsilon \to 0 } \frac { \ln ( F ( ( 1 + \epsilon ) \cdot d ) / F ( d ) ) } { \ln ( 1 + \epsilon ) } = \frac { d \cdot F ^ { \prime } ( d ) } { F ( d ) }$$

where 𝐷 is a positive random variable and the CDF 𝐹 ( 𝑑 ) of 𝐷 is continuously differentiable at distance 𝑑 &gt; 0. The local dimension at 𝑥 in turn is defined as 𝐿𝐼𝐷 𝐹 = lim 𝑑 → inf 𝐿𝐼𝐷 𝐹 ( 𝑑 ) . The 𝐿𝐼𝐷 𝐹 quantifies how quickly the CDF 𝐹 ( 𝑑 ) grows with the distance 𝑑 . It can be approximated by considering the distances between a point 𝑥 and its 𝑘 nearest neighbors within the dataset. The work in [104] hypothesizes that for the estimation of LID of the adversarial samples, the nearest neighbors drawn should come not only from the manifold of the adversarial samples but also from the manifold of the normal samples, as the adversarial submanifold lies close to the data manifold. This will increase the dimension of the adversarial submanifold leading to higher value of LID. This approach exhibits better generalization across different attacks than KD. The authors train logistic regression model with LID feature to discriminate between perturbed and unperturbed samples. The major drawback is that it fails against stronger attacks, which indicates that the characterization of the adversarial manifold with LID is not universal for all adversarial perturbation.

## 4.4 Interpretability-based Approaches

Conversely from adopting adversarial sample training [42, 109], the work in [152] examines the adversarial samples from the DNN interpretability point of view. Specifically, a novel adversarial sample detection named Attack Meets Interpretability (AmI) has been proposed for face recognition. In this work, the main hypothesis is that adversarial samples utilize complex features extracted by the DNN that are imperceptible to humans. As such, AmI initially extracts a set of neurons called attribute witness , which are entangled with the face attributes. They substitute parts of the face from one image onto different images and look for unchanged neurons. This is named the Attribute Preservation step. Another step is to substitute the same parts from different images onto a single image - generating versions of the image only part-substituted. The neurons which change in this case are likely to be attributed to that part, which is the Attribute Substitution step. The common neurons obtained from attribute substitution and attribute preservation steps results in attribute witness neurons. The authors then construct an attribute-steered model by increasing the values of the witness neuron and decreasing the values of non-witness neurons. For a given test input, the inconsistency observed between the two models indicates that the input is adversarial. The major drawbacks of this approach are the Manuscript submitted to ACM

necessary manual feature inspection, absence of features in test image, and limited generalizability to setting other than face recognition. The work in [43] constructs Shapely Additive Explanations (SHAP)-based [103] feature attribution for the features of the penultimate layer. The knowledge base constructed from these SHAP signatures for both natural and adversarial images are used to train a binary classifier. The key drawback of this approach is the requirement of the SHAP signature from both natural and adversarially perturbed inputs. This creates a potential generalization problem for the binary classifier used for discriminating between the natural and adversarial inputs.

## 4.5 Statistical Approaches

The work in this category falls attempts to extract statistical information from different layers of the model to detect adversarial samples. The work in [52] proposes a statistical detection method based on Maximum Mean Discrepancy (MMD). The authors hypothesize that only a limited number of samples is needed to observe a measurable difference between normal and adversarial samples using a statistical test. They use a two-sample hypothesis test on the distribution of MMD values to detect the difference between the normal samples and the adversarial samples. To detect single examples, they augment their classifier with an outlier class and train using the adversarial samples. The work in [83] proposed a Gaussian Discriminant Analysis (GDA)-based approach where the model features from different layers of the DNN as class conditional multivariate Gaussian and calculate the confidence score for a sample as the Mahalanobis Distance between the sample and the closest class conditional distribution. The authors extract such score from all the layers and integrate them using weighted averaging. Both [52, 83] utilize adversarial samples either to train their detector or find appropriate weights. These approaches are also model and attack specific and require separate detectors for different attack approaches. This severely limits their applicability.

The work in [153] approaches the detection of adversarial patches from a statistical perspective in the input space. The authors postulate that adversarial patches should contain a statistically higher amount of information, from an information theory perspective, compared to any random neighborhood from a natural image distribution. This leads to the proposal of Jedi , which detects adversarial patches in images based on entropy thresholds. They use a 50-pixel × 50-pixel kernel and compute the entropy threshold dynamically based on the entropy distribution of the dataset and the image under consideration. The extracted high-entropy patch is passed through a sparse autoencoder for improved localization. Finally, they use coherence transport-based image inpainting [13] which aims at defusing the patch. They obtain high robust accuracy with respect to the baseline methods.

## 4.6 Influence-based approaches

The work in [25] proposes a novel adversarial sample detection strategy by using an 'influence function' [75]. This approach can be employed by any pre-trained DNN. The key intuition is that the training samples have close correspondence to the DNN classification. When this relationship is disrupted, it strongly suggests the presence of an adversarial input. As such, the influence function measures the impact of training data in the decision-making of the DNN. The influence of a training image 𝑥 on the loss of a specific test image 𝑥 𝑡𝑒𝑠𝑡 can be measured as 𝐼 𝑢𝑝,𝑙𝑜𝑠𝑠 ( 𝑥, 𝑥 𝑡𝑒𝑠𝑡 ) = -∇ 𝜃 𝐿 𝑥 ( 𝑡𝑒𝑠𝑡 , 𝜃 ) 𝑇 𝐻 -1 𝜃 ∇ 𝜃 𝐿 𝑥,𝜃 ( ) where 𝐿 is the loss function and 𝐻 is the Hessian of the machine learning model. The work also applies k-nearest

Fig. 5. The embedding space of a DNN [25].

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00193v2-figure-acfcdaf0-c271-466b-8dca-475c70c097a9.jpg)

neighbor (k-NN) classifier at the embedding space of the DNN as the resemblance of the nearest neighbor in the embedding space also dictates the decision of the DNN. The combination of the influence function and k-NN classifier Manuscript submitted to ACM

enables the detection of adversarial samples as the inference of a sample should be influenced most by the samples that are closest in the feature space. For adversarial samples, we observe the lack of this correlation.

Figure 5 illustrates this relation, where the black star and the brown X illustrate a normal and adversarial image from the CIFAR-10 validation set. We observe that the 25 nearest neighbors of the normal sample (blue circle) and the 25 helpful training samples (blue squares) for the normal sample lie very close in the Principal Component Analysis (PCA) projected embedding space. A sharp contrast to this relation is observed in the case of the adversarial sample. Although this approach exhibits good performance in terms of generalization, it requires longer computation time as it calculates the influence function for the entire training dataset. The work in [187] masks the pixels of the input image and computes pixel level feature attribution by measuring the change in the output. The authors expedite the feature attribution to input space by using different sampling strategies. All these approaches assume access to the adversarial samples, which limit the utility of such approaches as they fail to detect stronger attacks.

## 4.7 Other Notable Approaches

Input Space Approaches: [188] introduce a diffusion based perturbation method and derive Expected Perturbation Score (EPS). They show that the distribution of EPS is different for normal and perturbed images. The difference in perturbation is measured using MMD. This approach achieves an AUC of 1, and its performance does not degrade for unseen attacks. This is because EPS models the distribution of the input data itself instead of modeling the feature space. One drawback of the EPS score is that it cannot differentiate between adversarial perturbation and noise perturbation. The work in [173] proposes ML-Leave-One-Out (ML-LOO) feature attribution based detection of adversarial samples. The authors observe that feature attribution or mapping of importance of input features to the final prediction behaves differently for unperturbed and adversarially perturbed images. Adversarial perturbation disperses the feature attribution scores with significant deviation from the normal samples. Equipped with this observation, the authors use simple statistics to characterize the deviation of the adversarial samples as it progress through the DNN and aggregate the statistics with a logistic regression model to differentiate between adversarial and normal samples. Although the authors utilize the leave-one-out feature attribution, their approach is generic to any feature attribution method. The work in [184] trains a binary classifier using the saliency data by concatenating the saliency map to the raw image along the channel, while [171] proposes to squeeze input features using bit-depth reduction and spatial smoothing (both local and non-local variants). The work compares the output probability distributions on the original input and the feature-squeezed inputs using 𝐿 1 distance which changes significantly for adversarial inputs while normal samples show no change. This is because feature squeezing removes unimportant non-robust features improving robustness.

Model Space Approaches: The work in [159] proposed to modify the DNN and has implemented a self-contained toolkit named mMutant that integrates mutation testing and statistical hypothesis testing on DNNs. The key observation is that the sensitivity of the mutation on DNN is more acute for adversarial samples compared to unperturbed samples. If the DNN is slightly altered, there is a greater chance that the mutated DNN will alter the label of the adversarial sample than the unperturbed sample. The empirical investigation confirms this inherent sensitivity of the adversarial samples against a group of DNN mutants in terms of label change rate (LCR). However, with the increase of mutation rate, the distance of LCR between adversarial and normal samples decreases. This approach is also prone to the generation of some false positives during adversarial detection. Shumailov et al. [142] proposed a mechanism called Certifiable Taboo Trap (CTT), which incorporated the Taboo Trap detection, as well as numerical bound propagation. It prioritizes on finding the overexcited neurons being driven by adversarial perturbations outside of a predetermined range. The incorporation of numerical bound propagation on CTT certifies the detection bounds on activation values Manuscript submitted to ACM

of Convolutional Neural Network (CNN) against specific input perturbation sizes. The authors proposed three variants of CTT namely lite, loose, and strict. Although CTT-lite does not require fine-tuning, its defense capacity is limited. Further optimization through fine-tuning results in CTT-loose having most samples detected as adversarial. On the other hand, CTT-strict guarantees detection of adversarial samples withing a specific range of 𝑙 ∞ bound. Deng et al. [30] proposed Lightweight Bayesian Refinement (LiBRe) - a Bayesian Neural Network (BNN) based approach that does not require training an extra model. The authors combined the expressiveness of deep ensemble [80] and efficiency of last layer Bayesian learning [77]. Conversely from using the entire DNN for deep ensemble, they convert the last few layers - for example, the last residual block of ResNet50 - to a deep ensemble-like architecture. They also proposed to quantify the uncertainty with feature variance instead of softmax variance. As LiBRe does not use any adversarial samples for its training, it is adversarial attack agnostic and treats the adversarial samples as OOD data. This broadens its application scenario to more than classification with no modification which is not observed for the supervised detection methods.

Finally, in [129] the authors proposed Halfspace Mass Depth Detector (HAMPER) to characterize the adversarial samples. They calculated the class conditional half-space mass depth for a subset of DNN layers and aggregate the scores as weighted sum. To tune the weight values, they utilized adversarial samples, and showed state-of-the-art performance in both attack-aware and blind-to-attack scenarios. It is shown that the last layers carry more important information for the detection of adversarial samples. HAMPER has the advantage of being robust against adaptive attacks. However, this work does not provide any understanding as to why the half-space mass depth metric is able to capture the difference between the adversarial and normal samples.

Output Space Approaches: Prior work has attempted to characterize adversarial examples in the output space to facilitate their detection. I-defender [193] modeled the distribution of the output of the linear layers and shows that the distributions are different for the normal and adversarially perturbed images. The authors used a mixture of Gaussian models to approximate the Intrinsic Hidden State Distribution (IHSD) for each class. If the class conditional probability is lower than a threshold for a sample, it is detected as adversarially perturbed. However, this approach performs poorly under moderate to strong attacks. Roth et al. [135] proposed a statistical metric for the detection of adversarial samples based on expected value of perturbed log-odds. They showed that the robustness properties of perturbed log-odds statistics are different for natural and adversarial samples. The idea is that geometrically optimal adversarial manipulations are embedded into a cone-like structure they call " Adversarial Cone ". They also reported an intriguing finding, i.e., that adversarial samples are much closer to the ground truth unperturbed class than any other class. Based on these observations, they proposed the maximum expected deviation of the perturbed log-odds from its expected value as an indicator of an adversarial sample. Chyou et al. [23] proposed an unsupervised adversarial sample detection method without any extra model. They proposed new training losses to improve detection accuracy. The main idea is to remove unnecessary features for false outputs and strengthen the true outputs. This is achieved by forcing all the false raw outputs in a mini-batch to have a uniform distribution during training. By doing so, false outputs become adversarially robust and only true outputs can be attacked. Any attack on the true output changes the raw false output values triggering an adversarial detection. The proposed training loss keeps the accuracy of the original classification task almost the same, around 86% when using ResNet18 architecture and CIFAR10. Although the authors used adversarial examples to determine their threshold, they show that the estimated threshold for their binary detector generalizes for other stronger attacks. The achieved accuracy on CIFAR10 dataset for Resnet architecture falls short of the reported accuracy of &gt;90% [55] although this can be related to the training strategy.

## 5 DETECTION OF OOD SAMPLES

Existing approaches for OOD detection mostly operate in the output space. Indeed, it has been shown that working with activations of earlier layers does not provide much improvement in detector performance [68, 148]. In this section, we first provide the discussion of seminal work in OOD detection, and then focus on the state of the art approaches. Table 2 provides a summary of the surveyed work regarding OOD detection.

## 5.1 Input Space Approaches

Among the few works in this space, Li et al. [86] forced the classifier to implicitly learn the ID data distribution instead of just learning features for classification. They applied a preprocessing step of masking and then learn to reconstruct the original image from the masked image. This calibrates the DNN for ID and OOD samples. Gao et al. [44] utilized the diffusion model to learn the distribution of ID data. They learned a Denoising Diffusion Implicit Model (DDIM) conditioned to the semantic labels, while during inference time, they inverted the image to obtain the latent representation using the DDIM and them reconstruct it from the latents.

## 5.2 Output Space Approaches

ODIN [93] is among the earliest work in the domain of OOD detection in the output (or logit) space. It adopted the baseline [60] where the authors utilize the softmax probability score to distinguish between ID and OOD samples and improve it by incorporating temperature scaling and input pre-processing steps. ODIN [93] shows that by manipulating the temperature parameter 𝑇 ∈ R + , it is possible to increase the separation between the ID and OOD examples. The resulting score function 𝑆 ˆ 𝑦 ( x ; 𝑇 ) is given by Equation 8:

$$S _ { i } ( x ; T ) = \frac { \ e x p ( f _ { i } ( x ) / T ) } { \sum _ { j = 1 } ^ { j = N } \ e x p ( f _ { j } ( x ) / T ) } \\ S _ { \hat { y } } ( x ; T ) = \max _ { i } S _ { i } ( x ; T )$$

Here, 𝑓 𝑖 ( x ) denote the logit value corresponding to i-th class for DNN f . The authors complemented the temperature scaling by perturbing the input image before feeding it into the DNN. The perturbation procedure was inspired by [49], which adds the perturbation to decrease the softmax score but here the perturbation boosts the softmax score prediction. The perturbation follows ˜ x = x -𝜖 sign (-∇ x log 𝑆 ˆ 𝑦 ( x ; 𝑇 )) .

Theauthors explained the effect of temperature scaling using 𝑈 1 = 1 1 -𝑁 ˝ 𝑖 ≠ ˆ 𝑦 [ 𝑓 ˆ 𝑦 ( x )-𝑓 𝑖 ( x )] and 𝑈 2 = 1 1 -𝑁 ˝ 𝑖 ≠ ˆ 𝑦 [ 𝑓 ˆ 𝑦 ( x )-𝑓 𝑖 ( x )] 2 . They showed that ID data contains some classes that are similar to each other, resulting in a higher value of 𝑈 2 even when 𝑈 1 value for ID and OOD data is the same. By taking the Taylor approximation, the softmax function can be expressed as 𝑆 ˆ 𝑦 = 1 𝑁 -( 𝑈 1 -𝑈 2 2 𝑇 )/ 𝑇 . This suggests that for very high values of 𝑇 , the softmax score is dominated by the term 𝑈 1 compensating the negative effect of 𝑈 2 on the detection performance. They argue that this makes the ID and OOD data more separable. The authors argue from the Taylor expansion of the log softmax of the perturbed input ˜ 𝑥 that ID images have a larger norm of the gradient of the score function compared to the OOD images which results in a higher score function values after perturbation.

$$\log S _ { \dot { y } } ( \tilde { x } ; T ) = \log S _ { \dot { y } } ( x ; T ) + \epsilon | | \nabla _ { x } \log S _ { \dot { y } } ( x ; T ) | | _ { 1 } + o ( \epsilon )$$

The key observations are (i) DNNs produce outputs with larger variance for ID examples; and (ii) DNNs have larger value of the gradients of the log-softmax score when applied to ID images.

Generalized ODIN [65] improved [93] without utilizing any OOD samples. The authors point out the limitation of the softmax classifier, as it is an approximation of the indicator function it gives a categorical distribution rather Manuscript submitted to ACM

Table 2. Summary of the major OOD detection methods. Here, the mapping of alphabets to datasets is as follows A = TinyImageNet [82], B = LSUN [178], C = iSUN [170], D = Places365 [194], E = CIFAR10 [78], F = CIFAR100 [78], G = Imagenet-1k [28], H = SVHN [120], I = Textures [24], J = Gaussian Noise, K = Uniform Noise, L = DomainNet [128], M = iNaturalist [64], N = ImageNet-21k [133], O = MS-COCO [95], P = OpenImages [76], Q = Pascal-VOC [40], R=BDD-100k [177], S = Flowers-102 [122], T = Caltech256[51]

∗ The authors use alternately one dataset as ID and the rest as OOD dataset similar to leave-one-out approach Manuscript submitted to ACM

| Work                    | Task(s)                                                  | ID Dataset(s)   | OOD Dataset(s)                                            | Base Model                                                       | Metrics                               | Code   |
|-------------------------|----------------------------------------------------------|-----------------|-----------------------------------------------------------|------------------------------------------------------------------|---------------------------------------|--------|
| Liang et al [93]        | Image Classification                                     | E, F            | A, B, C, J, K                                             | DenseNet-BC(k=12) WideResNet-28-10                               | FPR@95%TPR Detection Error AUROC AUPR | code   |
| Liu et al [96]          | Image Classification                                     | E, F            | B(Crop), B(Resize), C, D, H, I                            | WideResNet                                                       | FPR@95%TPR AUROC AUPR ID Test Error   | code   |
| Hsu et al. [65]         | Image Classification                                     | E,F,L           | A (crop), A (resize), B (Crop), B (Resize), C, H, L, J, K | DenseNet-BC(k=12) ResNet WideResNet-28-10                        | TNR@95%TPR AUROC                      | code   |
| Huang et al. [68]       | Image Classification                                     | E, F, G         | B (crop), C, D, H, I, M                                   | Google BiT-S DenseNet-121 ResNet-20                              | FPR@95%TPR AUROC                      | code   |
| Sun et al. [148]        | Image Classification                                     | E, F, G         | B (Crop), B(Resize), D, H, I, M                           | ResNet-50 ResNet-18                                              | FPR@95%TPR AUROC AUPR                 | code   |
| Wei et al. [164]        | Image Classification                                     | E, F            | B (crop), B (resize), C, D, H, I                          | WideResNet-40-2 ResNet-34                                        | FPR@95%TPR AUROC AUPR                 | code   |
| Djurisic et al. [31]    | Image Classification                                     | E, F, G         | B (crop), B (resize), C, D, H, I, M                       | DenseNet-101 ResNet-50 MobileNetV2                               | FPR@95%TPR AUROC AUPR                 | code   |
| Hendryks et al. [57]    | Image Classification Multi-label Prediction Segmentation | N, O, Q         | Species(Subset of M) Subset of N                          | DenseNet-101 ResNet-50 MobileNetV2                               | FPR@95%TPR AUROC AUPR                 | code   |
| Du et al. [37]          | Object Detection                                         | Q, R            | O, P                                                      | ResNet-50 RegNetX-4.0GF                                          | FPR@95%TPR AUROC                      | code   |
| Li et al. [86]          | Image Classification                                     | E, F, G         | B, D, E, F, H, S, T                                       | ResNet-50 RegNetX-4.0GF                                          | AUROC                                 | code   |
| Hendrycks et al. ∗ [61] | Image Classification                                     | A, D, E, F, H   | D, E, F, H                                                | PixelCNN++                                                       | FPR@95%TPR AUROC AUPR                 | code   |
| Wang et al. [160]       | Image Classification                                     | E, F, G         | B (Crop), C, D, H, M                                      | WRN-40-2 ResNet-50                                               | FPR@95%TPR AUROC                      | code   |
| Zhu et al. [197]        | Image Classification                                     | E, F            | B (Crop), B(Resize), D, H, I                              | WRN-40-2                                                         | FPR@95%TPR AUROC AUPR                 | code   |
| Du et al. [35]          | Object Detection                                         | Q, R            | O, P                                                      | DDETR                                                            | FPR@95%TPR AUROC                      | code   |
| Sun et al. [149]        | Image Classification                                     | E, F            | B (Crop), C, D, H, I                                      | ResNet-18                                                        | FPR@95%TPR AUROC                      | code   |
| Liu et al. [97]         | Image Classification                                     | E, F, G         | B (Crop), C, D, G, H, I, M, P                             | BiT-S-R101 × 1 ViT-B/16 RepVGG-B3 ResNet-50-D DeiT-B/16 Swin-B/4 | FPR@95%TPR AUROC                      | code   |

than a continuous distribution over the classes. To circumvent this limitation, the authors reformulated the posterior probability by incorporating domain of the input as a random variable, changing it from 𝑝 𝑦 ( | x ) to 𝑝 𝑦 ( | x , 𝑑 𝑖𝑛 ) , where 𝑦, x , and 𝑑 𝑖𝑛 represent the output, the input and the input domain. This lead to the decomposition of the posterior as 𝑝 𝑦 ( | x , 𝑑 𝑖𝑛 ) = 𝑝 ( 𝑦,𝑑 𝑖𝑛 | x ) 𝑝 𝑑 ( 𝑖𝑛 , x ) . Due to the lack of out-of-domain knowledge, the authors proposed to utilize the prior knowledge of the dividend/divisor structure of the posterior to provide the classifier capacity to decompose the confidence of the class probability. They modeled the logits 𝑓 𝑖 ( x ) as 𝑓 𝑖 ( x ) = ℎ 𝑖 ( x ) 𝑔 ( x ) . While the work shows improvement over [60, 83, 93], its key limitation is that it needs to change the structure of the DNN and perform training to learn ℎ 𝑖 ( x ) and 𝑔 ( x ) .

Liu et al. [97] proposed a generalized entropy-based approach. Specifically, the authors used the generalized entropy family defined by 𝐺 ( p ) = ˝ 𝑖 𝑝 𝛾 𝑖 ( 1 -𝑝 𝑖 ) 𝛾 , where p denotes a categorical probability distribution. This score function, coupled with truncation of very small probabilities, showed competitive result for state-of-the-art models like Swin Transformer and BiTS R101 × 1 on ImageNet benchmark. The work in [57] addressed the fact that

Fig. 6. Decomposing the logit values into dividend/divisor structure. g(x) allows incorporating domain knowledge into training and disentangles the probability over the domain and probability over the classes. [65]

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00193v2-figure-212a24bc-7e2c-4d5e-8d28-ba0cd6ca44dc.jpg)

OOD detection schemes underperform in real-world settings. This work introduces a new dataset called Species consisting of 700,000 images and over 1,000 anomalous species to test OOD detection performance on the ImageNet dataset. They proposed to use the negative of the maximum of the logit values (Max Logit) of a DNN as the score to distinguish between ID and OOD samples and established a baseline for large-scale setting. They also introduced new benchmarks for anomaly segmentation and OOD detection in multi-label prediction. Decoupling max-logit [189] decouples the magnitude and the direction of the logit vector. This decomposes the max-logit score into product of max cosine (cosine of the angle between the feature vector and respective weights for each class) and max norm (norm of the product between feature vector and respective weights for each class). While max-cosine outperforms max-logit score consistently, max norm falls short by a large margin.

GradNorm [68] is among the approaches using the gradients of the parameters of the DNN. The approach utilizes a label-agnostic score function to formulate the OOD detection problem as a binary classification problem. For a DNN parameterized with w , GradNorm formulates the score function for an input x as 𝑆 ( x ) = | | 𝜕𝐷 𝐾𝐿 ( u | | 𝑠𝑜𝑓 𝑡𝑚𝑎𝑥 ( 𝑓 ( x ) ) ) 𝜕 w | | 𝑝 , where u denotes the uniform distribution and 𝐷 𝐾𝐿 denotes the KL-divergence. The intuition behind this formulation is that the prediction tends to concentrate around the target class for ID inputs which should result in a larger value of the KL divergence and its derivative. The parameter vector w contains the concatenated parameters from different layers in a single vector regardless of their original shape. The authors further show that the gradients from the last layer's parameters are sufficiently informative for distinguishing between the ID and OOD inputs. Through empirical results, they showed that the 𝐿 1 norm is the most effective for GradNorm. The authors attribute the improvement brought by GradNorm to the joint information from the feature space and the output space that is utilized in this framework. Specifically, they showed that the score function can be written as equation 14, where the 𝑥 𝑖 s are the input features to the logit layer, 𝐶 is the number of classes, 𝑇 is the temperature, and 𝑓 𝑗 denotes the logit for class 𝑗 . 𝑈 = ˝ 𝑚 𝑗 = 1 | 𝑥 𝑖 | and 𝑉 represent the information about the feature space and output space respectively. Manuscript submitted to ACM

## 5.3 Approaches based on Energy Functions

The work [96] provides a solution to OOD detection problem in the logit space. It connects the posterior probability in Gibb's distribution and the posterior probability of the softmax layer and proposes the equivalent of Helmholtz energy function for the softmax function as the score function for discriminating between ID and OOD samples. The key observation is the similarity between Gibb's density function in Equation 10 and the softmax function in Equation 11. By connecting these two equations, the energy for a specific sample is defined by 𝐸 ( x , 𝑦 ) = -𝑓 𝑦 ( x ) :

$$p ( y | x ) = \frac { e ^ { - E ( x, y ) / T } } { \int _ { y ^ { \prime } } e ^ { - E ( x, y ^ { \prime } ) / T } } = \frac { e ^ { - E ( x, y ) / T } } { e ^ { - E ( x ) / T } }$$

$$p ( y | x ) = \frac { e f _ { y } ( x ) / T } { \sum _ { i = 1 } ^ { i = K } e ^ { f _ { i } ( x ) / T } } \quad & & ( 1 1 ) \\.. \quad \dots \quad. \quad. \quad. \quad. \quad. \quad. \quad.$$

This also leads to the equivalent of free-energy 𝐸 ( x ) for DNN as 𝐸 ( x ; 𝑓 ) = -𝑇 ˝ 𝑖 = 𝐾 𝑖 = 1 𝑒 𝑓 𝑖 ( x )/ 𝑇 . The authors show that the Energy function is affinely related to the logarithm of the probability of the input 𝑝 ( x ) . On the other hand, the logarithm of the softmax score or the softmax score itself is not related to 𝑙𝑜𝑔 𝑝 ( x ) linearly and depends on the maximum logit value. The authors also proposed an energy-based regularizer for training the DNN, shown in Equation 12 and Equation 13, and demonstrated that it improves the performance of the detector.

$$\ m i n _ { \theta } \, \Xi _ { ( \mathbf x, y ) \sim \mathcal { D } _ { i n } ^ { t r a i n } } [ - l o g F _ { y } ( \mathbf x ) ] + \lambda L _ { E n e r g y }$$

$$L _ { E n e r g y } = \mathbb { E } _ { ( x, y ) \sim \mathcal { D } _ { i n } ^ { t r a i n } } ( m a x ( 0, E ( x _ { i n } ) - m _ { i n } ) ) ^ { 2 } + \mathbb { E } _ { ( x, y ) \sim \mathcal { D } _ { o u t } ^ { t r a i n } } ( m a x ( 0, m _ { i n } - E ( x _ { o u t } ) ) ) ^ { 2 }$$

Despite having the advantage of being easy in nature, the energy-based OOD detection method requires access to the OOD distribution either to determine the threshold or to train the network with regularization.

## 5.4 Approaches based on Activation Shaping

Using True Statistics

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00193v2-figure-e4703fd4-c715-4eeb-928b-3f0e13cbed1c.jpg)

These approaches assume that the features of a DNN affect the output differently for ID and OOD samples. As such, they rely on some predefined transformation of the activation values of intermediate layers of DNN to differentiate between ID and OOD samples. ReAct [148] is a rectification operation on the activation in model space to facilitate the detection of OOD samples by making ID and OOD samples more separable. The activations from the penultimate layer ℎ ( x ) of the DNN are truncated using ReAct operation given by ¯ ℎ ( x ) = ReAct ( ℎ ( x ; 𝑐 )) , where ReAct ( 𝑥 𝑐 ; ) = 𝑚𝑖𝑛 𝑥,𝑐 ( ) .

Fig. 7. The per-unit activations for the penultimate layer for OOD data (iNaturalist) are considerably different when using true (top) statistics as opposed to mismatched (bottom) BatchNorm statistics [148].

The rectified activations are used to obtain the model output as 𝑓 𝑅𝑒𝐴𝑐𝑡 ( x ; 𝜃 ) = W 𝑇 ¯ ℎ ( x ) + b . These outputs can be used with any score function for OOD detection. The detection performance depends on the threshold value 𝑐 . Through empirical study, the authors showed that setting the value of 𝑐 to the 90th percentile of the activations works best for ReAct. The authors theoretically showed that ReAct suppresses the activations more for the OOD samples than ID samples. This translates to a larger reduction in the output value for the OOD samples making them separable from the ID samples. When batch-normalization statistics calculated for ID data are applied to OOD data, significantly different activation patterns emerge as shown by the authors in Figure 7. The approach is also shown to work for DNN architectures using normalization techniques other than batch-normalization (e.g. weight normalization, group normalization).

Activation Shaping (ASH) [31] modifies the activations of a DNN in a post-hoc manner to facilitate the detection of OOD samples using existing scoring methods. The work assumes that modern overparameterized DNN produce redundant representations for the task it is trained for. Therefore, the representation can be greatly simplified while preserving performance and providing additional benefits in OOD detection. Based on this hypothesis, the authors propose three variants of the ASH algorithm where they set values smaller than p-th percentile of the representation to 0 as a form of activation pruning. The first variant, ASH-P , keeps the un-pruned activations unchanged. In the second variant, ASH-B , the un-pruned activations are assigned a value such that their total value equals that of the total value of the original activations. The third variant, ASH-S , calculates 𝑠 1 and 𝑠 2, the sums of the activations before and after pruning, and scales the unpruned values with 𝑒𝑥𝑝 𝑠 ( 1 / 𝑠 2 ) . They apply this activation shaping to the penultimate layer of the DNN. The effect of the choice of the value of 𝑝 for setting the pruning threshold is dependent on the architecture and dataset and takes values in the range of [65, 95]. While this method is relatively simple, it integrates the detector into the architecture impacting the performance on ID data. The unique perspective of this method indicates that sparsity can be beneficial to OOD detection.

$$S ( { \mathbf x } ) = \frac { 1 } { C T } ( \sum _ { i = 1 } ^ { m } | { \mathbf x } _ { i } | ) ( \sum _ { j = 1 } ^ { C } | 1 - C. \frac { e ^ { f _ { j } / T } } { \sum _ { j = 1 } ^ { C } e ^ { f _ { j } / T } } | ) \stackrel { \Delta } { = } \frac { 1 } { C T } U \cdot V$$

This approach is label-agnostic, OOD data agnostic, and can be utilized using back-propagation. Conversely, other approaches that are comparable in accuracy require considerable computational overhead and also require access to OOD data [83]. The limitation of this work lies in the assumption that the softmax output probability would be more uniform for OOD data. It has been shown by [121] that DNN can assign a high probability to a specific class even when presented with random noise. This limitation is reflected in the results presented in the work where it is shown that GradNorm still reports False Positive Rate (FPR) of 43.16% at 95% True Positive Rate (TPR).

Similar to [31], the work in [1] suppresses the irrelevant activations. However it also suppresses weights based on Shapley value [140]. The impact on the performance for ID requires further study in such cases where the DNN architecture is changed. The work [198] takes a similar approachwhere they assume that the deep features follow a Gaussian distribution. They define features that fall into the high-probability region as typical features and they estimate the typical features with the batch-normalization layer statistics. For a batch-normalization layer with mean 𝜇 and standard deviation 𝜎 , the features that are within 𝜆𝜎 from 𝜇 are estimated to be the typical features. The features are then clamped to have values between 𝜇 -𝜆𝜎 and 𝜇 + 𝜆𝜎 meaning that only the typical features can pass unperturbed.

Variational Rectified Activation (VRA) [169] adopts a piece-wise activation shaping function similar to [198]. The difference is that they clamp the lower values to zero and shift the intermediate values by a fixed amount 𝛾 . The inspiration to shift the intermediate values comes from the variational formulation of the activation-shaping function, which shows the need to suppress abnormally low and high values and boosting intermediate feature values. Both [198] and [169] are inspired by ReAct [148]. The recent work [190] formulates the design of feature-shaping functions as an optimization problem and shows that the previous approaches [31, 148] approximate the optimal solution. The authors provide two versions of the optimization problem. The first one utilizes the OOD data and the second one is OOD data-free approach. For the OOD data-free approach, the authors also provide a closed-form solution.

## 5.5 Approaches based on Mitigating Overconfidence in Prediction

LogitNorm [164] observes that the norm of the logit vector increases as the training progresses. As up-scaling the logits increases the maximum softmax probability, to normalize the logit values to address the issue. Their key idea is to decouple the influence of the norm of the output from the training objective and optimization step. They decomposed Manuscript submitted to ACM

the logit vector f into its norm and directions as given by f = | | f | | · ˆ . f They proposed to constrain the logit norm to a fixed value 𝛼 , leading to the constrained optimization problem given by Equation 15:

$$\minimize \mathbb { E } _ { \mathcal { P } _ { X y } } [ \mathcal { L } _ { C E } ( f ( x ; \theta, y ) ) ] \\ \text{subject to } | | f ( x ; \theta ) | | _ { 2 } = \alpha$$

They showed that this constrained optimization does not work in practice and propose to use the logit vector normalized with its 𝐿 2 norm instead. This leads to the optimization objective given by Equation 16:

$$\mathcal { L } _ { l o g i t - n o r m } ( f ( x ; \theta ), y ) = - \log \frac { e f _ { y } / ( \tau | | f | | ) } { \sum _ { i = 1 } ^ { k } e f _ { i } / ( \tau | | f | | ) }$$

where the temperature parameter 𝜏 modulates the logit-norm. In this case, the compounded temperature parameter 𝜏 | | f | | shows dependence on the input as f = 𝑓 ( x ; 𝜃 ) . The authors argued that the input-dependent temperature scaling provides superior performance in terms of OOD detection and calibration of the output. While providing a simple fix to the overconfidence issue of the DNN, LogitNorm approach has the drawback that the output distribution gets shifted due to the normalization. The paper does not provide any theoretical guarantees for improvement in calibration performance. While this approach can improve the performance of different scoring functions, it requires modification of the training objective. Hence, it cannot be applied to pretrained DNNs.

Masked image modeling for OOD Detection (MOOD) [86] shows that the overconfident prediction in the OOD samples is due to incomplete learning of the ID features. As such, it can be mitigated by using masked image modeling (MIM) as a pretext task. The authors argued that using the reconstruction of the masked image as the pretext task forces the DNN to learn the ID data distribution instead of just patterns for classification and this improves the OOD classification performance. The Mahalanobis distance [83] proves to be the most effective. This suggests a Gaussian distribution of features and well-separated class boundaries. The overall approach of MOOD consists of four steps - a) pretraining the Masked Image Modelling Vision Transformer (ViT) on ImageNet-21k., b) intermediate fine-tuning of ViT on ImageNet-21k, c) fine-tuning of pre-trained ViT on the ID dataset, and d) extracting features from the pre-trained ViT and using Mahalanobis distance metric to detect OOD samples. Despite the superior performance shown by MOOD, a shortcoming is that only the Area Under Receiver Operating Characteristics (AUROC) is used as the evaluation metric.

The work in [37] addresses the problem of lack of supervision from OOD data during training resulting in overconfident predictions on data from unknown domains. The authors propose an unknown-aware training approach for object detection tasks without accessing OOD data. Conversely, they create virtual outliers to regularize the model. They model the feature representation ℎ ( x b , ) of object instances x ( b is the bounding box) with class-conditional multivariate Gaussian distribution having mean 𝜇 𝑘 , and covariance matrix Σ . After fitting the features from the penultimate layer of the network to the class conditional Gaussian distributions, they sample feature instances ( v ) from these distributions to use for regularizing the DNN. For regularization, they use an uncertainty loss L 𝑢𝑛𝑐𝑒𝑟𝑡𝑎𝑖𝑛𝑡𝑦 as follows:

$$\mathcal { L } _ { u n c e r t a i n t y } = \mathbb { E } _ { \nu \sim \mathcal { V } } [ - \log \frac { 1 } { 1 + e x p ^ { - \phi ( E ( \nu ; \theta ) ) } } ] + \mathbb { E } _ { \mathbf x \sim \mathcal { D } } [ - \log \frac { e x p ^ { - \phi ( E ( \mathbf x ; \theta ) ) } } { 1 + e x p ^ { - \phi ( E ( \mathbf x ; \theta ) ) } } ],$$

where V and D denote the virtual outlier and training datasets respectively, while 𝜙 (·) is a non-linear MLP function that allows learning flexible energy surface. This work provides a flexible approach for use in OOD detection for object detection tasks, which is an underexplored area. However, the performance still needs to improve significantly. Indeed, the reported FPR gets close to 50%, which may be unacceptable in real-world tasks.

## 5.6 Approaches based on Training with Regularization

Some prior work assumes access to OOD data - also referred to as surrogate OOD data - that can be used at training time. Specifically, these approaches propose a regularization term to be added to the training loss that helps improve detection of OOD samples. The key issue with surrogate OOD data is that the OOD distribution may not be fully characterized. As a result, these methods perform poorly on unseen OOD distributions. Outlier Exposure (OE) [61] is the first to propose to use such surrogate data. They assumed to have training OOD data that is disjoint from the test OOD data and proposed a generalized training scheme using the regularizer L 𝑂𝐸 , which is task-dependent. The authors set L 𝑂𝐸 to cross-entropy loss between the DNN output and uniform distribution for supervised learning tasks. If the task is of density estimation and without any label, then L 𝑂𝐸 is set to margin ranking loss between output of DNN 𝑓 -𝑓 ( 𝑥 ) and 𝑓 ( 𝑥 ′ ) , where 𝑥 and 𝑥 ′ are sampled from ID and surrogate OOD distribution respectively.

Distributional-agnostic Outlier Exposure (DOE) [160] proposes an improvement over OE by making it distributionagnostic. DOE is composed of OOD generation using model perturbation and worst OOD regret (WOR) based training. The perturbation is done with a matrix multiplying the DNN parameters, which leads to features resembling ones sampled from the OOD distribution. The WOR score measures the worst performance of the OOD detector and helps discover the hardest OOD samples. The hardest OOD samples are simulated by perturbing the model parameters. Diversified Outlier Exposure (DivOE) [197] took a similar approach as DOE. However, instead of generating new OOD samples through model perturbation, they perturbed a portion of the surrogate OOD data and use a mix of the perturbed (that maximize L 𝑂𝐸 loss) and unperturbed surrogate data to train the DNN.

The work by Choi et al. [22] proposed a balanced energy regularization loss 𝐿 𝑒𝑛𝑒𝑟𝑔𝑦,𝑏𝑎𝑙 , which is built upon the energy regularization loss 𝐿 𝑒𝑛𝑒𝑟𝑔𝑦 proposed in [96]. The authors fine-tuned the linear layers of a DNN with auxiliary samples (i.e. OOD samples) and used the proposed loss as a regularization factor, which increases the energy margin between ID and OOD samples. The proposed loss accounts for the imbalance in the OOD data used for training the DNN. However, such training requires access to OOD data and is computationally expensive. Another line of work considers adding regularizers to the loss function without using any surrogate OOD data. Conversely, these approaches impose constraints on latent representations of ID data that OOD data are unlikely to achieve. For example, [181] proposed to constrain the embedding generated by DNN to a union of 1-dimensional subspaces where inputs from each class occupy one dimension. The authors achieve such embeddings by imposing a cosine similarity constraint between the embeddings and the weight vector corresponding to the class it belongs to. They also imposed orthogonality constraints among the weight vectors of different classes so as not to impact the ID accuracy. The OOD samples are detected by estimating the probability of a test sample belonging to one of the classes using spectral discrepancy (angular deviation of the embedding from first singular vector corresponding to a class) measurement. If the probability of the sample belonging to any class is very low (ideally zero), then it is declared to be an OOD sample.

Du et al. [35] proposed Siren , which added a projection head after the penultimate layer of the DNN. Such projected features are modeled with a von Misses-Fisher (vMF) distribution [107] and trained to align with class conditional priors. The score function is designed with maximum class conditional likelihood, while the regularizer used to train the projections ensures that the ID representations align with their class conditional priors, which is unlikely for OOD samples. Finally, the work in [111] proposed CIDER, which utilized the distance from class prototypes in a hyper-spherical embedding space. The loss function in CIDER is divided into two sections. The first one makes the embedding compact around each class prototype, while the second increases the angular distance among the class prototypes. The authors show that using the two part loss provides better separability between ID and OOD samples.

## 5.7 Approaches based on Distance Functions

The intuition behind distance-based OOD detection is that the latent representation of the ID and OOD samples can be divided into separate clusters. As such, the distance between representations of ID samples should be smaller than the distance between ID and OOD representations. The typical distances used are Mahalanobis distance , Hamming distance , and L2 distance. As these approaches typically use class-conditional prototypes, they are a form of prototype learning .

Lee et al. [83] proposed to model the features using class-conditional Gaussian distributions and use the Mahalanobis distance as the score for OOD detection. The distance is calculated for each layer separately, while auxiliary OOD data is used to fit the distances collected from different layers into a logistic regression model which greatly diminishes its utility as it requires access to the OOD distribution to be detected and trains a separate logistic regression model for each OOD distribution. In contrast, [149] proposes to use L2 distance of the test sample to the 𝑘 -th nearest neighbor from the training dataset, thus leading to a DNN- and OOD-agnostic approach. The authors show that the performance of the KNN based OOD detector improves if the model is trained with contrastive learning. This is because the features form tighter clusters and the distance of the OOD data from any particular cluster increases.

Ming et al. [110] proposed an approach based on cosine similarity, where they measure the alignment of the features of an input image with concept vectors obtained from the language model. The similarity score is then used in place of logit to calculate the softmax probability over the concepts from the language model. The similarity represents the probability of the image aligning with a specific concept. The maximum of these probabilities work as the score for OOD detection. The work in [35, 111] uses von Mises-Fisher distribution to model the class prototypes in hyper-spherical embedding space. The recent work [8] used a similar loss function as [111] and showed that it improves the OOD generalization performance of DNN. Conversely, Lu et al. [101] showed that using a single class prototype is detrimental to learning the representation properly. The authors propose to use multiple prototypes for each class and assign each sample to a prototype using soft assignment. They also model the clusters using vMF and use Mahalanobis distance as their OOD scoring metric. Gomes et al. [47] proposes to use geodesic distance between two data distributions - i.e., the Fisher-Rao distance [7] - arguing that this information geometric distance is more suitable for measuring difference between two distributions than the Mahalanobis distance. Finally, Olber et al. [123] proposed an approach based on the concept of neural activation pattern . Specifically, the authors extracted activations from all layers of a DNN, binarize them based on a threshold value, and concatenate them in a vector. If the Hamming distance of this binary vector, which represents the activation pattern of the DNN, to the nearest ID sample is greater than a certain threshold, the input sample is considered as an OOD sample.

## 5.8 Other Approaches

The work in [99] proposed Neuron Activation Coverage (NAC), which considers the difference in behavior of neurons for ID and OOD samples. Specifically, NAC measures the frequency of activation of a neuron for training data in the form of coverage scores. The intuition is that OOD samples trigger neurons that are not usually triggered by ID data. The coverage score measures the alignment of the triggering of the neurons with training data and should be higher for ID than OOD input. Although inspired by [68, 148], the key innovation of this work is that NAC may also be used as a regularizer during training to improve OOD generalization.

Yi et al. [175] connected OOD generalization performance to shift in distribution. It also showed that the generalization bound tightens for DNN robust against perturbation to the input. The authors proposed to use adversarial training to make the DNN robust and showed that the performance of DNN improves when adversarially trained. This work

established a connection between adversarial robustness and OOD robustness and provided the insight that the problem of resilience of DNNs against intentional and unintentional perturbation is connected.

The work in [117] estimated the typicality of selected test-statistics 𝑇 𝑛 ( 𝑥 ) using KDEs on sets of statistics and creating a density estimate for individual samples by using product-of-experts (POE) structures. The POE provides the probability that an input is jointly typical for all test statistics. The ID samples are typical for all test statistics providing higher probability while OOD samples provide lower probability as they are not typical for all test statistics. The authors use different test statistics like posterior/prior cross-entropy, posterior entropy , posterior/prior KL divergence, posterior expected log-likelihood, and so on.

Finally, Huang et al. [69] proposed two density regularization methods, i.e., density consistency regularization and contrastive distribution regularization, to reliably calibrate and estimate sample density to identify OOD samples. Density consistency regularization enforces the agreement between analytical and empirical densities of categorical labels, while contrastive distribution regularization helps separate the densities between ID and OOD samples. The logarithm of the sample density function is used as the score function for OOD detection.

## 6 WHYAREINTENTIONAL AND UNINTENTIONAL PERTURBATION CONNECTED?

A fundamental similarity between intentional and unintentional perturbations is that their effect is similar. Indeed, both can be modeled as a perturbation 𝛿 such that for a DNN 𝑓 and input image x , 𝑓 ( x + 𝛿 ) ≠ 𝑓 ( x ) . This suggests that intentional perturbation can be modeled as OOD examples . This has been explored in [30], where the adversarial inputs are modeled as a special case of OOD samples. The presence of natural adversarial samples [63] strengthen the support for connection between intentionally and unintentionally perturbed samples.

Another similarity is that the DNN becomes overconfident in the case of both intentional and unintentional perturbation. This indicates that detection algorithms will share common characteristics. Prior work also reflects this intuition, as they attempt to present a common detection framework [83, 132]. On the other hand, the two communities for detection of intentional and unintentional perturbation can also benefit from exploring cross-domain approaches. For example, the detection of intentionally perturbed samples utilize the Bayesian Uncertainty estimation model [30, 42], yet this approach has not been explored for OOD detection. Moreover, intentional perturbation detection use class-conditional Gaussian distribution [83], mixture of Gaussian [193], kernel density estimation [42], and LID [104]. The community for unintentional perturbation detection has explored additional distributions - vMF [107] for example which is better at modeling embedding on a hypersphere. Another example of cross-domain opportunity is the modeling of the activation patterns for intentional and unintentional perturbation in terms of discrete states. The work in [102] models the activation from the penultimate layer using binary or quaternary codes, while [123] extracts binary neural activation pattern from multiple layers to characterize the label shift. These connections signify that the detection of intentional perturbation can benefit from exploring approaches adopted in unintentional perturbation detection. On the other hand, the assumption of access to perturbed samples is a common bottleneck that can be jointly explored. This leads to incomplete modeling of the perturbed samples.

Another connection lies in the intermediate representation in DNN. For example,[10] shows that adversarial inputs tend to activate all filters more or less uniformly. A similar phenomenon is observed in case of unintentional perturbation, especially in the case of label shift, where the OOD samples show unusually high activations for some filters. Such samples can be effectively detected by reshaping the filters activations through suppression [148], typical feature selection [198] and feature reshaping optimization [169]. Since mitigating overconfidence issue improves the detection of unintentional perturbation [164], we need to explore such approach also for intentional perturbation detection. Manuscript submitted to ACM

## 7 SUMMARY OF EXISTING RESEARCH CHALLENGES IN PERTURBATION DETECTION

Security and Privacy Issues: The work in [19] showed that a large difference between the training loss and the testing loss can lead to membership privacy risk, i.e., the adversary may increase their ability to conclude that an entity is in the input dataset [88]. Traditional training algorithms provide high confidence on both training and testing samples provided the DNN has low generalization gap, which leads to overconfidence. In order to alleviate the overconfidence issue, some approaches propose modified training loss to facilitate the detection of adversarial or OOD examples [22, 23, 96]. While these approaches reduce the test error, the change in generalization gap is neglected, which introduces possible privacy vulnerabilities. For example, [23] imposes specific pattern for unperturbed samples and any deviation from the pattern is detected as adversarial sample. The effect on privacy is not studied as the generalization gap with this training approach is not mentioned. If this approach increases divergence between the training loss distribution and test loss distribution, then the privacy of the DNN would be affected.

Research in Domains Different from Multi-class Classification: The OOD detection literature is predominantly focusing on DNNs for multi-class classification. Settings such as multi-label classification, object detection, segmentation tracking are largely understudied. We were able to find only the work [22, 65] focusing on unintentional perturbation for segmentation tasks and only the work [35-37, 165] focusing on unintentional perturbation object detection.

Detecting Perturbations at Scale: Most of the work in OOD detection reports detection performance on CIFAR10 and CIFAR100 benchmarks. On the other hand, the work in [57] has shown that OOD detection performance degrades significantly when considering real-world datasets. There is also a varying degree of performance as a function of the OOD dataset and architecture. For example, [97] reports the FPR for Imagenet-1k benchmark varying from 22.60% to 54% in the Swin transformer architecture, while in the BiT-S-R101 × 1 architecture it varies from 80.35% to 97.25%. This variability hints at a current lack of a dataset-agnostic OOD detection approach. Another key issue is determining which detector might work for a specific task during deployment. The average performance of the detectors would be unreliable in real-world as one would not know the distribution of OOD samples and might encounter worst-case distributions on which the detector provides high FPR. This opens up another research question: can we create a detector performance whose performance is OOD distribution agnostic?

Computation vs Performance Trade-off: The aspect of application and hardware specific constraints (e.g. computational cost, latency, energy etc.) is mostly ignored in the current literature of perturbation and semantic shift detection. There is no benchmark for latency, and computation cost metrics and these are rarely reported. [129] is the only work we could find that reports the computation time for detection of intentional perturbation but there is no analysis of the computational cost or energy expenditure. These are important metrics for implementation in edge devices. As a result, it is important that these metrics are reported as well.

Lack of Curated Dataset for OOD Detection: Most of the OOD detection literature focuses on semantic OOD and on the generalization capability of the DNN for non-semantic OOD. An interesting observation was recently reported in[174], where the authors introduced the Imagenet-OOD dataset to separatesemantic and covariate shifts. Moreover, the authors showed that existing methods are susceptible to detecting covariate shift rather than semantic shift. This suggests that existing methods are mostly designed for semantic OOD. This calls for curated datasets that (i) disentangle covariate shift and semantic shift and (ii) are sufficiently large-scale to capture real-world scenarios.

Lack of Explainability: The existing literature lacks an explainable way to separate semantic and non-semantic OOD. Specifically, current approaches toOOD detection cannot explain which features of the input lead the DNN to generate

the embeddings or the scores used to perform the detection. A more explainable OOD detection method can provide insights and drive further innovation.

Joint Resilience to Intentional and Unintentional Perturbation: In our view, guaranteeing resilience to intentional and unintentional perturbations cannot be seen as separate issues. Due to the disjoint communities of adversarial robustness and OOD robustness, the problem of joint resilience against both types of perturbations has been underexplored. The only work we have found is [199]. Although the work shows impressive performance on five different datasets, the number of images in different datasets and the variety is not sufficient to represent real-world conditions. The PACS [85], VLCS [41], and OfficeHome [38] datasets used contain 7, 5, and 65 classes with 9991, 10729, and 15588 example images respectively. To experiment on more realistic settings, a reasonable approach would be to consider datasets like DomainNet [128] and ImageNet-C [59].

## 8 CONCLUSIONS AND CALL TO ACTION

Over the last few years, a substantial amount of research has identified various issues regarding the resilience of DNNs, including susceptibility to adversarial attacks and distributional shifts. This has spurred a significant amount of research aimed at enhancing the robustness and reliability of DNNs. Efforts to bolster resilience have led to the development of novel techniques such as adversarial training, robust optimization, and domain adaptation. These approaches aim at making DNNs more robust against potential threats, while also improving their generalization across operational settings. The ongoing research into the underlying mechanisms of vulnerability can only advance through a collaborative effort from both intentional and unintentional perturbation detection communities. In this work, we have surveyed the state of the art in both intentional and unintentional perturbation detection, with the key aim of consolidating the approaches being employed. We have also listed a set of challenging research directions in this field. We hope that this survey will spur excitement in both communities and set new research directions in this topic.

## REFERENCES

- [1] Yong Hyun Ahn, Gyeong-Moon Park, and Seong Tae Kim. 2023. LINe: Out-of-Distribution Detection by Leveraging Important Neurons. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 19852-19862.
- [2] Ahmed Aldahdooh, Wassim Hamidouche, Sid Ahmed Fezza, and Olivier Déforges. 2022. Adversarial Example Detection for DNN Models: A Review and Experimental Comparison. Artificial Intelligence Review 55, 6 (2022).
- [3] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. 2020. Square Attack: A Query-efficient Black-box Adversarial Attack via Random search. In European conference on computer vision . Springer, 484-501.
- [4] Galina Angelova, Eleftherios Avramidis, and Sebastian Möller. 2022. Using Neural Machine Translation Methods for Sign Language Translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop , Samuel Louvan, Andrea Madotto, and Brielen Madureira (Eds.). Association for Computational Linguistics, Dublin, Ireland, 273-284. https://doi.org/10.18653/v1/2022.acl-srw.21
- [5] Daniel Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, Konrad Rieck, and CERT Siemens. 2014. Drebin: Effective and Explainable Detection of Android Malware in Your Pocket.. In Ndss , Vol. 14. 23-26.
- [6] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. 2018. Synthesizing robust adversarial examples. In International conference on machine learning . PMLR, 284-293.
- [7] Colin Atkinson and Ann FS Mitchell. 1981. Rao's distance measure. Sankhy¯ a: The Indian Journal of Statistics, Series A (1981), 345-365.
- [8] Haoyue Bai, Yifei Ming, Julian Katz-Samuels, and Yixuan Li. 2024. HYPO: Hyperspherical Out-Of-Distribution Generalization. In The Twelfth International Conference on Learning Representations . https://openreview.net/forum?id=VXak3CZZGC
- [9] Yang Bai, Yuyuan Zeng, Yong Jiang, Yisen Wang, Shu-Tao Xia, and Weiwei Guo. 2020. Improving Query Efficiency of Black-box Adversarial Attack. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16 . Springer, 101-116.
- [10] Yang Bai, Yuyuan Zeng, Yong Jiang, Shu-Tao Xia, Xingjun Ma, and Yisen Wang. 2021. Improving Adversarial Robustness via Channel-wise Activation Suppressing. In International Conference on Learning Representations . https://openreview.net/forum?id=zQTezqCCtNx
- [11] Brian Barrett. 2020. A Tiny Piece of Tape Tricked Teslas into Speeding up 50 mph. https://www.wired.com/story/tesla-speed-up-adversarialexample-mgm-breach-ransomware/
- [12] Anand Bhattad, Min Jin Chong, Kaizhao Liang, Bo Li, and D. A. Forsyth. 2020. Unrestricted Adversarial Examples via Semantic Manipulation. In International Conference on Learning Representations . https://openreview.net/forum?id=Sye\_OgHFwH

Manuscript submitted to ACM

- [13] Folkmar Bornemann and Tom März. 2007. Fast Image Inpainting Based on Coherence Transport. Journal of Mathematical Imaging and Vision 28 (2007), 259-278.
- [14] Amine Boulemtafes, Hamza Khemissa, Mohamed Saddek Derki, Abdelouahab Amira, and Nabil Djedjig. 2021. Deep Learning in Pervasive Health Monitoring, Design Goals, Applications, and Architectures: An Overview and A Brief Synthesis. Smart Health 22 (2021), 100221. https: //doi.org/10.1016/j.smhl.2021.100221
- [15] Andrew Brock, Soham De, Samuel L. Smith, and Karen Simonyan. 2021. High-Performance Large-Scale Image Recognition Without Normalization. arXiv:2102.06171
- [16] Jiajiong Cao, Yingming Li, and Zhongfei Zhang. 2018. Celeb-500k: A Large Training Dataset for Face Recognition. In 2018 25th IEEE International Conference on Image Processing (ICIP) . IEEE, 2406-2410.
- [17] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. 2018. Vggface2: A Dataset for Recognising Faces Across Pose and Age. In 2018 13th IEEE international conference on automatic face &amp; gesture recognition (FG 2018) . IEEE, 67-74.
- [18] Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp) . Ieee, 39-57.
- [19] Dingfan Chen, Ning Yu, and Mario Fritz. 2022. RelaxLoss: Defending Membership Inference Attacks without Losing Utility. In International Conference on Learning Representations (ICLR) .
- [20] Huanran Chen, Yichi Zhang, Yinpeng Dong, Xiao Yang, Hang Su, and Jun Zhu. 2024. Rethinking Model Ensemble in Transfer-based Adversarial Attacks. In The Twelfth International Conference on Learning Representations . https://openreview.net/forum?id=AcJrSoArlh
- [21] Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. 2017. Zoo: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks Without Training Substitute Models. In Proceedings of the 10th ACM workshop on artificial intelligence and security . 15-26.
- [22] Hyunjun Choi, Hawook Jeong, and Jin Young Choi. 2023. Balanced Energy Regularization Loss for Out-of-distribution Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 15691-15700.

[23]

Chien Cheng Chyou, Hung-Ting Su, and Winston H Hsu. 2023. Unsupervised Adversarial Detection without Extra Model: Training Loss Should

Change. In

The Second Workshop on New Frontiers in Adversarial Machine Learning

.

- [24] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. 2014. Describing Textures in the Wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) .
- [25] Gilad Cohen, Guillermo Sapiro, and Raja Giryes. 2020. Detecting Adversarial Samples Using Influence Functions and Nearest Neighbors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 14453-14462.
- [26] Francesco Croce and Matthias Hein. 2019. Sparse and Imperceivable Adversarial Attacks. In Proceedings of the IEEE/CVF international conference on computer vision . 4724-4732.

[27]

Francesco Croce and Matthias Hein. 2020. Reliable Evaluation of Adversarial Robustness with An Ensemble of Diverse Parameter-free Attacks. In

International conference on machine learning

.

PMLR, 2206-2216.

- [28] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale Hierarchical Image Database. In 2009 IEEE conference on computer vision and pattern recognition . Ieee, 248-255.

[29]

Li Deng. 2012.

The mnist database of handwritten digit images for machine learning research.

IEEE Signal Processing Magazine

29, 6 (2012),

141-142.

- [30] Zhijie Deng, Xiao Yang, Shizhen Xu, Hang Su, and Jun Zhu. 2021. Libre: A practical bayesian approach to adversarial detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 972-982.

[31]

Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. 2022. Extremely Simple Activation Shaping for Out-of-Distribution Detection.

In

The Eleventh International Conference on Learning Representations

.

- [32] Xiaoyi Dong, Dongdong Chen, Jianmin Bao, Chuan Qin, Lu Yuan, Weiming Zhang, Nenghai Yu, and Dong Chen. 2020. Greedyfool: Distortion-aware Sparse Adversarial Attack. Advances in Neural Information Processing Systems 33 (2020), 11226-11236.
- [33] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. 2018. Boosting Adversarial Attacks with Momentum. In Proceedings of IEEE CVPR . 9185-9193.
- [34] Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. 2019. Evading Defenses to Transferable Adversarial Examples by Translation-invariant Attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 4312-4321.

[35]

Xuefeng Du, Gabriel Gozum, Yifei Ming, and Yixuan Li. 2022. Siren: Shaping Representations for Detecting Out-of-Distribution Objects.

Advances in Neural Information Processing Systems

35 (2022), 20434-20449.

- [36] Xuefeng Du, Xin Wang, Gabriel Gozum, and Yixuan Li. 2022. Unknown-Aware Object Detection: Learning What You Don't Know from Videos in the Wild. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022).
- [37] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. 2022. VOS: Learning What You Don't Know by Virtual Outlier Synthesis. Proceedings of the International Conference on Learning Representations (2022).
- [38] Yann Dubois, Yangjun Ruan, and Chris J Maddison. 2021. Optimal Representations for Covariate Shifts. In NeurIPS 2021 workshop on distribution shifts: connecting methods and applications .
- [39] David Elliott and Eldon Soifer. 2022. AI Technologies, Privacy, and Security. Frontiers in Artificial Intelligence 5 (2022), 826737.
- [40] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. [n. d.]. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html.

Manuscript submitted to ACM

- [41] Chen Fang, Ye Xu, and Daniel N Rockmore. 2013. Unbiased Metric Learning: On the Utilization of Multiple Datasets and Web Images for Softening Bias. In Proceedings of the IEEE International Conference on Computer Vision . 1657-1664.

[42]

Reuben

Feinman,

Ryan

R.

Curtin,

Saurabh

Shintre, and

Andrew B.

Gardner.

2017.

Detecting

Adversarial

Samples from

Artifacts.

arXiv:1703.00410 [stat.ML]

- [43] Gil Fidel, Ron Bitton, and Asaf Shabtai. 2020. When Explainability Meets Adversarial Learning: Detecting Adversarial Examples Using Shap Signatures. In 2020 international joint conference on neural networks (IJCNN) . IEEE, 1-8.
- [44] Ruiyuan Gao, Chenchen Zhao, Lanqing Hong, and Qiang Xu. 2023. DiffGuard: Semantic Mismatch-Guided Out-of-Distribution Detection using Pre-trained Diffusion Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision .
- [45] Jacob R Gardner, Paul Upchurch, Matt J Kusner, Yixuan Li, Kilian Q Weinberger, Kavita Bala, and John E Hopcroft. 2015. Deep Manifold Traversal: Changing Labels with Convolutional Features. arXiv preprint arXiv:1511.06421 (2015).
- [46] Zhijin Ge, Wang Xiaosen, Hongying Liu, Fanhua Shang, and Yuanyuan Liu. 2024. Boosting Adversarial Transferability by Achieving Flat Local Maxima. Advances in Neural Information Processing Systems 36 (2024).
- [47] Eduardo Dadalto Camara Gomes, Florence Alberge, Pierre Duhamel, and Pablo Piantanida. 2022. Igeood: An Information Geometry Approach to Out-of-Distribution Detection. In International Conference on Learning Representations . https://openreview.net/forum?id=mfwdY3U\_9ea
- [48] Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. 2022. Note: Robust Continual Test-time Adaptation Against Temporal Correlation. Advances in Neural Information Processing Systems 35 (2022), 27253-27266.
- [49] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and Harnessing Adversarial Examples. arXiv e-prints (2014), arXiv-1412.
- [50] Sven Gowal, Jonathan Uesato, Chongli Qin, Po-Sen Huang, Timothy Mann, and Pushmeet Kohli. 2019. An Alternative Surrogate Loss for Pgd-based Adversarial Testing. arXiv preprint arXiv:1910.09338 (2019).
- [51] Gregory Griffin, Alex Holub, Pietro Perona, et al. 2007. Caltech-256 Object Category Dataset . Technical Report. Technical Report 7694, California Institute of Technology Pasadena.
- [52] Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick McDaniel. 2017. On the (Statistical) Detection of Adversarial Examples. arXiv:1702.06280 [cs.CR]

[53]

Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Weinberger. 2019. Simple Black-box Adversarial Attacks. In

International conference on machine learning

.

PMLR, 2484-2493.

Yiwen Guo, Qizhang Li, and Hao Chen. 2020. Backpropagating Linearly Improves Transferability of Adversarial Examples.

[54]

Advances in neural information processing systems

33 (2020), 85-95.

- [55] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 770-778.

[56]

Qunying He, Jingjing Liu, and Zhicheng Huang. 2023. WSRC: Weakly Supervised Faster RCNN Toward Accurate Traffic Object Detection.

IEEE

Access

11 (2023), 1445-1455.

https://doi.org/10.1109/ACCESS.2022.3231293

- [57] Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Xiaodong Song. 2022. Scaling Outof-Distribution Detection for Real-World Settings. In International Conference on Machine Learning . https://api.semanticscholar.org/CorpusID: 227407829
- [58] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. 2021. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision . 8340-8349.
- [59] Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. Proceedings of the International Conference on Learning Representations (2019).
- [60] Dan Hendrycks and Kevin Gimpel. 2016. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. In International Conference on Learning Representations .

[61]

Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. 2019. Deep Anomaly Detection with Outlier Exposure.

Proceedings of the International

Conference on Learning Representations

(2019).

- [62] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. 2019. Augmix: A Simple Data Processing Method to Improve Robustness and Uncertainty. arXiv preprint arXiv:1912.02781 (2019).
- [63] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. 2021. Natural Adversarial Examples. CVPR (2021).
- [64] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. 2018. The iNaturalist Species Classification and Detection Dataset. arXiv:1707.06642
- [65] Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. 2020. Generalized Odin: Detecting Out-of-Distribution Image Without Learning from Out-of-Distribution Data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 10951-10960.

[66]

Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. 2008. Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments. In

Workshop on faces in'Real-Life'Images: detection, alignment, and recognition

.

- [67] Hao Huang, Ziyan Chen, Huanran Chen, Yongtao Wang, and Kevin Zhang. 2023. T-sea: Transfer-based Self-ensemble Attack on Object Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 20514-20523.
- [68] Rui Huang, Andrew Geng, and Yixuan Li. 2021. On the Importance of Gradients for Detecting Distributional Shifts in the Wild. In Advances in Neural Information Processing Systems , M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran

Manuscript submitted to ACM

Associates, Inc., 677-689. https://proceedings.neurips.cc/paper\_files/paper/2021/file/063e26c670d07bb7c4d30e6fc69fe056-Paper.pdf

- [69] Wenjian Huang, Hao Wang, Jiahao Xia, Chengyan Wang, and Jianguo Zhang. 2022. Density-driven Regularization for Out-of-distribution Detection. In Advances in Neural Information Processing Systems , Vol. 35. Curran Associates, Inc., 887-900. https://proceedings.neurips.cc/paper\_files/paper/ 2022/file/05b69cc4c8ff6e24c5de1ecd27223d37-Paper-Conference.pdf
- [70] Yi Huang and Adams Wai-Kin Kong. 2022. Transferable Adversarial Attack based on Integrated Gradients. In International Conference on Learning Representations . https://openreview.net/forum?id=DesNW4-5ai9
- [71] Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. 2018. Black-box Adversarial Attacks with Limited Queries and Information. In International conference on machine learning . PMLR, 2137-2146.
- [72] Xiaojun Jia, Yong Zhang, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao. 2022. LAS-AT: Adversarial Training with Learnable Attack Strategy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 13398-13408.
- [73] Gaojie Jin, Xinping Yi, Dengyu Wu, Ronghui Mu, and Xiaowei Huang. 2023. Randomized Adversarial Training via Taylor Expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 16447-16457.
- [74] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment Anything. arXiv:2304.02643 (2023).
- [75] Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In International conference on machine learning . PMLR, 1885-1894.
- [76] Ivan Krasin, Tom Duerig, Neil Alldrin, Andreas Veit, Sami Abu-El-Haija, Serge Belongie, David Cai, Zheyun Feng, Vittorio Ferrari, Victor Gomes, Abhinav Gupta, Dhyanesh Narayanan, Chen Sun, Gal Chechik, and Kevin Murphy. 2016. OpenImages: A Public Dataset for Large-scale Multi-label and Multi-class Image Classification. Dataset available from https://github.com/openimages (2016).
- [77] Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. 2020. Being Bayesian, Even Just a Bit, Fixes overconfidence in ReLU Networks. In International conference on machine learning . PMLR, 5436-5446.
- [78] Alex Krizhevsky and Geoffrey Hinton. 2009. Learning Multiple Layers of Features from Tiny Images. (2009).
- [79] Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. 2018. Adversarial Examples in the Physical World. In Artificial intelligence safety and security . Chapman and Hall/CRC, 99-112.
- [80] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles. Advances in neural information processing systems 30 (2017).
- [81] Raz Lapid and Moshe Sipper. 2023. I See Dead People: Gray-box Adversarial Attack on Image-to-text Models. arXiv preprint arXiv:2306.07591 (2023).
- [82] Ya Le and Xuan Yang. 2015. Tiny Imagenet Visual Recognition Challenge. CS 231N 7, 7 (2015), 3.
- [83] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. Advances in neural information processing systems 31 (2018).

[84]

Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, Jilin Li, and Feiyue Huang. 2021.

Detecting Adversarial Patch Attacks through Global-Local

Consistency. In

Proceedings of the 1st International Workshop on Adversarial Learning for Multimedia

.

35-41.

- [85] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. 2017. Deeper, Broader and Artier Domain Generalization. In Proceedings of the IEEE international conference on computer vision . 5542-5550.
- [86] Jingyao Li, Pengguang Chen, Zexin He, Shaozuo Yu, Shu Liu, and Jiaya Jia. 2023. Rethinking Out-of-distribution (OOD) Detection: Masked Image Modeling is All You Need. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 11578-11589.
- [87] Jie Li, Rongrong Ji, Hong Liu, Jianzhuang Liu, Bineng Zhong, Cheng Deng, and Qi Tian. 2020. Projection &amp; Probability-driven Black-box Attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 362-371.

[88]

Ninghui Li, Wahbeh Qardaji, Dong Su, Yi Wu, and Weining Yang. 2013. Membership privacy: A unifying framework for privacy definitions. In

Proceedings of the 2013 ACM SIGSAC conference on Computer &amp; communications security

.

889-900.

- [89] Shasha Li, Ajaya Neupane, Sujoy Paul, Chengyu Song, Srikanth V Krishnamurthy, Amit K Roy Chowdhury, and Ananthram Swami. 2019. Stealthy Adversarial Perturbations Against Real-Time Video Classification Systems. In Proceedings 2019 Network and Distributed System Security Symposium . Internet Society.
- [90] Xu Li, Na Li, Jinghua Zhong, Xixin Wu, Xunying Liu, Dan Su, Dong Yu, and Helen Meng. 2020. Investigating Robustness of Adversarial Samples Detection for Automatic Speaker Verification. arXiv preprint arXiv:2006.06186 (2020).
- [91] Yingwei Li, Song Bai, Yuyin Zhou, Cihang Xie, Zhishuai Zhang, and Alan Yuille. 2020. Learning Transferable Adversarial Examples via Ghost Networks. In Proceedings of the AAAI conference on artificial intelligence , Vol. 34. 11458-11465.
- [92] Yandong Li, Lijun Li, Liqiang Wang, Tong Zhang, and Boqing Gong. 2019. Nattack: Learning the Distributions of Adversarial Examples for an Improved Black-box Attack on Deep Neural Networks. In International Conference on Machine Learning . PMLR, 3866-3876.
- [93] Shiyu Liang, Yixuan Li, and R. Srikant. 2018. Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. In International Conference on Learning Representations . https://openreview.net/forum?id=H1VGkIxRZ
- [94] Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E. Hopcroft. 2020. Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks. In International Conference on Learning Representations . https://openreview.net/forum?id=SJlHwkBYDH
- [95] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Proceedings of European Conference on Computer Vision (ECCV) . Springer, 740-755.

Manuscript submitted to ACM

- [96] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. 2020. Energy-based Out-of-Distribution Detection. Advances in neural information processing systems 33 (2020), 21464-21475.
- [97] Xixi Liu, Yaroslava Lochman, and Christopher Zach. 2023. Gen: Pushing the Limits of Softmax-based Out-of-Distribution Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 23946-23955.
- [98] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. 2017. Delving into Transferable Adversarial Examples and Black-box Attacks. In International Conference on Learning Representations . https://openreview.net/forum?id=Sys6GJqxl

[99]

Yibing Liu, XING Chris, Haoliang Li, Lei Ma, and Shiqi Wang. 2023. Neuron Activation Coverage: Rethinking Out-of-distribution Detection and

Generalization. In

The Twelfth International Conference on Learning Representations

.

- [100] Yuyang Long, Qilong Zhang, Boheng Zeng, Lianli Gao, Xianglong Liu, Jian Zhang, and Jingkuan Song. 2022. Frequency Domain Model Aaugmentation for Adversarial Attack. In European conference on computer vision . Springer, 549-566.

[101]

Haodong Lu, Dong Gong, Shuo Wang, Jason Xue, Lina Yao, and Kristen Moore. 2023. Learning with Mixture of Prototypes for Out-of-Distribution

Detection. In

The Twelfth International Conference on Learning Representations

.

- [102] Jiajun Lu, Theerasit Issaranon, and David Forsyth. 2017. Safetynet: Detecting and Rejecting Adversarial Examples Robustly. In Proceedings of the IEEE international conference on computer vision . 446-454.
- [103] Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model predictions. Advances in neural information processing systems 30 (2017).
- [104] Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey. 2018. Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality. arXiv preprint arXiv:1801.02613 (2018).
- [105] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards Deep Learning Models Resistant to Adversarial Attacks. In International Conference on Learning Representations . https://openreview.net/forum?id=rJzIBfZAb
- [106] Debapriya Maji, Soyeb Nagori, Manu Mathew, and Deepak Poddar. 2022. YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss. arXiv:2204.06806
- [107] Kanti V Mardia and Peter E Jupp. 2009. Directional Statistics . John Wiley &amp; Sons.
- [108] Rohan Reddy Mekala, Sai Yerramreddy, and Adam Porter. 2021. Metamorphic Adversarial Detection Pipeline for Face Recognition Systems. In The AAAI-22 Workshop on Adversarial Machine Learning and Beyond .
- [109] Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. 2017. On detecting adversarial perturbations. arXiv preprint arXiv:1702.04267 (2017).
- [110] Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. 2022. Delving into Out-of-Distribution Detection with Vision-Language Representations. In Advances in Neural Information Processing Systems .

[111]

Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. 2023. How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?. In

The

Eleventh International Conference on Learning Representations

.

https://openreview.net/forum?id=aEFaE0W5pAd

- [112] Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. 2019. Sparsefool: A Few Pixels Make a Big Difference. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 9087-9096.

[113]

Hadi Mohaghegh Dolatabadi, Sarah Erfani, and Christopher Leckie. 2020. Advflow: Inconspicuous Black-box Adversarial Attacks Using Normalizing

Flows.

Advances in Neural Information Processing Systems

33 (2020), 15871-15884.

- [114] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. 2017. Universal Adversarial Perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition . 1765-1773.
- [115] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. Deepfool: A Simple and Accurate Method to Fool Deep Neural Networks. In Proceedings of the IEEE conference on computer vision and pattern recognition . 2574-2582.
- [116] Konda Reddy Mopuri, Utkarsh Ojha, Utsav Garg, and R Venkatesh Babu. 2018. Nag: Network for Adversary Generation. In Proceedings of the IEEE conference on computer vision and pattern recognition . 742-751.
- [117] Warren R. Morningstar, Cusuh Ham, Andrew Gallagher, Balaji Lakshminarayanan, Alexander A. Alemi, and Joshua V. Dillon. 2020. Density of States Estimation for Out-of-Distribution Detection. In International Conference on Artificial Intelligence and Statistics . https://api.semanticscholar. org/CorpusID:219708243
- [118] Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. 2017. Agedb: The First Manually Collected, In-the-Wild Age Database. In proceedings of the IEEE conference on computer vision and pattern recognition workshops . 51-59.
- [119] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. 2017. Voxceleb: A Large-scale Speaker Identification Dataset. arXiv preprint arXiv:1706.08612 (2017).
- [120] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. 2011. Reading Digits in Natural Images with Unsupervised Feature Learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2 (2011).
- [121] Anh Nguyen, Jason Yosinski, and Jeff Clune. 2015. Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images. In Proceedings of the IEEE conference on computer vision and pattern recognition . 427-436.
- [122] Maria-Elena Nilsback and Andrew Zisserman. 2008. Automated Flower Classification over a Large Number of Classes. In Indian Conference on Computer Vision, Graphics and Image Processing .
- [123] Bartłomiej Olber, Krystian Radlak, Adam Popowicz, Michal Szczepankiewicz, and Krystian Chachuła. 2023. Detection of Out-of-Distribution Samples using Binary Neuron Activation Patterns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 3378-3387.

Manuscript submitted to ACM

- [124] OpenAI. 2023. GPT-4 Technical Report. ArXiv abs/2303.08774 (2023). https://arxiv.org/abs/2303.08774
- [125] Dhruv Bhandarkar Pai, Andres Carranza, Rylan Schaeffer, Arnuv Tandon, and Sanmi Koyejo. 2023. FACADE: A Framework for Adversarial Circuit Anomaly Detection and Evaluation. In The Second Workshop on New Frontiers in Adversarial Machine Learning .
- [126] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. 2016. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277 (2016).

[127]

Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. 2017.

Practical Black-Box Attacks against Machine Learning. In

Proceedings of the 2017 ACM on Asia conference on computer and communications security

.

506-519.

- [128] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. 2019. Moment Matching for Multi-source Domain Adaptation. In Proceedings of the IEEE/CVF international conference on computer vision . 1406-1415.
- [129] Marine Picot, Federica Granese, Guillaume Staerman, Marco Romanelli, Francisco Messina, Pablo Piantanida, and Pierre Colombo. 2023. A Halfspace-Mass Depth-Based Method for Adversarial Attack Detection. Transactions on Machine Learning Research (2023). https://openreview.net/ forum?id=YtU0nDb5e8
- [130] Samira Pouyanfar, Saad Sadiq, Yilin Yan, Haiman Tian, Yudong Tao, Maria Presa Reyes, Mei-Ling Shyu, Shu-Ching Chen, and S. S. Iyengar. 2018. A Survey on Deep Learning: Algorithms, Techniques, and Applications. ACM Comput. Surv. 51, 5, Article 92 (sep 2018), 36 pages. https://doi.org/10.1145/3234150
- [131] Joaquin Quiñonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. 2008. Dataset Shift in Machine Learning . Mit Press.

[132]

Jayaram Raghuram, Varun Chandrasekaran, Somesh Jha, and Suman Banerjee. 2021. A General Framework for Detecting Anomalous Inputs to

DNN Classifiers. In

International Conference on Machine Learning

.

PMLR, 8764-8775.

- [133] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. 2021. ImageNet-21K Pretraining for the Masses. arXiv:2104.10972
- [134] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 10684-10695.
- [135] Kevin Roth, Yannic Kilcher, and Thomas Hofmann. 2019. The odds are odd: A statistical test for detecting adversarial examples. In International Conference on Machine Learning . PMLR, 5498-5507.
- [136] M Salehi, H Mirzaei, D Hendrycks, Y Li, MH Rohban, M Sabokrou, et al. 2022. A Unified Survey on Anomaly, Novelty, Open-Set, and Out of-Distribution Detection: Solutions and Future Challenges. Transactions on Machine Learning Research 234 (2022).
- [137] Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M. Patel, Rama Chellappa, and David W. Jacobs. 2016. Frontal to profile face verification in the wild. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) . 1-9. https://doi.org/10.1109/WACV.2016.7477558
- [138] Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñez, and Jordi Luque. 2019. Input Complexity and Out-of-distribution Detection with Likelihood-based Generative Models. In International Conference on Learning Representations .
- [139] Ali Shahin Shamsabadi, Ricardo Sanchez-Matilla, and Andrea Cavallaro. 2020. Colorfool: Semantic Adversarial Colorization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 1151-1160.
- [140] LS SHAPLEY. 1997. A VALUE FOR n-PERSON GAMES 1 . Classics in Game Theory (1997), 69.
- [141] Akihiko Shimomura, Sho Shiino, Junpei Kawauchi, Satoko Takizawa, Hiromi Sakamoto, Juntaro Matsuzaki, Makiko Ono, Fumitaka Takeshita, Shumpei Niida, Chikako Shimizu, et al. 2016. Novel Combination of Serum microRNA for Detecting Breast Cancer in the Early Stage. Cancer science 107, 3 (2016), 326-334.
- [142] Ilia Shumailov, Yiren Zhao, Robert Mullins, and Ross Anderson. 2020. Towards certifiable adversarial sample detection. In Proceedings of the 13th ACM Workshop on Artificial Intelligence and Security . 13-24.

[143]

Chunfeng Song, Yongzhen Huang, Weining Wang, and Liang Wang. 2022. CASIA-E: a large comprehensive dataset for gait recognition.

IEEE

Transactions on Pattern Analysis and Machine Intelligence

45, 3 (2022), 2801-2815.

- [144] Gaurang Sriramanan, Sravanti Addepalli, Arya Baburaj, et al. 2020. Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses. Advances in Neural Information Processing Systems 33 (2020), 20297-20308.

[145]

David Stutz, Matthias Hein, and Bernt Schiele. 2019. Disentangling Adversarial Robustness and Generalization. In

Proceedings of the IEEE/CVF

Conference on Computer Vision and Pattern Recognition (CVPR)

.

- [146] Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. 2019. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation 23, 5 (2019), 828-841.
- [147] Jiaze Sun, Siyuan Long, Xianyan Ma, and Yanmei Tang. 2023. A Model Robustness Optimization Method Based on Adversarial Sample Detection. In Proceedings of the 2022 5th International Conference on Artificial Intelligence and Pattern Recognition (Xiamen, China) (AIPR '22) . Association for Computing Machinery, New York, NY, USA, 304-310. https://doi.org/10.1145/3573942.3574026
- [148] Yiyou Sun, Chuan Guo, and Yixuan Li. 2021. React: Out-of-Distribution Detection with Rectified Activations. Advances in Neural Information Processing Systems 34 (2021), 144-157.
- [149] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. 2022. Out-of-Distribution Detection with Deep Nearest Neighbors. In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162) . PMLR, 20827-20840. https://proceedings.mlr. press/v162/sun22d.html
- [150] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing Properties of Neural Networks. arXiv:1312.6199

- [151] Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey J Gordon. 2020. Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift. Advances in Neural Information Processing Systems 33 (2020), 19276-19289.
- [152] Guanhong Tao, Shiqing Ma, Yingqi Liu, and Xiangyu Zhang. 2018. Attacks Meet Interpretability: Attribute-Steered Detection of Adversarial Samples. Advances in Neural Information Processing Systems 31 (2018).
- [153] Bilel Tarchoun, Anouar Ben Khalifa, Mohamed Ali Mahjoub, Nael Abu-Ghazaleh, and Ihsen Alouani. 2023. Jedi: Entropy-based Localization and Removal of Adversarial Patches. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 4087-4095.
- [154] Simen Thys, Wiebe Van Ranst, and Toon Goedemé. 2019. Fooling Automated Surveillance Cameras: Adversarial Patches to Attack Person Detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops . 0-0.
- [155] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. 2018. Ensemble Adversarial Training: Attacks and Defenses. In International Conference on Learning Representations . https://openreview.net/forum?id=rkZvSe-RZ
- [156] Hoang Tran, Dan Lu, and Guannan Zhang. 2022. Exploiting the Local Parabolic Landscapes of Adversarial Losses to Accelerate Black-box Adversarial Attack. In European conference on computer vision . Springer, 317-334.

[157]

Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. 2022.

Open-Set Recognition: A Good Closed-Set Classifier is All You Need. In

International Conference on Learning Representations

.

https://openreview.net/forum?id=5hLP5JY9S2d

- [158] Hanrui Wang, Shuo Wang, Zhe Jin, Yandan Wang, Cunjian Chen, and Massimo Tistarelli. 2021. Similarity-based Gray-box Adversarial Attack Against Deep Face Recognition. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (Jodhpur, India). IEEE Press, 1-8. https://doi.org/10.1109/FG52635.2021.9667076
- [159] Jingyi Wang, Guoliang Dong, Jun Sun, Xinyu Wang, and Peixin Zhang. 2019. Adversarial Sample Detection for Deep Neural Network Through Model Mutation Testing. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 1245-1256.
- [160] Qizhou Wang, Junjie Ye, Feng Liu, Quanyu Dai, Marcus Kalander, Tongliang Liu, Jianye HAO, and Bo Han. 2023. Out-of-distribution Detection with Implicit Outlier Transformation. In The Eleventh International Conference on Learning Representations . https://openreview.net/forum?id= hdghx6wbGuD

[161]

Xiaosen Wang and Kun He. 2021. Enhancing the Transferability of Adversarial Attacks through Variance Tuning. In

Proceedings of the IEEE/CVF

conference on computer vision and pattern recognition

.

1924-1933.

- [162] Xiaosen Wang, Xuanran He, Jingdong Wang, and Kun He. 2021. Admix: Enhancing the Transferability of Adversarial Attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 16158-16167.

[163]

Xiaosen Wang, Zeliang Zhang, and Jianping Zhang. 2023. Structure Invariant Transformation for Better Adversarial Transferability. In

Proceedings of the IEEE/CVF International Conference on Computer Vision

.

4607-4619.

- [164] Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. 2022. Mitigating Neural Network Overconfidence with Logit Normalization. In International Conference on Machine Learning . PMLR, 23631-23644.

[165]

Samuel Wilson, Tobias Fischer, Feras Dayoub, Dimity Miller, and Niko Sünderhauf. 2023. SAFE: Sensitivity-Aware Features for Out-of-Distribution

Object Detection. In

Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

.

23565-23576.

- [166] Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. 2018. Spatially Transformed Adversarial Examples. In International Conference on Learning Representations . https://openreview.net/forum?id=HyydRMZC-
- [167] Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv preprint arXiv:1708.07747 (2017).
- [168] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. 2019. Improving Transferability of Adversarial Examples with Input Diversity. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2730-2739.
- [169] Mingyu Xu, Zheng Lian, Bin Liu, and Jianhua Tao. 2023. VRA: Variational Rectified Activation for Out-of-Distribution Detection. Advances in Neural Information Processing Systems 36 (2023), 28941-28959.
- [170] Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R. Kulkarni, and Jianxiong Xiao. 2015. TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking. arXiv:1504.06755
- [171] Weilin Xu, David Evans, and Yanjun Qi. 2018. Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks. In Proceedings 2018 Network and Distributed System Security Symposium . Internet Society.
- [172] Keiichiro Yamamura, Haruki Sato, Nariaki Tateiwa, Nozomi Hata, Toru Mitsutake, Issa Oe, Hiroki Ishikura, and Katsuki Fujisawa. 2022. Diversified Adversarial Attacks Based on Conjugate Gradient Method. In International Conference on Machine Learning . PMLR, 24872-24894.
- [173] Puyudi Yang, Jianbo Chen, Cho-Jui Hsieh, Jane-Ling Wang, and Michael Jordan. 2020. Ml-loo: Detecting Adversarial Examples with Feature Attribution. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 34. 6639-6647.
- [174] William Yang, Byron Zhang, and Olga Russakovsky. 2024. ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms. In The Twelfth International Conference on Learning Representations . https://openreview.net/forum?id=VTYg5ykEGS
- [175] Mingyang Yi, Lu Hou, Jiacheng Sun, Lifeng Shang, Xin Jiang, Qun Liu, and Zhiming Ma. 2021. Improved OOD Generalization via Adversarial Training and Pretraing. In International Conference on Machine Learning . PMLR, 11987-11997.
- [176] Jaemin Yoo, Minyong Cho, Taebum Kim, and U Kang. 2019. Knowledge Extraction with No Observable Data. Advances in Neural Information Processing Systems 32 (2019).
- [177] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. 2020. Bdd100k: A Diverse Driving Dataset for Heterogeneous Multitask Learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . Manuscript submitted to ACM

2636-2645.

- [178] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. 2015. LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop. arXiv preprint arXiv:1506.03365 (2015).
- [179] Yunrui Yu and Cheng-Zhong Xu. 2023. Efficient Loss Function by Minimizing the Detrimental Effect of Floating-point Errors on Gradient-based Attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 4056-4066.
- [180] Longhui Yuan, Binhui Xie, and Shuang Li. 2023. Robust Test-time Adaptation in Dynamic Scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 15922-15932.
- [181] Alireza Zaeemzadeh, Niccolo Bisagno, Zeno Sambugaro, Nicola Conci, Nazanin Rahnavard, and Mubarak Shah. 2021. Out-of-Distribution Detection using Union of 1-Dimensional Subspaces. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition . 9452-9461.
- [182] Chaoning Zhang, Philipp Benz, Tooba Imtiaz, and In-So Kweon. 2020. CD-UAP: Class Discriminative Universal Adversarial Perturbation. In Proceedings of the AAAI conference on artificial intelligence , Vol. 34. 6754-6761.
- [183] Chaoning Zhang, Philipp Benz, Adil Karjauv, and In So Kweon. 2021. Data-free Universal Adversarial Perturbation and black-box Attack. In Proceedings of the IEEE/CVF international conference on computer vision . 7868-7877.
- [184] Chiliang Zhang, Zhimou Yang, and Zuochang Ye. 2018. Detecting Adversarial Perturbations with Saliency. In Proceedings of the 6th International Conference on Information Technology: IoT and Smart City (Hong Kong, Hong Kong) (ICIT '18) . Association for Computing Machinery, New York, NY, USA, 25-30. https://doi.org/10.1145/3301551.3301588
- [185] Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. 2019. You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle. Advances in neural information processing systems 32 (2019).
- [186] Jianping Zhang, Jen-tse Huang, Wenxuan Wang, Yichen Li, Weibin Wu, Xiaosen Wang, Yuxin Su, and Michael R Lyu. 2023. Improving the Transferability of Adversarial Samples by Path-augmented Method. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 8173-8182.
- [187] Shigeng Zhang, Yudong Li, Shuxin Chen, Xuan Li, and Jian Zhang. 2023. A Fast Adversarial Sample Detection Approach for Industrial Internet-ofThings Applications. In 2023 IEEE/ACM 31st International Symposium on Quality of Service (IWQoS) . IEEE, 01-10.
- [188] Shuhai Zhang, Feng Liu, Jiahao Yang, Yifan Yang, Changsheng Li, Bo Han, and Mingkui Tan. 2023. Detecting Adversarial Data by Probing Multiple Perturbations Using Expected Perturbation Score. In International Conference on Machine Learning (ICML) .

[189]

Zihan Zhang and Xiang Xiang. 2023.

Decoupling MaxLogit for Out-of-Distribution Detection. In

Proceedings of the IEEE/CVF Conference on

Computer Vision and Pattern Recognition

.

3388-3397.

- [190] Qinyu Zhao, Ming Xu, Kartik Gupta, Akshay Asthana, Liang Zheng, and Stephen Gould. 2024. Towards Optimal Feature-Shaping Methods for Out-of-Distribution Detection. In The Twelfth International Conference on Learning Representations . https://openreview.net/forum?id=dm8e7gsH0d
- [191] Tianyue Zheng and Weihong Deng. 2018. Cross-pose LFW: A Database for Studying Cross-pose Face Recognition in Unconstrained Environments. Beijing University of Posts and Telecommunications, Tech. Rep 5, 7 (2018).
- [192] Tianyue Zheng, Weihong Deng, and Jiani Hu. 2017. Cross-age lfw: A database for studying cross-age face recognition in unconstrained environments. arXiv preprint arXiv:1708.08197 (2017).
- [193] Zhihao Zheng and Pengyu Hong. 2018. Robust Detection of Adversarial Attacks by Modeling the Intrinsic Properties of Deep Neural Networks. In Advances in Neural Information Processing Systems , Vol. 31. Curran Associates, Inc. https://proceedings.neurips.cc/paper\_files/paper/2018/file/ e7a425c6ece20cbc9056f98699b53c6f-Paper.pdf
- [194] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Places: A 10 million Image Database for Scene Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2017).
- [195] Mingyi Zhou, Jing Wu, Yipeng Liu, Shuaicheng Liu, and Ce Zhu. 2020. Dast: Data-free substitute training for adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 234-243.
- [196] Hegui Zhu, Yuchen Ren, Xiaoyan Sui, Lianping Yang, and Wuming Jiang. 2023. Boosting Adversarial Transferability via Gradient Relevance Attack. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 4741-4750.
- [197] Jianing Zhu, Geng Yu, Jiangchao Yao, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Bo Han. 2023. Diversified Outlier Exposure for Out-ofDistribution Detection via Informative Extrapolation. In Thirty-seventh Conference on Neural Information Processing Systems . https://openreview. net/forum?id=RuxBLfiEqI
- [198] Yao Zhu, YueFeng Chen, Chuanlong Xie, Xiaodan Li, Rong Zhang, Hui Xue, Xiang Tian, Yaowu Chen, et al. 2022. Boosting Out-of-Distribution Detection with Typical Features. Advances in Neural Information Processing Systems 35 (2022), 20758-20769.
- [199] Xin Zou and Weiwei Liu. 2023. On the Adversarial Robustness of Out-of-distribution Generalization Models. In Thirty-seventh Conference on Neural Information Processing Systems . https://openreview.net/forum?id=IiwTFcGGTq
Sazzad Sayyed Milin Zhang Shahriar Rifat Ananthram Swami Michael De Lucia Francesco Restuccia
Resilience and Security of Deep Neural Networks Against Intentional and Unintentional Perturbations: Survey and Research Challenges
In order to deploy deep neural networks (DNNs) in high-stakes scenarios, it
is imperative that DNNs provide inference robust to external perturbations -
both intentional and unintentional. Although the resilience of DNNs to
intentional and unintentional perturbations has been widely investigated, a
unified vision of these inherently intertwined problem domains is still
missing. In this work, we fill this gap by providing a survey of the state of
the art and highlighting the similarities of the proposed approaches.We also
analyze the research challenges that need to be addressed to deploy resilient
and secure DNNs. As there has not been any such survey connecting the
resilience of DNNs to intentional and unintentional perturbations, we believe
this work can help advance the frontier in both domains by enabling the
exchange of ideas between the two communities.