summaries: This paper aims to improve adversarial robustness of the classifiers in a different perspective than the existing works. Usually, the networks are trained using adversarial examples to improve robustness (adversarial training). This work extend this line of thought and make an input robust to adversarial attacks. Instead of updating the network, they make updates to the input to gain robustness. In other words, this work explore the existence of safe spots near the input samples that are robust against adversarial attacks. Results on CIFAR-10 and ImageNet reveals that there exists such safe spots which are resistant to adversarial perturbations and improve adversarial robustness when combined with adversarial training (the authors term it as safe-spot aware adversarial training). Based on this approach, the authors also propose out-of-distribution detection method that outperforms previous works. This paper proposes a new adversarial framework where the defender could preemptively modify classifier inputs to find safe spots that are robust to adversarial attacks. They then introduce a novel bi-level optimization algorithm that can find safe spots on over 90% of the correctly classified images for adversarially trained classifiers on CIFAR-10 and ImageNet datasets and show that they can be used to improve both the empirical and certified robustness on smoothed classifiers. Besides, they propose a new training scheme based on their conjecture about safe spots for out-of-distribution detection which achieves state-of-the-art results on near-distribution outliers.  The authors argue that there are some safe "spots" in the data space that are less prone to adversarial attacks. The authors propose a technique to identify such "safe spots". They then leverage them for robust training and observe higher robust accuracy than baseline. Finally, they leverage this observation to identify out of distribution data. 
abstractText: Recent advances on adversarial defense mainly focus on improving the classifier’s robustness against adversarially perturbed inputs. In this paper, we turn our attention from classifiers to inputs and explore if there exist safe spots in the vicinity of natural images that are robust to adversarial attacks. In this regard, we introduce a novel bi-level optimization algorithm that can find safe spots on over 90% of the correctly classified images for adversarially trained classifiers on CIFAR-10 and ImageNet datasets. Our experiments also show that they can be used to improve both the empirical and certified robustness on smoothed classifiers. Furthermore, by exploiting a novel safe spot inducing model training scheme and our safe spot generation method, we propose a new out-of-distribution detection algorithm which achieves the state of the art results on near-distribution outliers.
references.authors: Anish Athalye Nicholas Carlini David Wagner
references.title: Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples
references.authors: Yair Carmon Aditi Raghunathan Ludwig Schmidt Percy Liang John Duchi
references.title: Unlabeled data improves adversarial robustness
references.venue: In NeurIPS,
references.authors: Jeremy M Cohen Elan Rosenfeld J Zico Kolter
references.title: Certified adversarial robustness via randomized smoothing
references.venue: In ICML,
references.authors: Francesco Croce Matthias Hein
references.title: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
references.venue: In ICML,
references.authors: Chelsea Finn Pieter Abbeel Sergey Levine
references.title: Model-agnostic meta-learning for fast adaptation of deep networks
references.venue: In ICML,
references.authors: Ian J Goodfellow Jonathon Shlens Christian Szegedy
references.title: Explaining and harnessing adversarial examples
references.venue: In ICLR,
references.authors: Chuan Guo Mayank Rana Moustapha Cisse Laurens van der Maaten
references.title: Countering adversarial images using input transformations
references.venue: In ICLR,
references.authors: Dan Hendrycks Kevin Gimpel
references.title: A baseline for detecting misclassified and out-of-distribution examples in neural networks
references.venue: In ICLR,
references.authors: Dan Hendrycks Kimin Lee Mantas Mazeika
references.title: Using pre-training can improve model robustness and uncertainty
references.venue: In ICML,
references.authors: Dan Hendrycks Mantas Mazeika Thomas Dietterich
references.title: Deep anomaly detection with outlier exposure
references.venue: In ICLR,
references.authors: Dan Hendrycks Mantas Mazeika Saurav Kadavath Dawn Song
references.title: Using self-supervised learning can improve model robustness and uncertainty
references.venue: In NeurIPS,
references.authors: Alex Krizhevsky Geoffrey Hinton
references.title: Learning multiple layers of features from tiny images
references.venue: Technical report, Citeseer,
references.authors: Mathias Lecuyer Vaggelis Atlidakis Roxana Geambasu Daniel Hsu Suman Jana
references.title: Certified robustness to adversarial examples with differential privacy
references.venue: In IEEE Symposium on Security and Privacy (SP),
references.authors: Kimin Lee Kibok Lee Honglak Lee Jinwoo Shin
references.title: A simple unified framework for detecting out-of-distribution samples and adversarial attacks
references.venue: NeurIPS,
references.authors: Xuanqing Liu Minhao Cheng Huan Zhang Cho-Jui Hsieh
references.title: Towards robust neural networks via random self-ensemble
references.venue: In ECCV,
references.authors: Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras Adrian Vladu
references.title: Towards deep learning models resistant to adversarial attacks
references.venue: In ICLR,
references.authors: Yuval Netzer Tao Wang Adam Coates Alessandro Bissacco Bo Wu Andrew Y Ng
references.title: Reading digits in natural images with unsupervised feature learning
references.venue: NeurIPS-W,
references.authors: Anh Nguyen Jason Yosinski Jeff Clune
references.title: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
references.venue: In CVPR,
references.authors: Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury Gregory Chanan Trevor Killeen Zeming Lin Natalia Gimelshein Luca Antiga
references.title: Pytorch: An imperative style, high-performance deep learning library
references.venue: NeurIPS,
references.authors: Chongli Qin James Martens Sven Gowal Dilip Krishnan Krishnamurthy Dvijotham Alhussein Fawzi Soham De Robert Stanforth Pushmeet Kohli
references.title: Adversarial robustness through local linearization
references.venue: NeurIPS,
references.authors: Leslie Rice Eric Wong J. Zico Kolter
references.title: Overfitting in adversarially robust deep learning
references.venue: In ICML,
references.authors: Olga Russakovsky Jia Deng Hao Su Jonathan Krause Sanjeev Satheesh Sean Ma Zhiheng Huang Andrej Karpathy Aditya Khosla Michael Bernstein Alexander C. Berg Li Fei-Fei
references.title: ImageNet Large Scale Visual Recognition Challenge
references.venue: In IJCV,
references.authors: Hadi Salman Jerry Li Ilya Razenshteyn Pengchuan Zhang Huan Zhang Sebastien Bubeck Greg Yang
references.title: Provably robust deep learning via adversarially trained smoothed classifiers
references.venue: NeurIPS,
references.authors: Chandramouli S. Sastry Sageev Oore
references.title: Detecting out-of-distribution examples with gram matrices
references.venue: In ICML,
references.authors: Yang Song Taesup Kim Sebastian Nowozin Stefano Ermon Nate Kushman
references.title: Pixeldefend: Leveraging generative models to understand and defend against adversarial examples
references.venue: In ICLR,
references.authors: Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna Dumitru Erhan Ian Goodfellow Rob Fergus
references.title: Intriguing properties of neural networks
references.venue: In ICLR,
references.authors: Florian Tramer Nicholas Carlini Wieland Brendel Aleksander Madry
references.title: On adaptive attacks to adversarial example defenses
references.venue: In arXiv:2002.08347,
references.authors: Yisen Wang Difan Zou Jinfeng Yi James Bailey Xingjun Ma Quanquan Gu
references.title: Improving adversarial robustness requires revisiting misclassified examples
references.venue: In ICLR,
references.authors: Eric Wong Leslie Rice J. Zico Kolter
references.title: Fast is better than free: Revisiting adversarial training
references.venue: In ICLR,
references.authors: Fisher Yu Ari Seff Yinda Zhang Shuran Song Thomas Funkhouser Jianxiong Xiao
references.title: Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop
references.authors: Sergey Zagoruyko Nikos Komodakis
references.title: Wide residual networks
references.venue: In arXiv:1605.07146,
references.authors: Haichao Zhang Jianyu Wang
references.title: Defense against adversarial attacks using feature scattering-based adversarial training
references.venue: In NeurIPS,
references.authors: Jingfeng Zhang Xilie Xu Bo Han Gang Niu Lizhen Cui Masashi Sugiyama Mohan Kankanhalli
references.title: Attacks which do not kill training make adversarial learning stronger
sections.heading: 1 INTRODUCTION
sections.text: Deep neural networks have achieved significant performance on various artificial intelligence tasks such as image classification, speech recognition, and reinforcement learning. Despite the results, Szegedy et al. (2013) demonstrated that deep neural networks are vulnerable to adversarial examples, minute input perturbations designed to mislead networks to yield incorrect predictions. There have been a large number of studies to improve the robustness of networks against adversarial perturbations (Song et al., 2017; Guo et al., 2018), while many of the proposed methods have been shown to fail against stronger adversaries (Athalye et al., 2018; Tramer et al., 2020). Adversarial training (Madry et al., 2017) and randomized smoothing (Cohen et al., 2019) are some of the few methods that survived the harsh verifications, each focusing on empirical and certified robustness, respectively.
  To summarize, the study of adversarial examples has been an arms race between adversaries, who manipulate inputs to raise network malfunction, and defenders, who aim to preserve the network performance against the corrupted inputs. In this paper, we approach the adversarial robustness problem from a different perspective. Instead of defending networks from already perturbed examples, we assume the situation where the defenders can also influence inputs slightly for their interest before the adversaries’ incursion. The defenders’ goal for this manipulation will be to improve robustness by searching for spots in the input space that are resistant to adversarial attacks, given a pre-trained classifier. We explore methods for finding those safe spots from natural images under a given input modification budget and the degree of robustness achievable by utilizing these spots, which we denote as preemptive robustness. Ultimately, we tackle the following question:
  • Do safe spots always exist in the vicinity of natural images?
  One practical example of the proposed framework is the case where a user uploads his or her photo from local storage (e.g., mobile device) to social media (e.g., Instagram), as illustrated in Figure 1. Suppose there is an uploader (A) who posts a photo on social media, a web user (B) who queries a search engine (e.g., Google) for an image, and a search engine that crawls images from social media, indexes them with a neural network, and retrieves the relevant images to B. Our threat model considers an adversary (M) that can download A’s image from social media, perturb it maliciously, and re-upload the perturbed image on the web, where the search engine may crawl and index images from. The classifier on the search engine will wrongly index the perturbed image, causing the search
  engine to malfunction. Suppose an African-American uploader (A) posts a photo of him or herself on social media, and a racist adversary (M) perturbs it to be misclassified as “gorilla” by the search engine. When another person (B) searches “gorilla” on Google, the perturbed image would appear, though the image content shows a photo of A. This attack fools both A and B since the perturbed image is used contrary to A’s purpose and is not the image B wanted. To prevent this, the social media company, cooperating with the search engine company, could ask if A agrees that the images are slightly changed when uploaded to make them robust to such attacks. The purpose of the modification process, corresponding to the “safe spot filter” in Figure 1, will be to ensure that the uploaded images are used under A’s intention and provide more accurate search results to B.
  We develop a novel optimization problem for searching safe spots in the vicinity of natural images and observe that over 90% of the correctly classified images have safe spots nearby for adversarially trained models on both CIFAR-10 (Krizhevsky & Hinton, 2009) and ImageNet (Russakovsky et al., 2015). We also find that safe spots can enhance both empirical and certified robustness when applied on smoothed classifiers. Furthermore, we propose a novel safe spot inducing model training scheme to improve the preemptive robustness. By exploiting these safe spot-aware classifiers along with our safe spot search method, we also propose a new algorithm for out-of-distribution detection, which is often addressed together with robustness (Hendrycks et al., 2019a;c). Our algorithm outperforms other baselines on near-distribution outlier datasets such as CIFAR-100 (Krizhevsky & Hinton, 2009).
sections.heading: 2 RELATED WORKS
sections.text: Adversarial training Goodfellow et al. (2015) first show that the robustness of a neural network can be enhanced by generating adversarial examples and including them in training set. PGD adversarial training improves the robustness against stronger adversarial attacks by augmenting training data with multi-step PGD adversarial examples (Madry et al., 2017). Some recent works report performance gains over PGD adversarial training by modifying the adversarial example generation procedure (Qin et al., 2019; Zhang & Wang, 2019; Zhang et al., 2020). However, most of the recent algorithmic improvements can be matched by simply using early stopping with PGD adversarial training (Rice et al., 2020; Croce & Hein, 2020). Other line of works achieve performance gains by utilizing additional datasets (Carmon et al., 2019; Wang et al., 2020; Hendrycks et al., 2019a).
  Randomized smoothing Injecting random noise during the forward pass can smooth the classifier’s decision boundary and improve empirical robustness (Liu et al., 2018). Using differential privacy, Lecuyer et al. (2019) give theoretical guarantees for `1 and `2 robustness of classifiers smoothed with Gaussian and Laplacian noise. Cohen et al. (2019) provide a tight bound of `2 robustness of networks smoothed with Gaussian noise via the Neyman-Pearson lemma. Another proof of the robustness bound was given in Salman et al. (2019) using Lipschitz property of smoothed classifiers, where they also propose a new adversarial training scheme for building robust smoothed classifiers.
  Out-of-Distribution detection with deep networks Although deep networks achieve high performance on various classification tasks, they also tend to yield high confidence in out-of-distribution samples (Nguyen et al., 2015). To filter out the anomalous examples, Hendrycks & Gimpel (2017) use the maximum value of a classifier’s softmax distribution as a score function, while Lee et al. (2018)
  propose Mahalanobis distance-based metric which spots out-of-distribution samples using hidden features. Hendrycks et al. (2019b) show that leveraging auxiliary datasets disjoint from test-time data can improve the detection performance. Recently, Sastry & Oore (2020) characterize activity patterns of hidden features by Gram matrices and use the matrix values to identify anomalies.
sections.heading: 3 METHODS
sections.heading: 3.1 GENERAL DEFINITION OF SAFE SPOT AND PREEMPTIVE ROBUSTNESS
sections.text: We first establish a formal definition of safe spot and preemptive robustness. Let c : X → Y be a classifier which maps images to class labels. We define the safe region of the classifier c as the set of images that c can output robust predictions in the presence of slight adversarial perturbations. Definition 1 ( -safe region). Let c : X → Y be a classifier and ∈ R+ be the perturbation budget of an adversary. The -safe region of the classifier c is defined by S (c) := {x ∈ X | c(x′) = c(x), ∀x′ ∈ B (x)}. Here, B (x) = {x′ ∈ X | d(x, x′) ≤ } is the x-centered -ball. In this paper, we assume `p threat model, i.e., d(x, x′) = ‖x − x′‖p, which is the most common setting on adversarial robustness literature, and consider p ∈ {2,∞}. Now, suppose a defender can preemptively manipulate a natural image xo under a small modification budget, knowing its ground-truth label yo. We denote the modified output image as xs. Then, the defender’s objective is to make xs be correctly classified as yo and locate in the safe region S (c) to improve the robustness against adversarial attacks. If xs satisfies these two conditions, then we say xo is preemptively robust and xs is a safe spot of xo. Definition 2 (Preemptive robustness). Let c : X → Y be a classifier and δ, ∈ R+ be the modification budgets of the defender and the adversary, respectively. A natural image xo with its ground-truth label yo is called (δ, )-preemptively robust on the classifier c if there exists a safe spot xs ∈ Bδ(xo) such that (i) c(xs) = yo and (ii) xs ∈ S (c).
sections.heading: 3.2 SAFE SPOT SEARCH ALGORITHM
sections.text: In this subsection, we develop an algorithm for searching a safe spot with a natural image. Given a classifier c, finding a safe spot xs from a natural image xo can be formulated as the following problem, which is directly from the definition of safe spot:
  minimize xs 1c(xs)6=yo + 1xs /∈S (c)
  subject to ‖xs − xo‖p ≤ δ,
  where 1 is the 0-1 loss function.
  Note in this formulation the defender requires the ground-truth label yo for the safe spot search. However, images in the real-world (e.g., social media) are usually unlabeled, unless uploaders annotate labels to their images by hand. So, it is natural to assume that the defender cannot access the ground-truth label yo. In this case, we utilize the classifier’s prediction c(xo) instead of yo:
  minimize xs 1c(xs)6=c(xo) + 1xs /∈S (c)
  subject to ‖xs − xo‖p ≤ δ.
  As xs /∈ S (c) implies there exists an adversarial example xa ∈ B (xs) such that c(xa) 6= c(xs), we can reformulate the optimization problem as
  minimize xs 1c(xs)6=c(xo) + sup xa 1c(xa)6=c(xs)
  subject to ‖xs − xo‖p ≤ δ and ‖xa − xs‖p ≤ .
  Since the 0-1 loss is not differentiable, we employ the cross-entropy loss ` : X × Y → R+ of the classifier c as the convex surrogate loss function:
  minimize xs `(xs, c(xo)) + sup xa `(xa, c(xs)) (1)
  subject to ‖xs − xo‖p ≤ δ and ‖xa − xs‖p ≤ .
  Let h(xs) denote the objective in Equation (1). Instead of minimizing h(xs), we minimize h̃(xs) = supxa `(xa, c(xo)), since it upper bounds h(xs) when sufficiently minimized by Lemma 1.
  Algorithm 1 Finding a safe spot input An image and its prediction (xo, c(xo)), the cross-entropy
  function ` xs = xo for i = 1, . . . ,MAXITER do
  Generate N adversarial examples for n = 1, . . . , N do x (0) a,n = xs + ηn where ηn ∼ U(B (0))
  for t = 1, . . . , T do x (t) a,n = Πxs, ( f(x (t−1) a,n ; c(xo), `) ) end for
  end for xs ← Πxo,δ ( xs − β · 1
  N N∑ n=1 ∂`(x (T ) a,n, c(xo)) ∂xs ) end for
  output xs
  Lemma 1. If h̃(xs) ≤ − log(0.5) ' 0.6931, then h(xs) ≤ 2h̃(xs).
  Proof. See Supplementary A.1.
  Finally, we have the following optimization problem:
  minimize xs sup xa `(xa, c(xo)) (2)
  subject to ‖xs − xo‖p ≤ δ and ‖xa − xs‖p ≤ .
  To solve Equation (2), we first approximate the inner maximization problem by running T -step PGD (Madry et al., 2017) whose dynamics is given by
  x(0)a = xs + η (random start) x̃(t)a = f ( x(t−1)a ; c(xo), ` ) (adversarial update)
  x(t)a = Πxs, ( x̃(t)a ) , (projection)
  where η is a random noise uniformly sampled from the `p zero-centered -ball, f is FGSM (Goodfellow et al., 2015) defined by
  f(x; y, `) =  x+ α · sgn (∇x`(x, y)) if p =∞
  x+ α · ∇x`(x, y)‖∇x`(x, y)‖2 if p = 2,
  and Πxs, is a projection operation to B (xs). Then, we iteratively solve the approximate problem given by replacing xa to x (T ) a in Equation (2). To update xs, we need to compute the gradient of `(x (T ) a , c(xo)) with respect to xs expressed as
  ∂` ( x (T ) a , c(xo) ) ∂xs = ∂f̃ ( x (0) a ) ∂x ᵀ · · · ∂f̃ ( x (T−1) a ) ∂x ᵀ · ∇x` ( x(T )a , c(xo) ) ,
  where ∂f̃/∂x is the Jacobian matrix of f̃ = Πxs, ◦ f which is easily computed via back-propagation. After computing the gradient, we update xs by projected gradient descent method:
  x(i+1)s = Πxo,δ ( x(i)s − β · ∂`(x (T ) a , c(xo))
  ∂xs
  ) .
  Note that the loss `(x(T )a , c(xo)) is now a random variable dependent on η. Therefore, we generate N adversarial examples {x(T )a,n}Nn=1 with different noises and optimize the sample mean of the losses instead. Algorithm 1 shows the overall safe spot search algorithm and Figure 2 illustrates our optimization process.
sections.heading: 3.3 COMPUTING UPDATE GRADIENT WITHOUT SECOND-ORDER DERIVATIVES
sections.text: Computing the update gradient with respect to xs involves the use of second-order derivatives of the loss function ` since the dynamics f contains the loss gradient ∇x`(x, y). Standard deep learning libraries, such as PyTorch (Paszke et al., 2019), support the computation of higher-order derivatives. However, it imposes a huge memory burden as the size of the computational graph increases. Furthermore, for the case of p = 2, computing the update gradient with the second-order derivatives might cause exploding gradient problem if the loss gradient vanishes by Proposition 1. Lemma 2. Suppose ` is twice-differentiable and its second partial derivatives are continuous. If p = 2, the Jacobian of the dynamics f is
  ∂f ∂x = I + α ·
  ( I − ( g
  ‖g‖2
  )( g
  ‖g‖2
  )ᵀ) H
  ‖g‖2 ,
  where g = ∇x`(x, y) and H = ∇2x`(x, y).
  Proof. See Supplementary B.1.
  Proposition 1. If the maximum eigenvalue of H in absolute value is σ, then∥∥∥∥∂f∂x ᵀ · a ∥∥∥∥ 2 ≤ ( 1 + α · σ‖g‖2 ) ‖a‖2.
  Proof. See Supplementary B.2.
  As we update xs, the loss gradients of xs and its adversarial examples xa get reduced to zero, which might cause the update gradient to explode and destabilize the update process. To address this problem, we approximate the update gradient by excluding the second-order derivatives, following the practice in Finn et al. (2017). We also include an experiment in comparison to using the exact update gradient in supplementary B.3. For the case of p =∞, the second-order derivatives naturally vanish since we take the sign on the loss gradient ∇x`(x, y). Therefore, the approximate gradient is equal to the exact update gradient.
sections.heading: 3.4 FINDING A SAFE SPOT FOR CLASSIFIERS WITH RANDOMIZED SMOOTHING
sections.text: To further enhance the robustness of our safe spot framework, we can leverage the randomized smoothing technique along with our algorithm. Given a base classifier c : X → Y , the smoothed classifier g : X → Y is defined by
  g(x) = argmax y∈Y
  P (c(x+ η) = y) ,
  where η ∼ N (0, σ2I). To find a safe spot xs of a natural image xo, we have to find an adversarial example xa of xs that maximizes the cross-entropy loss `(xa, c(xo)) for solving the inner maximization problem in Equation (2). However, crafting adversarial examples for the smoothed classifier is ill-behaved since the argmax is non-differentiable. To address the problem, we follow the approach in Salman et al. (2019) and approximate the smoothed classifier g with the smoothed soft classifier G : X → P (Y) defined as
  G(x) = E η∼N (0,σ2I) [C(x+ η)] ,
  where P (Y) is the set of probability distribution over Y and C : X → P (Y) is the soft version of the base classifier c such that argmaxy∈Y C(x)y = c(x). Finally, the adversarial example xa is found by maximizing the cross-entropy loss of G instead:
  maximize xa
  − log ( G(xa)c(xo) ) (3)
  subject to ‖xa − xs‖p ≤ ,
  which can be approximated by T -step randomized PGD with M restarts, where random noises are sampled from Gaussian distribution to compute the sample mean of the objective at each step. By replacing the inner maximization problem in Equation (2) by the randomized PGD, we can update xs similarly.
sections.heading: 3.5 SAFE SPOT-AWARE ADVERSARIAL TRAINING
sections.text: In Section 3.2, we investigated how the defender can find a safe spot from a natural image, given a pre-trained classifier. In this subsection, we explore the defender’s training scheme for a classifier on which data points are preemptively robust. Suppose the defender has a labeled training set, which is drawn from a true data distribution D. To induce a classifier to have safe spots in the vicinity of data points, the defender’s optimal training objective should have the following form:
  minimize θ E (xo,yo)∼D
  [`(x∗a, yo; θ)]
  subject to x∗a = argmax xa∈B (x∗s) `(xa, yo) and x∗s = argmin xs∈Bδ(xo) sup xa∈B (xs) `(xa, yo),
  where θ is the set of trainable parameters. Concretely, the defender finds a safe spot candidate x∗s of a datapoint xo and generates an adversarial example x∗a from x ∗ s . Then, the defender minimizes the cross-entropy loss `(x∗a, yo; θ) so that x ∗ s becomes an actual safe spot. Note that the ground-truth label yo is used instead of the prediction c(xo), since we assume that the defender can access the ground-truth label during training.
  The most direct way to optimize the objective would be to find x∗s from xo using our safe spot search algorithm and perform k-step PGD adversarial training (Madry et al., 2017) with x∗s . However, since the safe spot search algorithm requires running T -step PGD dynamics per each update, the proposed training procedure would be more computationally demanding than PGD adversarial training. To ease this problem, we consider replacing the inner maximization supxa `(xa, yo) in the safe spot search by `(xs, yo):
  x∗s = argmin xs∈Bδ(xo) sup xa∈B (xs) `(xa, yo) =⇒ x∗s = argmin xs∈Bδ(xo) `(xs, yo).
  Then, x∗s can be easily computed by running targeted FGSM or k-step PGD on xo towards the ground-truth label yo. We denote this training scheme as safe spot-aware adversarial training.
sections.heading: 3.6 OUT-OF-DISTRIBUTION DETECTION
sections.text: The safe spot-aware adversarial training method induces the learned data distribution to have safe spots near its data points. Thus, we can naturally conjecture that the samples from the learned distribution will have a higher probability of having safe spots compared to the out-of-distribution (OOD) samples, as shown in Figure 3. We leverage this conjecture to propose a new out-of-distribution detection algorithm that jointly utilizes our safe spot generation method and safe spot-aware adversarial training.
  Following the framework of Hendrycks et al. (2019b), which use auxiliary outlier data to tune anomaly detectors, we consider there are three types of data distributions, Din, Dtrainout , and Dtestout . Din refers to the learned distribution, also called the in-distribution. Dtrainout is the given distribution of outliers used to tune the detection algorithm, which is orthogonal to Dtestout . Dtestout is the distribution we want to detect as OOD during inference, which is unknown. We include the auxiliary outlier data to
  our safe spot-ware training procedure and adapt the training objective as below:
  minimize θ E (xo,yo)∼Din [`(x∗a, yo; θ)] + E x̂o∼Dtrainout [γ ·DKL(ȳ ‖ C(x̂o; θ))− λ · `(x̂∗a, c(x̂o); θ)] (4) subject to x∗a = argmax xa∈B (x∗s) `(xa, yo) and x ∗ s = argmin xs∈Bδ(xo) sup xa∈B (xs) `(xa, yo)
  x̂∗a = argmax x̂a∈B (x̂∗s) `(x̂a, c(x̂o)) and x̂ ∗ s = argmin x̂s∈Bδ(x̂o) sup x̂a∈B (x̂s) `(x̂a, c(x̂o)),
  where ȳ is the uniform distribution and C(x̂o; θ) is the softmax probability of x̂o. Since x̂o is unlabeled, we use the prediction c(x̂o) instead for safe spot searching. Note that if ≥ δ, the first term in Equation (4) also maximizes the confidence of the original in-distribution samples, since xo ∈ Bδ(x∗s) ⊆ B (x∗s) and therefore `(xo, y; θ) ≤ `(x∗a, y; θ). Similarly, the second and third terms minimize the prediction confidence and the probability of safe spot existence of the outlier samples, respectively.
  With the trained classifier, we measure the safe spot objective value from Equation (2) along with the maximum softmax probability (MSP) and use the values as indicators to detect OOD samples. Concretely, we define the score function as a linear combination of the two indicators. Considering they have a different range of possible values, we replace the safe spot objective value with the MSP of the adversarial example for the safe spot solution. Finally, the score function is formulated as
  D(xo) := µ ·max y∈Y C(xo)y + (1− µ) ·max y∈Y
  C(x∗a)y,
  where x∗s ∈ Bδ(xo) is the optimal solution of the safe spot algorithm for xo and x∗a ∈ B (x∗s) is the adversarial example of x∗s . We filter inputs with low scores as OOD.
sections.heading: 4 EXPERIMENTS
sections.text: As it is natural to assume that the defender and the adversary have the same modification budget, we set δ = for all experiments. We evaluate our methods by measuring clean and adversarial accuracies, where adversarial accuracy refers to the prediction score under 20-step untargeted PGD attack with a step size of /4. In the experiment tables, None column indicates using original images as inputs, and S-Full uses safe spot images from Algorithm 1. We also evaluate safe spot search via targeted FGSM and 20-step PGD towards the class inferred from the classifier, each denoted as S-FGSM and S-PGD. Detailed settings are listed on supplementary C.
sections.heading: 4.1 CIFAR-10
sections.text: We use Wide-ResNet-34-10 (Zagoruyko & Komodakis, 2016) and consider two threat models, `∞ with = 8/255 and `2 with = 0.5. We run our experiments on four differently trained models. The natural model is trained in a standard manner without considering adversaries. ADV is a PGD adversarially trained model. S-FGSM+ADV and S-PGD+ADV are safe spot-aware adversarially trained models, with safe spot search approximated by FGSM or 10-step PGD with a step size of δ/4.
  The `∞ threat model result in Table 1 (left) shows our methods can find safe spots on over 85% of the test set images, except for the natural model. This performance is near the upper bound, which is the classifier’s clean accuracy since we use predicted labels for safe spot search. We also observe that safe spot search via targeted FGSM or PGD is also feasible for ADV, S-FGSM+ADV, and S-PGD+ADV models, but they still miss on about 10% of correctly classified images. When jointly used with our safe spot search method, the safe spot-aware training achieves the highest adversarial accuracy, along with a clean accuracy much higher than PGD adversarial training.
  The `2 threat model result in Table 1 (right) shows similar results as the `∞ experiment, except that the adversarial accuracy of safe spots generated by S-Full on the natural model is much higher. However, we note that the adversarial accuracy of safe spots on the natural model may go down to about 20% when the attack gets stronger, for example, by increasing PGD iterations. The results on stronger PGD attacks and other types of attacks are considered in supplementary D.3 and D.4.
sections.heading: 4.2 IMAGENET
sections.text: We use ResNet-50 and consider three threat models: `∞ with ∈ {4/255, 8/255} and `2 with = 3.0. For safe spot-aware adversarial training experiments, we utilize “fast” adversarial training (Wong et al., 2020) and train the safe spot-aware model S-FGSM+Fast to reduce the training cost.
  Table 2 (left) shows results on `∞ attack under = 4/255. Similar to results on CIFAR-10, our methods are capable of finding safe spots near to original images that are correctly classified on the robust classifiers. Also, our proposed safe spot-aware classifier outperforms the original robust classifier by a large margin in both clean and adversarial accuracies. Table 2 (right) shows results on `∞ on = 8/255. In this setting, we also apply our algorithm to the ADV model trained with train = 4/255. Note that by changing only the train value on adversarial training, we get a 10% gain on our safe spot’s adversarial accuracy. Surprisingly, classifiers adversarially trained with smaller train performs substantially better in terms of preemptive robustness compared to using more robust classifiers. This implies that the conventional notion of robustness does not necessarily translate to preemptive robustness. Experiments on `2 attacks show similar results and can be found in Supplementary D.1.
sections.heading: 4.3 RANDOMIZED SMOOTHING
sections.text: We also evaluate our algorithm for classifiers with randomized smoothing. Here, we consider the `2 threat model, where = 0.5 for CIFAR-10 and = 3.0 for ImageNet. We run experiments on the smoothed classifiers based on the natural and the Gaussian-noise augmented model, which are considered certifiably robust (Lecuyer et al., 2019; Cohen et al., 2019). We measure empirical robustness against the randomized PGD on both models and measure certified robustness on the Gaussian model. Detailed settings such as noise level σ and randomized PGD are listed in supplementary C.2.
  Table 3 shows our results on the empirical robustness against the randomized PGD. We observe that our algorithm can find safe spots for the natural model with randomized smoothing on 57% and 37% of correctly classified images of CIFAR-10 and ImageNet, respectively. Furthermore, as shown in supplementary D.3, the adversarial accuracy of the smoothed natural model does not suffer from accuracy drop when the attack becomes stronger, in contrast to the natural model. Also, the smoothed Gaussian model, whose training cost is comparable to standard training much less than PGD adversarial training, achieves higher clean and adversarial accuracy compared to the ADV model. Certified robustness results of smoothed classifiers can be found in Supplementary D.2, where our safe spot algorithm also improves the certified robustness on both the datasets.
sections.heading: 4.4 OUT-OF-DISTRIBUTION DETECTION
sections.text: We evaluate the performance of our proposed detection algorithm on models trained with CIFAR-10. We consider various OOD datasets including CIFAR-100, SVHN (Netzer et al., 2011), TinyImageNet (Johnson et al.), LSUN (Yu et al., 2015), and synthetic noise. Following the experimental protocol of Hendrycks et al. (2019b), we evaluate the OOD detection methods on three metrics: area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPR), and the false positive rate at 95% true positive rate (FPR95). We compare our method’s performance to Mahalanobis (Lee et al., 2018), OE (Hendrycks et al., 2019b), and Gram (Sastry & Oore, 2020). Since Lee et al. (2018) utilizes a subset of the Dtestout data for tuning the detection procedure while our method and OE do not, we modify Mahalanobis to tune with Dtrainout for fair comparison. For detailed descriptions of the datasets and the experiments, refer to Supplementary E.1 and E.2.
  Table 4 shows the evaluation results. While Mahalanobis and Gram works slightly better on synthetic datasets such as Gaussian noise, on more near-distribution outliers such as CIFAR-100, TinyImageNet, and LSUN, our method outperforms these baselines by a large margin, which leads to a gain in overall performance. Our method also outperforms OE on most metrics including the Gaussian noise.
sections.heading: 5 CONCLUSION
sections.text: Parting from recent studies on adversarial examples, we present a new adversarial framework where the defender preemptively modifies classifier inputs. We introduce a novel optimization algorithm for finding safe spots in the vicinity of original inputs as well as a new network training method suited for enhancing preemptive robustness. The experiments show that our algorithm can find safe spots for robust classifiers on most of the correctly classified images. Further results show that they can be used to improve empirical and certified robustness on smooth classifiers. Finally, we combine the new network training scheme and the safe spot generation method to devise a new out-of-distribution detection algorithm that achieves the state of the art performance on near-distribution outliers.