summaries: This paper proposes a method to produce image classifiers which are adversarially robust against multiple $\ell_p$ threat models—in particular, against $\ell_1$, $\ell_2$, and $\ell_\infty$ attacks. The method involves training against $\ell_1$ and $\ell_\infty$ attacks with the hypothesis that this will additionally give robustness for $\ell_p$ threat models with $1 \leq p \leq \infty$. This hypothesis is supported by prior results that proved that affine classifiers robust to $\ell_1$ and $\ell_\infty$ threat models are also be robust to other $\ell_p$ threat models. The authors test their method on CIFAR-10 and ImageNet for both training classifiers from scratch and for fine-tuning robust models trained on one $\ell_p$ threat model to the other $\ell_p$ threat models. The paper tackles the problem of robustness against multiple perturbations and proposes extreme norms adversarial training (E-AT) that adaptively alternates between $\ell_1$ and $\ell_\infty$-norm. Furthermore, the paper fine-tunes Gowal et al. (2020) to improve its multi-norm robustness. Finally, the experiments are conducted on CIFAR-10 and ImageNet with APGD for training, showing the proposed method's effectiveness. This paper addresses the problem of multiple perturbation adversarial robustness for attacks subsumed within $\ell_p$ regions for $p\in{1,2,\infty}$. The main contribution of this work is to show how a model robust to a particular attack type (typically $\ell_\infty$) can be fine-tuned (at low cost) to be robust against multiple (or alternate) perturbation types. The authors build on prior formalization about the geometry of $\ell_p$ balls (by Croce et. al.) to empirically demonstrate its effect. The results are convincing and evaluated against AutoAttack which is This paper mainly studies the problem of defending multiple norm adversarial perturbations. The authors propose extreme norms adversarial training (E-AT), which leverages different geometry of the $\ell_{p}$-balls to conduct adversarial training by adaptively alternating between the $\ell_{1}$ norm and $\ell_{\infty}$ norm. They also show that using E-AT fine-tune could turn $\ell_{p}$ robust model into a model that is robust against the union of $\ell_{p}$ adversarial perturbations. The authors also provide some theoretical proof for their method. 
abstractText: Adversarial training (AT) in order to achieve adversarial robustness wrt single lpthreat models has been discussed extensively. However, for safety-critical systems adversarial robustness should be achieved wrt all lp-threat models simultaneously. In this paper we develop a simple and efficient training scheme to achieve adversarial robustness against the union of lp-threat models. Our novel E-AT scheme is based on geometric considerations of the different lp-balls and costs as much as normal adversarial training against a single lp-threat model. Moreover, we show that using our E-AT scheme one can fine-tune with just 3 epochs any lp-robust model (for p ∈ {1, 2,∞}) and achieve multiple norm adversarial robustness. In this way we boost the state-of-the-art for multiple-norm robustness to more than 51% on CIFAR-10 and report up to our knowledge the first ImageNet models with multiple norm robustness. Moreover, we study the general transfer of adversarial robustness between different threat models and in this way boost the previous SOTA l1-robustness on CIFAR-10 by almost 10%.
references.authors: Abdullah Al-Dujaili Una-May O’Reilly
references.title: There are no bit parts for sign bits in black-box attacks
references.venue: In ICLR,
references.authors: Maksym Andriushchenko Francesco Croce Nicolas Flammarion Matthias Hein
references.title: Square attack: a query-efficient black-box adversarial attack via random search
references.venue: In ECCV,
references.authors: Raman Arora Amitabh Basu Poorya Mianjy Anirbit Mukherjee
references.title: Understanding deep neural networks with rectified linear unit
references.venue: In ICLR,
references.authors: Anish Athalye Nicholas Carlini David Wagner
references.title: Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples
references.authors: Maximilian Augustin Alexander Meinke Matthias Hein
references.title: Adversarial robustness on in- and out-distribution improves explainability
references.venue: In ECCV,
references.authors: Wieland Brendel Jonas Rauber Matthias Bethge
references.title: Decision-based adversarial attacks: Reliable attacks against black-box machine learning models
references.venue: In ICLR,
references.authors: Nicholas Carlini David Wagner
references.title: Towards evaluating the robustness of neural networks
references.venue: In IEEE Symposium on Security and Privacy,
references.authors: Yair Carmon Aditi Raghunathan Ludwig Schmidt John C Duchi Percy S Liang
references.title: Unlabeled data improves adversarial robustness
references.venue: In NeurIPS,
references.authors: Pin-Yu Chen Yash Sharma Huan Zhang Jinfeng Yi Cho-Jui Hsieh
references.title: Ead: Elastic-net attacks to deep neural networks via adversarial examples
references.venue: In AAAI,
references.authors: Tianlong Chen Sijia Liu Shiyu Chang Yu Cheng Lisa Amini Zhangyang Wang
references.title: Adversarial robustness: From self-supervised pre-training to fine-tuning
references.authors: Minhao Cheng Thong Le Pin-Yu Chen Jinfeng Yi Huan Zhang Cho-Jui Hsieh
references.title: Queryefficient hard-label black-box attack: An optimization-based approach
references.venue: In ICLR,
references.authors: Francesco Croce Matthias Hein
references.title: Minimally distorted adversarial examples with a fast adaptive boundary attack
references.venue: In ICML,
references.authors: Francesco Croce Matthias Hein
references.title: Provable robustness against all adversarial lp-perturbations for p ≥ 1
references.venue: In ICLR,
references.authors: Francesco Croce Matthias Hein
references.title: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
references.venue: In ICML,
references.authors: Francesco Croce Matthias Hein
references.title: Mind the box: l1-apgd for sparse adversarial attacks on image classifiers
references.venue: In ICML,
references.authors: Francesco Croce Maksym Andriushchenko Vikash Sehwag Nicolas Flammarion Mung Chiang Prateek Mittal Matthias Hein
references.title: Robustbench: a standardized adversarial robustness benchmark
references.venue: arXiv preprint arXiv:2010.09670,
references.authors: Francesco Croce Maksym Andriushchenko Naman D. Singh Nicolas Flammarion Matthias Hein
references.title: Sparse-rs: a versatile framework for query-efficient sparse black-box adversarial attacks
references.venue: arXiv preprint,
references.authors: Nilesh Dalvi Pedro Domingos Mausam Sumit Sanghai Deepak Verma
references.title: Adversarial classification
references.venue: In KDD,
references.authors: Gavin Weiguang Ding Yash Sharma Kry Yik Chau Lui Ruitong Huang
references.title: Mma training: Direct input space margin maximization through adversarial training
references.venue: In ICLR,
references.authors: Logan Engstrom Andrew Ilyas Hadi Salman Shibani Santurkar Dimitris Tsipras
references.title: Robustness (python library), 2019
references.venue: URL https://github.com/MadryLab/robustness
references.authors: Ian Goodfellow Yoshua Bengio Aaron Courville
references.title: Deep Learning
references.authors: Ian J Goodfellow Jonathon Shlens Christian Szegedy
references.title: Explaining and harnessing adversarial examples
references.venue: In ICLR,
references.authors: Sven Gowal Krishnamurthy Dvijotham Robert Stanforth Rudy Bunel Chongli Qin Jonathan Uesato Relja Arandjelovic Timothy Mann Pushmeet Kohli
references.title: On the effectiveness of interval bound propagation for training verifiably robust models
references.authors: Sven Gowal Jonathan Uesato Chongli Qin Po-Sen Huang Timothy Mann Pushmeet Kohli
references.title: An alternative surrogate loss for pgd-based adversarial testing
references.authors: Sven Gowal Chongli Qin Jonathan Uesato Timothy Mann Pushmeet Kohli
references.title: Uncovering the limits of adversarial training against norm-bounded adversarial examples
references.venue: arXiv preprint arXiv:2010.03593v2,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Identity mappings in deep residual networks
references.venue: In ECCV,
references.authors: Dan Hendrycks Thomas Dietterich
references.title: Benchmarking neural network robustness to common corruptions and perturbations
references.venue: In ICLR,
references.authors: Dan Hendrycks Kimin Lee Mantas Mazeika
references.title: Using pre-training can improve model robustness and uncertainty
references.venue: In ICML,
references.authors: Jeremy Howard Sebastian Ruder
references.title: Universal language model fine-tuning for text classification
references.venue: In ACL,
references.authors: Ahmadreza Jeddi Mohammad Javad Shafiee Alexander Wong
references.title: A simple fine-tuning is all you need: Towards robust deep learning via adversarial fine-tuning
references.venue: arXiv preprint,
references.authors: Daniel Kang Yi Sun Tom Brown Dan Hendrycks Jacob Steinhardt
references.title: Transfer of adversarial robustness between perturbation types
references.venue: arXiv preprint,
references.authors: Daniel Kang Yi Sun Dan Hendrycks Tom Brown Jacob Steinhardt
references.title: Testing robustness against unforeseen adversaries
references.venue: arXiv preprint arXiv:1908.08016,
references.authors: Alexey Kurakin Ian Goodfellow Samy Bengio
references.title: Adversarial examples in the physical world
references.venue: In ICLR Workshop,
references.authors: Cassidy Laidlaw Sahil Singla Soheil Feizi
references.title: Perceptual adversarial robustness: Defense against unseen threat models
references.venue: In ICLR,
references.authors: Sijia Liu Pin-Yu Chen Xiangyi Chen Mingyi Hong
references.title: signSGD via zeroth-order oracle
references.venue: In ICLR,
references.authors: Daniel Lowd Christopher Meek
references.title: Adversarial learning
references.venue: In KDD,
references.authors: Divyam Madaan Jinwoo Shin Sung Ju Hwang
references.title: Learning to generate noise for multi-attack robustness
references.venue: In ICML,
references.authors: Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras Adrian Vladu
references.title: Towards deep learning models resistant to adversarial attacks
references.venue: In ICLR,
references.authors: Pratyush Maini Eric Wong Zico Kolter
references.title: Adversarial robustness against the union of multiple perturbation models
references.venue: In ICML,
references.authors: Laurent Meunier Jamal Atif Olivier Teytaud
references.title: Yet another but more efficient black-box adversarial attack: tiling and evolution strategies
references.venue: arXiv preprint,
references.authors: Apostolos Modas Seyed-Mohsen Moosavi-Dezfooli Pascal Frossard
references.title: Sparsefool: a few pixels make a big difference
references.authors: Marius Mosbach Maksym Andriushchenko Thomas Trost Matthias Hein Dietrich Klakow
references.title: Logit pairing methods can fool gradient-based attacks
references.venue: NeurIPS
references.authors: Sukrut Rao David Stutz Bernt Schiele
references.title: Adversarial training against location-optimized adversarial patches
references.venue: In ECCV Workshop on the Dark and Bright Sides of Computer Vision: Challenges and Oppurtinities for Privacy and Security,
references.authors: Jérôme Rony Eric Granger Marco Pedersoli Ismail Ben Ayed
references.title: Augmented Lagrangian adversarial attacks
references.venue: In ICCV,
references.authors: Lukas Schott Jonas Rauber Matthias Bethge Wieland Brendel
references.title: Towards the first adversarially robust neural network model on mnist
references.venue: In ICLR,
references.authors: David Stutz Matthias Hein Bernt Schiele
references.title: Disentangling adversarial robustness and generalization
references.authors: David Stutz Matthias Hein Bernt Schiele
references.title: Confidence-calibrated adversarial training: Generalizing to unseen attacks
references.venue: In ICML,
references.authors: Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna Dumitru Erhan Ian Goodfellow Rob Fergus
references.title: Intriguing properties of neural networks
references.venue: In ICLR,
references.authors: Florian Tramèr Dan Boneh
references.title: Adversarial training and robustness for multiple perturbations
references.venue: In NeurIPS,
references.authors: Florian Tramèr Nicholas Carlini Wieland Brendel Aleksander Madry
references.title: On adaptive attacks to adversarial example defenses
references.venue: In NeurIPS,
references.authors: Eric Wong J. Zico Kolter
references.title: Learning perturbation sets for robust machine learning
references.venue: In ICLR,
references.authors: Eric Wong Frank R Schmidt Jan Hendrik Metzen J Zico Kolter
references.title: Scaling provable adversarial defenses
references.venue: In NeurIPS,
references.authors: Boxi Wu Jinghui Chen Deng Cai Xiaofei He Quanquan Gu
references.title: Do wider neural networks really help adversarial robustness
references.venue: arXiv preprint arXiv:2010.01279v2,
references.authors: Dongxian Wu Shu tao Xia Yisen Wang
references.title: Adversarial weight perturbation helps robust generalization
references.venue: In NeurIPS,
references.authors: Cong Xu Min Yang
references.title: Adversarial momentum-contrastive pre-training
references.venue: arXiv preprint,
references.authors: Sergey Zagoruyko Nikos Komodakis
references.title: Wide residual networks
references.venue: In BMVC,
references.authors: Michał Zajac Konrad Zołna Negar Rostamzadeh Pedro O Pinheiro
references.title: Adversarial framing for image and video classification
references.venue: In AAAI,
references.authors: Hongyang Zhang Yaodong Yu Jiantao Jiao Eric Xing Laurent El Ghaoui Michael Jordan
references.title: Theoretically principled trade-off between robustness and accuracy
references.authors: Pu Zhao Sijia Liu Pin-Yu Chen Nghia Hoang Kaidi Xu Bhavya Kailkhura Xue Lin
references.title: On the design of black-box adversarial examples by leveraging gradient-free optimization and operator splitting method
references.venue: In ICCV,
references.authors: Komodakis
references.title: 2016) we use a cyclic schedule for the learning rate with maximum value 0.1 for 30 epochs. We use SGD optimizer with momentum of 0.9 and weight decay of 5 · 10−4, batch size of 128. We use random cropping and horizontal flipping as augmentation techniques. For adversarial training of models robust wrt a single norm and with SAT and our novel scheme E-AT we use APGD with default parameters, while for the retrained AVG, MAX, and MSD we use PGD for l∞
references.authors: Croce Hein
references.title: B ADDITIONAL ANALYSIS, EVALUATION AND EXPERIMENTS FOR E-AT We here analyze in more details our E-AT scheme and expand the comparison to existing methods presented above
references.venue: WRT
references.authors: Maini
references.title: same robustness in the union but is better in l1- and l2-robustness and requires only about about a third or half of the training time
references.authors: Rao
references.title: E-AT outperforms even the PAT model which is trained wrt LPIPS and aims at generalization to unseen attacks. Experimental details: To test robustness against l0-attacks we use Sparse-RS (Croce et al., 2020b) with a budget of 18 pixels and 10k queries. We adopt patches of size 5 × 5 pixels, optimized with the PGD-based
references.authors: Maini
references.title: l∞-JPEG = 0.25). Finally, we average the classification accuracy over the 5 severities of the common corruptions. All the statistics are on 1000 test points. E EXPERIMENTS ON MNIST Table 15: MNIST - Comparison of full training schemes and fine-tuning with E-AT for multiple norm robustness: We train classifier (architecture
references.authors: Maini
references.title: SAT and E-AT we report, together with the statistics over multiple random seeds, the results of the best run. Additionally, we show the results of fine-tuning the l2-AT model with E-AT for different numbers of epochs, which achieves the best results
references.authors: Maini
references.title: 2020) as architecture and ∞ = 0.3, 2 = 2 and 1 = 10 as threshold at which evaluating robustness, as done by Maini et al. (2020)
references.authors: Tramèr
references.title: l∞-adversarial training induces gradient obfuscation when using
references.authors: Maini
references.title: 2020) had to use many PGDsteps (up to 100), and Tramèr & Boneh (2019) even a ramp-up schedule for the during training
references.authors: Maini
references.title: First, E-AT outperforms the available classifiers trained with AVG, MAX
sections.heading: 1 INTRODUCTION
sections.text: The problem of adversarial examples, that is small adversarial perturbations of the input (Szegedy et al., 2014; Kurakin et al., 2017) changing the decision of a classifier, is a serious obstacle for the use of machine learning in safety-critical systems. Many adversarial defenses have been proposed but most of them could be broken either by stronger attacks with a higher computational budget (Carlini & Wagner, 2017; Athalye et al., 2018; Mosbach et al., 2018) or using adaptive attacks (Tramèr et al., 2020). Apart from provable adversarial defenses which are however still restricted to rather simple CNNs (Wong et al., 2018; Gowal et al., 2018), the only successful technique so far remains adversarial training (Madry et al., 2018) and its improvements (Zhang et al., 2019; Carmon et al., 2019; Wu et al., 2020; 2021). We refer to Gowal et al. (2020) for a recent overview of “tricks of the trade” for improving adversarial training yielding the currently most robust models for l2 and l∞ on CIFAR-10. While the community initially focused on adversarial examples for l∞-perturbations, there has been recently more interest in other lp-attacks, such as l1 and l2, or perceptual threat models (Stutz et al., 2019; Wong & Kolter, 2021; Laidlaw et al., 2021). It is well known that robustness in one lp-ball does not necessarily generalize to some other lq-ball for p 6= q (Kang et al., 2019a; Tramèr & Boneh, 2019). However, in safety-critical systems we need robustness against all lp-norms simultaneously which has triggered recent extensions of adversarial training for multiple lp-norms (Tramèr & Boneh, 2019; Maini et al., 2020) and provable defenses for all lp with p ≥ 1 (Croce & Hein, 2020b). In this paper we show that, using the geometry of the lp-balls, the computationally expensive multiple norm training procedures of Tramèr & Boneh (2019); Maini et al. (2020), which cost up to three times as much as normal adversarial training, can be replaced by a very effective and simple form of adaptively alternating between the two extreme norms, namely l1 and l∞. This simple scheme achieves similar robustness for the union of the threat models as more costly previous approaches. Moreover, we show that already 3 epochs of fine-tuning with our extreme norms adversarial training (E-AT) are enough to turn any lp-robust model for p ∈ {1, 2,∞} into a model which is robust
  against all lp-threat models for p ∈ {1, 2,∞}, even if the original classifier was completely nonrobust against one of the threat models. We apply the proposed method to many existing models originally trained to be robust in a single norm: fine-tuning the currently most robust network in the l∞-threat model from Gowal et al. (2020) for CIFAR-10 we improve the current state-of-the-art performance for multiple norm robustness by more than 6% (robustness over the union of l1, l2 and l∞-balls). Moreover, we employ our E-AT fine-tuning scheme with just a single epoch to yield the first ImageNet model which is robust against multiple attacks at the same time. Finally, we show that fine-tuning with just 3 epochs for CIFAR-10 and one epoch for ImageNet is sufficient to transfer robustness from one threat model to another one which very quickly yields baselines for all threat models. In this way we achieve SOTA l1-robustness on CIFAR-10 by fine-tuning the most robust l∞-model from Gowal et al. (2020) to become l1-robust, and get an ImageNet model robust wrt l1. These results are quite striking as the original classifiers show no or only very little l1-robustness. Fig. 1 summarizes the results of robust fine-tuning on CIFAR-10 (see Table 2 and 4 for details).
sections.heading: 2 RELATED WORK
sections.text: Adversarial training: In image classification adversarial examples were first described by Szegedy et al. (2014) even though earlier discussion of adversarial examples in email spam classification can be found in Dalvi et al. (2004); Lowd & Meek (2005). An early approach of adversarial training was the Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015), which then was extended to a multi-step attack in Kurakin et al. (2017). Nowadays, adversarial training as formulated in Madry et al. (2018) as a min-max optimization problem has been one of the few adversarial defenses which could not be broken by stronger attacks (Athalye et al., 2018). Other types of defenses use more sophisticated techniques, typically preventing the direct optimization of the attack. However, adaptive attacks specifically designed for these defenses have often shown that these alternative techniques are non-robust or much less robust than claimed (Tramèr et al., 2020). Thus adversarial training remains the only general method ensuring adversarial robustness across architectures and datasets. Recent improvements have been achieved by using different objectives (Zhang et al., 2019), unlabeled data (Carmon et al., 2019), adversarial weights perturbations (Wu et al., 2020) and wider networks Wu et al. (2021). In Gowal et al. (2020) several recent variants were systematically explored and for a very large architecture, a WideResNet-70-16, they obtain the most robust models for l∞ (radius 8255 ) and l2 (radius 0.5) for CIFAR-10, which we use for our fine-tuning experiments.
  Multiple norm robustness: It was early on discovered that adversarial robustness against a specific lp threat model does typically not transfer to lq threat models for p 6= q (see Kang et al. (2019a); Tramèr & Boneh (2019) for extensive studies). On the other hand to achieve really reliable machine learning models lp-robustness wrt all p is necessary. The first approach to such a generally robust model was done in Schott et al. (2019) which uses multiple variational auto-encoders for an analysis by synthesis (ABS) architecture. While this model is restricted to MNIST, it is robust against l0, l2 and l∞-attacks. However, it has been recently shown that with a stronger black-box attack (Croce et al., 2020b) the l0-robustness is significantly lower than originally claimed. Tramèr & Boneh (2019); Maini et al. (2020); Madaan et al. (2021) use variants of adversarial training to
  achieve robustness in multiple norms. Since these are the most similar methods to ours, we present them in detail below. Madaan et al. (2021) additionally proposes a meta-learning approach where one learns optimal noise to augment the samples and uses consistency regularization to enforce similar predictions on clean, augmented and adversarial samples. Finally, Stutz et al. (2020) combine adversarial training with a reject option for the classifier by down-weighting the confidence of adversarial samples. Their models generalize to other threat models but the comparison to normal adversarially trained models is difficult as their model is non-robust without the reject option.
  Provable robustness: In the area of provably robust defenses, Croce & Hein (2020b) motivated a regularization approach based on the geometry of the lp-balls which enforces multiple-norm robustness during training which allows then to derive provable guarantees for multiple-norm robustness in contrast to the empirical evaluation of adversarial training. However, their approach works only for small network architectures and relatively small radii of the lp-balls.
  Fine-tuning of robust models: Fine-tuning of an existing neural network is a commonly used technique in deep learning (Goodfellow et al., 2016) to quickly adapt an existing model to a different objective e.g. for language models (Howard & Ruder, 2018). More recently, it has been shown in the area of adversarial robustness that fine-tuning of pre-trained models, possibly using self-supervision, yields better adversarial robustness (Hendrycks et al., 2019; Chen et al., 2020; Xu & Yang, 2020). In Jeddi et al. (2020) it is shown that fine-tuning of non-robust models with 10 epochs can yield robust models with the caveat that their robustness evaluation is done using only a single run of PGD with 20 steps. We are not aware of any prior work discussing fine-tuning to transform an existing robust model wrt a single lp into a robust model wrt multiple threat models or wrt another lq-threat model. In particular, we show that one can fine-tune an l∞-robust model so that it becomes l1-robust even though the original model has not been l1-robust at all.
  Evaluation of adversarial robustness: Many white-box attacks for l∞ (Madry et al., 2018; Gowal et al., 2019), l2 (Madry et al., 2018; Carlini & Wagner, 2017) and l1 (Chen et al., 2018; Modas et al., 2019; Rony et al., 2021) have been proposed as well as several black box attacks (Brendel et al., 2018; Liu et al., 2019; Cheng et al., 2019; Al-Dujaili & O’Reilly, 2020; Meunier et al., 2019; Zhao et al., 2019; Andriushchenko et al., 2020) for different threat models. It has recently been shown that AutoAttack (Croce & Hein, 2020c), a parameter-free ensemble of the white-box attacks APGD for the cross-entropy and DLR-loss, FAB-attack (Croce & Hein, 2020a) and the black-box Square-attack (Andriushchenko et al., 2020) is reliably evaluating adversarial robustness for l2 and l∞. AutoAttack has recently been extended to l1 (Croce & Hein, 2021) outperforming all existing state-of-the art attacks for l1. Croce & Hein (2020c; 2021) showed that on models defended with adversarial training the two versions of APGD (with budget as in AutoAttack) already give an accurate robustness evaluation. As we have to evaluate very large models always for three threat models, we use those as a strong standard attack in our experiments.
  2.1 ADVERSARIAL TRAINING FOR THE UNION OF l1-, l2- AND l∞-BALLS
  Let us denote by fθ : Rd → RK the classifier parameterized by θ ∈ Rn, with input x ∈ Rd and fθ(x) ∈ RK where K is the number of classes of the task. Let further D = {(xi, yi)}i be the training set, with yi the correct label of xi, and L : RK × RK → R a given loss function. The aim is to enforce adversarial robustness in all multiple lp-balls simultaneously, i.e., defining Bp( p) = {x ∈ Rd : ‖x‖p ≤ p}, the threat model is the union of the individual lp-balls
  ∆ = B1( 1) ∪B2( ) ∪B∞( ∞), which is a non convex set, since in practice the p are chosen such that no lp-ball contains any of the others. In adversarial training the worst case loss for input perturbation in the threat model, maxδ∈∆ L(fθ(xi + δ), yi), is minimized. Efficiently maximizing the loss L in the union of threat models is non-trivial and different approaches to extend adversarial training to this setting have been proposed. They basically differ in the way how the inner maximization problem is tackled.
  MAX: Tramèr & Boneh (2019) suggest to run the three attacks for each Bp( p) for p ∈ {1, 2,∞} independently and then use the one which realizes the highest loss, that is
  max δ∈∆ L(fθ(xi + δ), yi) = max p∈{1,2,∞} max δ∈Bp( p) L(fθ(xi + δ), yi).
  This training optimizes directly the worst case in the union but comes at the price of being nearly three times as expensive as normal adversarial training wrt a single lp-ball.
  AVG: In their AVG alternative Tramèr & Boneh (2019) suggest to run the three attacks for each Bp( p) for p ∈ {1, 2,∞} independently but replace the inner maximization problem with∑
  p∈{1,2,∞}
  max δ∈Bp( p)
  L(fθ(xi + δ), yi),
  and thus one basically averages the updates of all lp-balls with the motivation of not “wasting” the computed attacks, in particular when the attained loss values are rather similar and thus the max is ambiguous. Again this costs three times as much as normal adversarial training.
  MSD: Maini et al. (2020) argue that the correct way to maximize the loss in the union is to test during the PGD attacks all the three steepest ascent updates corresponding to the three norms (sign of the gradient for l∞, normalized gradient for l2 and a smoothed l1-step by using the top-k components in magnitude of the gradient) and then take the step which yields the highest loss. Maini et al. (2020) report that MSD outperforms both AVG and MAX, also in terms of a more stable training. As all three updates (forward passes) are tested but only one backward pass is needed (gradient is the same) this costs roughly two times as much as normal adversarial training.
  SAT: Madaan et al. (2021) introduce Stochastic Adversarial Training (SAT) which randomly samples p ∈ {1, 2,∞} for each batch and performs PGD only for the corresponding lp-norm. While SAT has the same cost as standard adversarial training, Madaan et al. (2021) report that it does not perform very well.
sections.heading: 3 FAST MULTIPLE-NORM ROBUSTNESS VIA EXTREME NORMS ADVERSARIAL TRAINING AND FINE-TUNING
sections.text: All previous methods assume that for achieving robustness to multiple norms each of the threat models has to be used at training time. In the following we first present an argument, using recent results from Croce & Hein (2020b), showing that this is not the case. Based on this analysis we introduce our extreme norms adversarial training (E-AT) which achieves multiple norm robustness at the same price as training for a single norm threat model. Finally, we show that only a few epochs of fine-tuning with E-AT turns a model robust wrt a single norm into one which has competitive robustness wrt multiple norms.
  3.1 GEOMETRY OF THE UNION OF lp-BALLS AND THEIR CONVEX HULL
  The main insight we use for E-AT is that a linear classifier which is robust in both an l1- and an l∞-ball is also robust wrt the largest lp-ball for 1 ≤ p ≤ ∞ which fits into the convex hull of the union of the l1- and l∞-ball. This ball is significantly larger than the largest lp-ball contained into the union of the l1- and l∞-ball (see Fig. 2). Thus it is sufficient to be robust wrt the two “extreme” norms l1 and l∞ to ensure robustness. While this is exact for affine classifiers, we conjecture that for neural networks this will at least hold approximately true (note that typical ReLU-networks yield piecewise affine classifiers (Arora et al., 2018)) and for the model it is the most efficient way in terms of capacity to be l1- and l∞-robust.
  We now state the main results from Croce & Hein (2020b) which are the basis for our E-AT. We work in the non-trivial setting, where the balls are not included in each other, that is B1( 1) * B∞( ∞) and B∞( ∞) * B1( 1) as otherwise the problem of enforcing multiple norms (including l1 or l∞) robustness boils down again to single norm robustness. In order to be in this non-trivial setting it has to hold 1 ∈ ( ∞, d ∞). For ∞ = 8255 and dimension d = 3072 as in CIFAR-10 this yields an upper bound 1 ≤ 96.38 which is far higher than what has been used for l1-threat models (we use 1 = 12 for CIFAR-10). We denote by U1,∞( 1, ∞) = B1( 1) ∪B∞( ∞) the union of the l1- and l∞-balls. One can then ask the obvious question: how much lp-robustness do I get for 1 < p < ∞ from a classifier robust in U1,∞( 1, ∞)? This question is answered in the following result:
  Proposition 3.1 (Croce & Hein (2020b)) If d ≥ 2 and 1 ∈ ( ∞, d ∞), then
  min x∈Rd\U1,∞( 1, ∞)
  ‖x‖p = ( p∞ + ( 1 − ∞)p
  (d− 1)p−1
  ) 1 p
  . (1)
  Thus a classifier which is robust for the union U1,∞( 1, ∞) has automatically a non-trivial robustness for all intermediate lp-norms. For us the case p = 2 is most interesting which given that 1 ∞ can be tightly upper bounded as
  2 := min x∈Rd\U1,∞( 1, ∞) ‖x‖2 ≤
  √ 2∞ +
  21 d− 1 . (2)
  For the case of CIFAR-10 where d = 3072 and 1 = 12 and ∞ = 8255 as chosen in Maini et al. (2020) one gets 2 ≤ 0.2188. As the radius of the l2-threat model in Maini et al. (2020) is chosen as 2 = 0.5, robustness in the union alone would not be sufficient to achieve the desired robustness wrt lp for p ∈ {1, 2,∞}. But the above point of view is the worst case. In the following we see that if we consider affine classifiers then a guarantee forB1( 1) and B∞( ∞) implies a guarantee with respect to the convex hull C of their union B1( 1) ∪ B∞( ∞) as an affine classifier generates a half-space and thus only the extreme points of B1( 1) resp. B∞( ∞) matter (see Figure 2 for illustrations of B1, B∞, their union and their convex hull).
  Theorem 3.1 (Croce & Hein (2020b)) Let C be the convex hull of B1( 1) ∪ B∞( ∞). If d ≥ 2 and 1 ∈ ( ∞, d ∞), then
  min x∈Rd\C
  ‖x‖p = 1
  ( 1/ ∞ − α+ αq) 1/q , (3)
  where α = 1 ∞ − b 1 ∞ c and 1p + 1 q = 1.
  As standard architectures using ReLU activation function yield a piecewise affine classifier one can anticipate that this result gives at least a rule of thumb on the expected lp-robustness when one is l1and l∞-robust. Again with the choice of 1, ∞ from above one gets for the radius of the l2-ball that fits into the convex hull C of the union of B1( 1) and B∞( ∞):
  2 := min x∈Rd\C
  ‖x‖2 = 1√
  1/ ∞ − α+ α2 ≈ 0.6178. (4)
  Thus for a desired l2-robustness with radius less than 0.6178 it is sufficient for an affine classifier, and at least plausible for a ReLU network, to enforce l1-robustness with 1 = 12 and l∞-robustness with ∞ = 8255 . This motivates our extreme norms adversarial training (E-AT) and fine-tuning.
sections.heading: 3.2 EXTREME NORMS ADVERSARIAL TRAINING (E-AT)
sections.text: In light of the geometrical argument presented in the previous section, we propose to train only on adversarial perturbations for the l∞- and l1-threat models if the l2-radius obtained from Theorem 3.1 is larger than the radius 2 of the l2-threat model. In this case it is sufficient to just train for the extremes l1 and l∞ in order to achieve robustness also to the intermediate lp-attacks with p ∈ (1,∞). Since we seek a method as expensive as standard adversarial training, for each batch we do either the l1- or the l∞-attack. For full training from a random initialization simply alternating or sampling
  uniformly at random from the l1- and l∞-attack, works already well. However, for very quick finetuning, e.g. just one epoch in the case of ImageNet, for multiple norm robustness from an existing classifier robust wrt a single threat model, one has to take into account the existing robustness of the model. Thus we use an adaptive sampling strategy based on the running averages, reset at every epoch, of the robust training errors rerr1 and rerr∞ (note that these running averages are computed just from averaging the robust error on the batches where the respective attack has been performed, thus no extra attacks are necessary), such that the probability for sampling the lp-threat model is
  rerrp rerr1 + rerr∞ , for p ∈ {1,∞}. (5)
  The motivation for this sampling scheme is that the robust error in the union ∆ is mainly influenced by the worst threat model. We show the effect of the biased sampling in E-AT fine-tuning in Sec. C.3. The next section shows that E-AT is a very competitive baseline compared to the significantly more involved and up to three times more expensive training schemes discussed in Sec. 2.1.
sections.heading: 3.3 MULTIPLE NORM ROBUSTNESS FROM RANDOM INITIALIZATION
sections.text: In this experiment on CIFAR-10 we evaluate the performance of the different adversarial training variants to achieve adversarial robustness wrt the union of the l∞, l2 and l1-threat models where we use the standard radii 1 = 12, 2 = 0.5 and ∞ = 8255 . As baselines we report the lp-robustness of models specifically trained for only a single norm: l1-AT, l2-AT and l∞-AT. We train WideResNet28-10 (Zagoruyko & Komodakis, 2016) with the TRADES-XENT loss since Gowal et al. (2020) show that this yields slightly higher robustness than standard TRADES (Zhang et al., 2019) without additional data (see Sec. A for more details). For lp-AT, SAT and E-AT we use APGD (Croce & Hein, 2020c) for training, while for AVG, MAX and MSD we use standard PGD for l2, l∞ and SLIDE (Tramèr & Boneh, 2019) for l1-PGD, even though we have observed no strong difference for multiple norms training compared to APGD. However, note that for MSD we do not use the l1-PGD variant suggested by Maini et al. (2020) as this resulted in significantly worse performance (worse than all other methods). For MNG-AC (Madaan et al., 2021) we use the original code (with the 1 we consider here and rescaled step size). Finally, we show also the performance of E-AT using a uniformly random sampling scheme (named E-AT unif.) for selecting the lp-attack to use for each batch rather than the biased one presented in Eq. (5).
  In the results in Table 1 we can see that the models we retrained using MSD and MAX perform best with 46.9% and 47.4% adversarial robustness in the union, but our E-AT attains very close results with 46.4% (within the range of standard deviation) with 1.7× and 2.7× faster runtime per epoch. Given that a pure l∞-training yields 52.0% l∞-robustness which turns out to be the most difficult norm, it is quite remarkable that multiple-norm robustness can be achieved with a relatively minor loss. The other variants SAT, MNG-AT, AVG and E-AT unif. perform worse with higher or similar computational cost to E-AT. Note that E-AT unif. performs similarly to SAT (which also samples uniformly the lp-attack for each batch) in every threat model without seeing l2-attacks at training
  time. Thus E-AT provides a very efficient and effective alternative to MSD and MAX which allows to scale adversarial training for multiple norms to larger problems such as ImageNet. We repeat the comparison with the same setup but on the smaller PreAct ResNet-18 (He et al., 2016) in Sec. B.3 with standard adversarial training on the cross-entropy loss and obtain similar results. Thus, E-AT generalizes across architectures and training schemes. Finally, we observe how almost all methods for multiple norms lead to a drop of clean accuracy compared to the single norm lp-AT, showing that the threat model of the union is significantly more challenging.
sections.heading: 3.4 MULTIPLE-NORM ROBUSTNESS VIA FAST FINE-TUNING OF EXISTING ROBUST MODELS
sections.text: In the previous section, we have shown that training for only the two extreme threat models is sufficient to achieve robustness wrt all three norms. Prior works (Tramèr & Boneh, 2019; Kang et al., 2019a) observed that models adversarially trained wrt l∞ give non trivial robustness to l2attacks, although lower than what one gets directly training against such attacks, and vice versa. This is confirmed by our evaluation of the models in Table 1 (top part), where we also notice that l1-AT provides good robust accuracy in l2. On the other hand, training for l∞ resp. l1 does not yield particular robustness to the dual norm, which is reasonable since the perturbations generated in the two threat models are very different, while l2 can be seen as an intermediate case which can defend at least partially against l∞- and l1-attacks. Therefore, we propose to use models trained for robustness wrt a single norm as good initializations to achieve, within a small computational budget, multiple norms robustness. This is done by fine-tuning pretrained models with our E-AT for 3 epochs when using CIFAR-10 and 1 epoch with ImageNet-1k, starting with learning rate 0.05 or 0.01, depending on the model, and decreasing by a factor of 10 at each epoch. We do 10 steps of APGD in adversarial training for CIFAR-10, while 5 and 15 with l∞ and l1 respectively on ImageNet since optimizing in
  the l1-ball requires more iterations in that case. When the model was originally trained with extra data beyond the training set on CIFAR-10, we use the 500k images introduced by Carmon et al. (2019) as additional data for fine-tuning (see also Sec. A).
  CIFAR-10: RobustBench (Croce et al., 2020a) provides a collection of the currently most robust classifiers. We took the most robust models, among those which do not use synthetic data, for l2and l∞-norm and the l1-robust one from Croce & Hein (2021) (all are trained with the same radii p as in our experiment). Each of them is fine-tuned for 3 epochs with E-AT where (∗) denotes models trained using extra data, which also means that we used the extra data from Carmon et al. (2019) for fine-tuning (Gowal et al. (2020) have created their own extra data but which is not available). We present in Table 2 the results. First of all the fine-tuning works for all tested architectures and results in many cases in stronger robustness in the union than for the specifically trained WideResNet-28-10 models. In particular, the most robust l∞-model from Gowal et al. (2020) with 65.6% l∞-robustness and only 8.1% l1-robustness can be fine-tuned to a multiple-norm robust model with 51.2% robustness which is up to our knowledge the best reported multiple-norm robustness. Very interesting is that the l2-robustness of 78.2% is quite close to the 81.7% l2-robustness of the specifically l2-trained model from Gowal et al. (2020). Moreover, the l1-robustness of 58.3% is close to the best reported one of 60.3% from Croce & Hein (2021) (however we improve this a lot in the next section) and the model has even higher clean accuracy. Clearly, this comes at the price of a significant loss in l∞ but this is to be expected. Striking is that fine-tuning the l2-robust model from Gowal et al. (2020) results in a very similar result. In a nutshell, E-AT fine-tuning of existing lp-robust models yields very efficient and competitive baselines for future research in this area.
  ImageNet: Similarly to CIFAR-10 we start with the specific l2-resp. l∞-robust models from Engstrom et al. (2019) which have been trained with 2 = 3 and ∞ = 4255 . We use = 2 for the experiments as the robust accuracy is still in a reasonable range of 40% and together with our choice of 1 = 255 and ∞ = 4255 the l2-radius from Theorem 3.1 is almost exactly 2. We fine-tune the two models only for a single epoch and report the results in Table 3. Again, we note that our evaluation is stronger than the original one (although this is on a different number of data points) which effects in particular the l∞-robustness. Note that the initial l∞-model is completely non-robust for l1 but achieves 24.0% l1-robust accuracy and also the l2-robust accuracy improves from 17.1% to 41.1% and this at the price of a relatively small loss in l∞-robust and clean accuracy. For the l2-robust model all robust accuracies improve as the original model was trained for = 3. It yields the best multiple-norm robust accuracy of 23.1%. Up to our knowledge no multiple-norm robustness has been reported before for ImageNet and thus these results are an important baseline.
  Additional experiments: Sec. C in the appendix contains further studies about E-AT fine-tuning: we show that fine-tuning a naturally trained model does not provide competitive robustness and leads to low clean accuracy. Moreover, we show the stability of the scheme over random seeds, and that increasing the number of epochs progressively improves the robustness in the union.
  4 FINE-TUNING lp-ROBUST MODELS TO BECOME lq-ROBUST FOR p 6= q
  Motivated by our results for the multiple-norm threat model we study to which extent we can finetune an lp-robust model to a lq-robust model with p 6= q. Again the emphasis is on an extremely short fine-tuning time so that this is much faster than full adversarial training.
  CIFAR-10: We fine-tune for 3 epochs the most l∞-robust model at ∞ = 8255 of Gowal et al. (2020) with adversarial training wrt l2 and l1 with 2 = 0.5 and 1 = 12. Table 4 shows that fine-tuning for l1-robustness yields 70.2% l1-robust accuracy which is 9.9% more than the previously most robust model. Also we get a strikingly high l2-robust accuracy for the l2-fine-tuned model of 80.0% not far away from the 81.7% which ones gets by training for l2 from scratch. Surprisingly, fine-tuning the l2-robust model of Gowal et al. (2020) wrt l1 does not outperform the l1-robustness achieved by fine-tuning their l∞-robust model. Interestingly, fine-tuning the l1-robust PreAct ResNet-18 for l2 yields a better l2-robustness than l2-training from scratch, see Table 5. Again this shows that finetuning of existing models is minimal effort and already provides strong baselines for adversarial robustness obtained by adversarial training from scratch and in some cases even outperforms them.
  ImageNet: We fine-tune the l2- and l∞-robust models from Engstrom et al. (2019) to the other threat model respectively and wrt l1. The results are in Table 4. The resulting classifiers are both more robust than the originally trained models from Engstrom et al. (2019): we get a l2-robust accuracy of 41.5% which is 0.6% higher than the l2-model and a l∞-robust accuracy of 31.5% which is 1.7% better than the original l∞-model. For both models it is possible to achieve within one epoch of fine-tuning a non-trivial robustness in l1. In particular, notice that the model for l∞ initially has 0.0% robustness wrt l1, while after fine-tuning it achieves 27.4% robust accuracy which is similar to what obtained by fine-tuning the l2-robust model for l1 which yields 28.7%. Note that when we fine-tuned for multiple-norm robustness we got a l1-robust accuracy of 21.7% resp. 23.1% (Table 3). Thus by fine-tuning specifically for l1-robustness one gains 5.7% resp. 5.6% l1-robust accuracy. Up to our knowledge our ImageNet models are the first ones for which l1-robustness is reported.
sections.heading: 5 CONCLUSION
sections.text: Based on the geometry of the lp-balls we have introduced E-AT, a novel training scheme for multiple norm robustness which achieves comparable adversarial robustness in the union while being significantly faster. We also show for the first time that fine-tuning can be used to transfer adversarial robustness from a single lp-threat model to the multiple norms one, and that one can even obtain an lq-robust classifier with a quick fine-tuning of an lp-robust one with p 6= q. This yields strong baselines for future research. We have in this way generated models with SOTA performance for multiple-norm and l1-robustness on CIFAR-10 and the first models on ImageNet which show significant multiple-norm as well as l1-robustness.
sections.heading: ETHICS STATEMENT
sections.text: There are no conflicts of interest in this work. As we consider the efficient training and fine-tuning of adversarially robust models, we think that this work contributes to more trustworthy AI systems and thus has rather positive implications.
sections.heading: REPRODUCIBILITY STATEMENT
sections.text: We provide information about training in Sec. 2.1 and Sec. 4 of the main part, and add further details in Sec. A of the appendix. We provide in the supplementary material an implementation of our method in PyTorch. Note that most of the models used for fine-tuning are available in RobustBench, and the evaluation code is part of AutoAttack. We report for most of the experiments the mean and standard deviations over different random seeds, as well as the runtime and infrastructure used.
sections.heading: A EXPERIMENTAL DETAILS
sections.text: For the comparison of training schemes we use for multiple-norm robustness we train PreAct ResNet-18 (He et al., 2016) with softplus activation function for 80 epochs with initial learning rate of 0.05 reduced by a factor of 10 after 70 epochs. When training WideResNet-28-10 (Zagoruyko & Komodakis, 2016) we use a cyclic schedule for the learning rate with maximum value 0.1 for 30 epochs. We use SGD optimizer with momentum of 0.9 and weight decay of 5 · 10−4, batch size of 128. We use random cropping and horizontal flipping as augmentation techniques. For adversarial training of models robust wrt a single norm and with SAT and our novel scheme E-AT we use APGD with default parameters, while for the retrained AVG, MAX, and MSD we use PGD for l∞ (step size ∞/4) and l2 (step size /3), SLIDE (Tramèr & Boneh, 2019) for l1 (standard parameters). For all methods we use 10 steps for the inner maximization problem in adversarial training (note that AVG and MAX repeat the attack for all threat models, and MSD tests multiple steps, thus they are more expensive). For all schemes we select the best performing checkpoint for the comparison when using the piecewise schedule, the final checkpoint with the cyclic schedule. Moreover, we use the TRADES-XENT loss (TRADES loss (Zhang et al., 2019) with adversarial points maximizing the cross-entropy loss) since Gowal et al. (2020) show that this gives slightly better robustness on CIFAR-10 without additional data, while we use standard adversarial training (Madry et al., 2018) for PreAct ResNet-18. We train the classifiers of MNG-AC (Madaan et al., 2021) with the original code, where we set 1 = 12 and rescale the step size linearly. Finally, for the runtime comparison we run each method on a single Tesla V100 GPU.
  For fine-tuning on CIFAR-10 we use 3 epochs and the same setup as for full training except for the learning rate schedule, since in this case we use as initial value the best performing one in {0.01, 0.05} (the larger value works best for the smaller networks) and reduce it by a factor of 10 at the beginning of each epoch. When the model was originally trained with extra data beyond the training set on CIFAR-10, we use the 500k images introduced by Carmon et al. (2019) as additional data for fine-tuning, and each batch is splitted equally between standard and extra images, and we count 1 epoch when the whole standard training set has been used: note that in this way, using only 3 epochs not the whole pseudo-labelled dataset is exploited.
  For fine-tuning on ImageNet we use 1 epoch, initial learning rate of 0.01, reduced by a factor of 10 every 1/3 of training steps. We follow the setup of Engstrom et al. (2019) for data augmentation and setting batch size to 256 and weight decay to 10−4. For adversarial training we use APGD with 5 steps for l∞ and l2, 15 steps for l1 since optimizing in the l1-ball intersected with the box constraints is more challenging, see Croce & Hein (2021).
sections.heading: B ADDITIONAL ANALYSIS, EVALUATION AND EXPERIMENTS FOR E-AT
sections.text: We here analyze in more details our E-AT scheme and expand the comparison to existing methods presented above.
  B.1 ROBUSTNESS WRT l2 OF E-AT
  To show the effect of E-AT on l2-robustness we plot in Fig. 3 the robust accuracy wrt l2 computed with FAB (Croce & Hein, 2020a), which minimizes the size of the perturbations, as a function of the threshold 2 for a PreAct ResNet-18 trained with l2-AT at 2 = 0.5 and one using E-AT (see complete results for such models in Sec. B.3). Theorem 3.1 suggests that the extreme norms training provides robustness at 2 ≈ 0.62, which is confirmed by the plots. Although no l2-attack has been used during training by the E-AT model, it has robustness wrt l2 similar to that of the classifier specifically trained for such threat model.
sections.heading: B.2 CONFIRMATION THAT ADVERSARIAL TRAINING WITH THE EXTREME NORMS IS SUFFICIENT: ANALYSIS OF MAX AND MSD TRAINING
sections.text: Both MAX and MSD schemes perform adversarial training considering all the threat models simultaneously, and we analyze here their training procedure in more detail. Fig. 4 shows for the MAX strategy how many times, in percentage, for each epoch the points computed for each threat model realize the maximal loss over the three attacks (l1, l2, l∞), and then are subsequently used for the
  update of the model. Similarly, for MSD, we show the frequency with which a step wrt each lp-norm is taken when computing the adversarial points (average over all iterations and training points). In both cases the l∞-threat model is the most used one with the l1-threat model being used 3 − 4 times less often. However, the l2-threat model is almost never chosen. This empirically confirms the analysis from Theorem 3.1 which shows that training only wrt l∞ and l1 is (at least for a linear classifier) sufficient to achieve l2-robustness for the chosen 2 and thus during training no extra updates wrt the l2-threat model are necessary. This is in line with the results reported in Fig. 3 which show that for different thresholds the l2-robustness achieved by E-AT-training is similar to that of standard l2-training (for generating Fig. 3 we use FAB attack (Croce & Hein, 2020a) to compute robust accuracy at varying 2).
sections.heading: B.3 E-AT FROM RANDOM INITIALIZATION WITH DIFFERENT ARCHITECTURE
sections.text: We repeat the experiment from the main part about training for multiple norms robustness from random initialization with a smaller architecture like PreAct ResNet-18 (He et al., 2016). MAXtraining yields the most robust model but E-AT is only 1.6% worse outperforming SAT and AVG but note that the standard deviation is rather high. Since it has the same architecture we additionally include the original MSD model of Maini et al. (2020), marked with (*), which obtains 41.4% robustness in the union, whereas with our reimplementation we get 43.9%, improving their results significantly. Note that they reported in their paper 47.0% robustness in the union, while our APGD-
  based evaluation reduces this to 41.4% which shows that our robustness evaluation is significantly stronger.
sections.heading: B.4 ADDITIONAL STATISTICS
sections.heading: WideResNet-28-10
sections.heading: PreAct ResNet-18
sections.text: Table 6 repeats the results from Table 1 and Table 5 with additionally the average robust accuracy over the three threat models (last column), with standard deviation over 3 random seeds. On WRN-
  28-10, in this metric E-AT achieves the best results, and in particular significantly outperforms MAX while having the same clean accuracy. For the smaller RN-18, E-AT unif. attains the highest average robustness, with E-AT and SAT being slightly worse.
sections.heading: C FINE-TUNING WITH E-AT FOR MULTIPLE NORMS ROBUSTNESS
sections.heading: C.1 FINE-TUNING ROBUST AND NATURAL MODELS
sections.text: We fine-tune for 3 epochs PreAct ResNet-18 either naturally trained or robust wrt a single lp-norm. Table 7 shows clean and robust accuracy for each threat model for the initial classifier and after E-AT fine-tuning: while for all robust models the fine-tuning yields values competitive with the full training for multiple norms (see Table 5), starting from a standard model leads to significantly lower both clean performance and robustness in the union of the three lp-balls.
sections.heading: C.2 RUNTIME WITH LARGE MODELS
sections.text: We reported in Table 1 and Table 5 the runtime per epoch of E-AT. For larger architectures the computational cost increases significantly, and adversarial training with the WideResNet-70-16, the largest one we consider, on CIFAR-10 takes, in our experiments, around 6100 s per epoch when using only the training set and over 10000 s if the unlabelled data is used (since twice more training steps are effectively used). This shows how transfering robustness with fine-tuning might allow to obtain classifiers robust wrt different threat models fast and at much lower computational cost.
sections.heading: C.3 RESULTS OVER RANDOM SEEDS AND EFFECT OF BIASED SAMPLING SCHEME
sections.text: We study the effect of randomness in the training process when fine-tuning for robustness wrt multiple norms using E-AT. Moreover, we compare the biased sampling scheme introduced in Eq. (5) to a uniform one when to choose which lp, with p ∈ {1,∞}, attack to use for adversarial training for each batch. In Table 7 we report the average results and corresponding standard deviation over 5 runs with different seeds on CIFAR-10 models originally robust wrt a single norm. For all metrics and starting models different runs show similar performance (the largest standard deviation is 0.96%) showing the stability of the scheme. Note that in the previous experiments we used the same seed for all runs and methods, without selecting the best one in a pool. Moreover, the biased sampling schemes yields slightly better results when fine-tuning the l2- and in particular the l1-robust model: we hypothesize that, since l∞ is the most challenging threat model, it is important to use it more often at training time when the initial model is non robust wrt l∞.
sections.heading: C.4 RESULTS USING MORE EPOCHS
sections.text: Table 9 shows the effect of our E-AT-fine-tuning for different numbers of epochs on either a RN-18 robust wrt l∞ or naturally trained. For the robust model, with longer training the clean accuracy progressively improves, as well as the robustness in the union of the threat models. In particular, the models fine-tuned for 15 epochs has robust accuracy similar to that achieved by the MAX-training (43.2% compared to 43.3%, see Table 5), while still being significantly faster even considering the training time of the initial model. When starting from a standard model, E-AT fine-tuning leads to a large drop in clean accuracy while the robustness in the union remains lower than what can be achieved with robust models, even when using 15 epochs.
  Additionally we fine-tune for 3 epochs, instead of 1 as done above, the l2-robust model on ImageNet from Engstrom et al. (2019) with our E-AT. Table 10 shows that the longer fine-tuning improves all the performance metrics between 0.6% and 1.3%.
sections.heading: C.5 RESULTS ON CIFAR-10 USING A SINGLE EPOCH
sections.text: Since we use a single epoch of fine-tuning on ImageNet, we here test its effect on CIFAR-10. In Table 11 we fine-tune with E-AT models adversarially trained wrt a single norm for 1 epoch: this is sufficient to significantly increase the robustness in the union of the threat models, which gets close to that obtained with the standard 3 epochs (differences are in the range 0.6% to 2.5%). In particular, the l∞-robust classifier is again the most suitable for the fine-tuning, since it has been trained in the most challenging threat model.
sections.heading: C.6 FINE-TUNING WITH OTHER METHODS
sections.text: We here explore the option of fine-tuning robust models with techniques other than our E-AT, and report the results in Table 12. In particular, we test SAT, MAX and MSD for fine-tuning the RN-18 robust to single norms as done in Table 8 (the results of E-AT unif. and E-AT are taken from there).
  First, one sees that SAT performs significantly worse than the other methods in this scenario, for all the fine-tuned classifiers. Second, we observe that E-AT achieves very similar results to MAX and MSD, especially when using the l∞-robust classifier which yields the highest robustness in the union. However, we highlight that MAX and MSD are 3x and 2x more expensive than E-AT (see Tables 1 and 5). Thus, we include in Table 12 E-AT with twice the budget, that is 6 epochs, which has comparable cost to MSD and closes the small gap (on average MAX and E-AT with 6 epochs perform equal and are slightly better than MSD). Since one of our goals is to reduce the cost of getting multiple-norm robust classifiers, we use E-AT as the main tool for fine-tuning.
sections.heading: C.7 FINE-TUNING PERCEPTUALLY ROBUST MODELS
sections.text: We test the effect of E-AT fine-tuning on a model trained to be robust to perturbations which are aligned with human perception. In particular, we use the classifier obtained with perceptual adversarial training (PAT), i.e. wrt the LPIPS metric, from Laidlaw et al. (2021), and compare it to two models with the same architecture (ResNet-50) adversarially trained wrt l∞ and l2. Table 13 shows the robustness in every threat model for the original models and those obtained with 3 epochs of E-AT fine-tuning. The PAT classifier has initially the highest robustness in the union, confirming the obersevation of Laidlaw et al. (2021) that PAT provides some robustness to unseen attacks. After fine-tuning, all three models achieve similar worst-case robustness, with the classifier originally l∞robust being slightly better. This shows that our E-AT fine-tuning is effective even when applied to models adversarially trained not wrt an lp-norm.
  D ROBUSTNESS AGAINST UNSEEN NON lp-BOUNDED ATTACKS
  We here want to study to which extent the robustness achieved with multiple norms training generalizes to unseen and possibly very different threat models on CIFAR-10. We select three sparse attacks (l0-bounded, patches and frames) and five adversarial corruptions (fog, snow, Gabor noise, elastic,
  l∞-JPEG) from Kang et al. (2019b). Additionally, we compute the accuracy of the classifiers on the common corruptions, which are not adversarially optimized, of CIFAR-10-C (Hendrycks & Dietterich, 2019). Table 14 shows the results against such attacks of WRN-28-10 adversarially trained wrt single norms and with E-AT, and the PAT model from Laidlaw et al. (2021), which uses RN-50 as architecture. We add also a naturally trained model as further baseline. The model trained wrt multiple norms shows much higher robustness in both the union and average of these attacks, almost 2x higher in union than the best of the other models. Moreover, for almost all threat models, the E-AT model attains the highest robustness or very close, while the best among the single lp-robust models varies: this means that training for multiple norms robustness allows to summarize different kinds of robustness in the same classifier. Finally, E-AT outperforms even the PAT model which is trained wrt LPIPS and aims at generalization to unseen attacks.
  Experimental details: To test robustness against l0-attacks we use Sparse-RS (Croce et al., 2020b) with a budget of 18 pixels and 10k queries. We adopt patches of size 5 × 5 pixels, optimized with the PGD-based from Rao et al. (2020) (without constraints on the position of the patch on the images), and frames of width 1 pixel (Zajac et al., 2019), again optimized with PGD: in both cases we use 10 random restarts of 100 iterations. For the adversarial corruptions we use the original implementation (Kang et al., 2019b) with 100 iterations and search for a budget for which the models show different levels of robustness (in details, for fog = 128, snow = 0.5, Gabor noise = 60, elastic = 0.125, l∞-JPEG = 0.25). Finally, we average the classification accuracy over the 5 severities of the common corruptions. All the statistics are on 1000 test points.
sections.heading: E EXPERIMENTS ON MNIST
sections.text: We further test the different techniques on the MNIST dataset. We use the same CNN of Maini et al. (2020) as architecture and ∞ = 0.3, 2 = 2 and 1 = 10 as threshold at which evaluating robustness, as done by Maini et al. (2020). We note that while it is an easier dataset, MNIST is challenging when it comes to adversarial training since it presents unexpected phenomena: e.g. Tramèr & Boneh (2019) noted that l∞-adversarial training induces gradient obfuscation when using
  attack wrt l2 and l1, and both Tramèr & Boneh (2019); Maini et al. (2020) had to use many PGDsteps (up to 100), and Tramèr & Boneh (2019) even a ramp-up schedule for the during training. While other modifications to the training setup might be beneficial for some or all the methods, we just increased the number of APGD-steps to 50 for l1 (see more details below). In Table 15 we compare E-AT to SAT, AVG, MAX and MSD (for the last three we use the models provided by Maini et al. (2020)). First, E-AT outperforms the available classifiers trained with AVG, MAX and MSD, meaning that even on MNIST it is a strong baseline. However, in this case, SAT, which trains on all types of perturbations, achieves better results than E-AT on average: E-AT has higher variance over runs but the best run (over multiple seeds) is close to the best one of SAT in terms of robustness in the union (55.3% vs 54.4%). Interestingly, SAT has much higher robustness wrt l2 compared to E-AT, but this is somehow expected since Eq. (3) would “predict” robustness for E-AT at 2 ≈ 1.7 while 2 = 2 is used for testing, and this is precise only for linear models. Thus the slightly worse performance of E-AT compared to SAT for the chosen radii of the threat models is to be expected from our geometric analysis.
  Moreover, since we have shown that fine-tuning an lp-robust model with E-AT yields high multiple norms robustness, and given that E-AT from random initialization is weak mostly wrt l2, we finetune the l2-AT classifier with E-AT. This, with just 3 or 5 epochs, significantly outperforms SAT (up to +2.3% robustness in the union), while preserving l2-robustness. In total, we improve the previous SOTA for multiple-norm robustness for MNIST from 48.7% (MSD) to 57.5% (E-AT fine-tuning of an l2-robust model with 5 epochs) which is a significant improvement. Note that in this case we increase the radii ∞ and 1 to 0.33 and 14 respectively to preserve the l2-robustness: in fact, with such values Eq. (3) yields 2 ≈ 2.16. However, training from random initialization, in the standard setup, with the larger thresholds leads to worse robustness in the union. We hypothesize that this is due to the increased difficulty of the task to learn: it is known that even single norm adversarial training is problematic when increasing the value of (Ding et al., 2020).
  Experimental details: For training we use 30 epochs with cyclic learning rate (maximum value 0.05, also used for fine-tuning) and no data augmentation (other settings as for CIFAR-10). As mentioned, we use in adversarial training for multiple norms (SAT, E-AT unif. and E-AT) 50 steps of APGD for l1, and to reduce the training cost we decrease to 5 those for l∞. Moreover, for training, in l1-APGD we increase the parameter to control the initial sparsity of the updates to 0.1 (default is 0.05). For evaluation, we use the full AutoAttack, since on MNIST FAB (Croce & Hein, 2020a) and Square Attack (Andriushchenko et al., 2020) are at times stronger than PGD-based attacks, as shown in the original papers.
title: THREAT MODELS AT THE PRICE OF ONE AND HOW TO QUICKLY FINE-TUNE ROBUST MODELS TO ANOTHER THREAT MODEL