summaries: The paper extends current adversarial learning approaches beyond imperceptible L_p norm perturbations. The proposed approach can handle many models of natural variation, such as a change in brightness. The main idea behind the approach is to use unsupervised approaches such as GANs to model the natural variation. Given this model of natural variation, the paper replaces the adversarial learning objective of finding the worst example in an L_p norm ball around a data point to finding the worst example based on the model of natural variation. This is expensive, so the paper also proposes more computationally efficient approaches based on data augmentation. The experimental results demonstrate that the proposed approach performs well on a variety of tasks. This paper proposes a model-based framework for improving the robustness of image classifiers to average-case corruptions of varying severity. The proposed framework can be thought of as adversarial training where the perturbation is replaced by a function that transforms the image according to a specific corruption. A nuisance parameter controls the instantiation and severity of the corruption that is applied to the input. The paper compares baselines to different versions of this general model-based framework with experiments on several datasets. This paper proposes a “paradigm shift” for augmenting datasets when training CNN-based image classifiers. On one side, traditional augmentations include blur, Gaussian noise, color distortions. On the other side, methods like adversarial training consider augmentations under norm bounds in the image space. The proposed method, instead, uses models of natural variation to augment the images. Increased out-of-domain accuracy is shown on ImageNet-C and several slices of the CURE-TSR dataset. 
abstractText: While deep learning (DL) has resulted in major breakthroughs in many applications, the frameworks commonly used in DL remain fragile to seemingly innocuous changes in the data. In response, adversarial training has emerged as a principled approach for improving the robustness of DL against norm-bounded perturbations. Despite this progress, DL is also known to be fragile to unbounded shifts in the data distribution due to many forms of natural variation, including changes in weather or lighting in images. However, there are remarkably few techniques that can address robustness to natural, out-of-distribution shifts in the data distribution in a general context. To address this gap, we propose a paradigm shift from perturbation-based adversarial robustness to model-based robust deep learning. Critical to our paradigm is to obtain models of natural variation, which vary data over a range of natural conditions. Then by exploiting these models, we develop three novel model-based robust training algorithms that improve the robustness of DL with respect to natural variation. Our extensive experiments show that across a variety of natural conditions in twelve distinct datasets, classifiers trained with our algorithms significantly outperform classifiers trained via ERM, adversarial training, and domain adaptation techniques. Specifically, when training on ImageNet and testing on various subsets of ImageNet-c, our algorithms improve over baseline methods by up to 30 percentage points in top-1 accuracy. Further, we show that our methods provide robustness (1) against natural, out-ofdistribution data, (2) against multiple simultaneous distributional shifts, and (3) to domains entirely unseen during training.
authors.name: IZING TO
references.authors: Amjad Almahairi Sai Rajeswar Alessandro Sordoni Philip Bachman Aaron Courville
references.title: Augmented cyclegan: Learning many-to-many mappings from unpaired data
references.venue: arXiv preprint arXiv:1802.10151,
references.authors: Vinicius F Arruda Thiago M Paixão Rodrigo F Berriel Alberto F De Souza Claudine Badue Nicu Sebe Thiago Oliveira-Santos
references.title: Cross-domain car detection using unsupervised image-toimage translation: From day to night
references.venue: In 2019 International Joint Conference on Neural Networks (IJCNN),
references.authors: Anish Athalye Nicholas Carlini David Wagner
references.title: Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples
references.venue: arXiv preprint arXiv:1802.00420,
references.authors: Alesia Chernikova Alina Oprea Cristina Nita-Rotaru BaekGyu Kim
references.title: Are self-driving cars secure? evasion attacks against deep neural networks for steering angle prediction
references.venue: IEEE Security and Privacy Workshops (SPW),
references.authors: Yunjey Choi Minje Choi Munyoung Kim Jung-Woo Ha Sunghun Kim Jaegul Choo
references.title: Stargan: Unified generative adversarial networks for multi-domain image-to-image translation
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Yunjey Choi Youngjung Uh Jaejun Yoo Jung-Woo Ha
references.title: Stargan v2: Diverse image synthesis for multiple domains
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Tarin Clanuwat Mikel Bober-Irizar Asanobu Kitamoto Alex Lamb Kazuaki Yamamoto David Ha
references.title: Deep learning for classical japanese literature
references.venue: arXiv preprint arXiv:1812.01718,
references.authors: Gregory Cohen Saeed Afshar Jonathan Tapson Andre Van Schaik
references.title: Emnist: Extending mnist to handwritten letters
references.venue: In 2017 International Joint Conference on Neural Networks (IJCNN),
references.authors: Jia Deng Wei Dong Richard Socher Li-Jia Li Kai Li Li Fei-Fei
references.title: Imagenet: A large-scale hierarchical image database
references.venue: IEEE conference on computer vision and pattern recognition,
references.authors: Josip Djolonga Jessica Yung Michael Tschannen Rob Romijnders Lucas Beyer Alexander Kolesnikov Joan Puigcerver Matthias Minderer Alexander D’Amour Dan Moldovan
references.title: On robustness and transferability of convolutional neural networks
references.venue: arXiv preprint arXiv:2007.08558,
references.authors: Edgar Dobriban Hamed Hassani David Hong Alexander Robey
references.title: Provable tradeoffs in adversarially robust classification
references.venue: arXiv preprint arXiv:2006.05161,
references.authors: Tommaso Dreossi Alexandre Donzé Sanjit A Seshia
references.title: Compositional falsification of cyberphysical systems with machine learning components
references.venue: Journal of Automated Reasoning,
references.authors: Isaac Dunn Tom Melham Daniel Kroening
references.title: Generating realistic unrestricted adversarial inputs using dual-objective gan training
references.venue: arXiv preprint arXiv:1905.02463,
references.authors: Logan Engstrom Brandon Tran Dimitris Tsipras Ludwig Schmidt Aleksander Madry
references.title: Exploring the landscape of spatial robustness
references.venue: arXiv preprint arXiv:1712.02779,
references.authors: Kevin Eykholt Ivan Evtimov Earlence Fernandes Bo Li Amir Rahmati Chaowei Xiao Atul Prakash Tadayoshi Kohno Dawn Song
references.title: Robust physical-world attacks on deep learning visual classification
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Mahyar Fazlyab Manfred Morari George J Pappas
references.title: Safety verification and robustness analysis of neural networks via quadratic constraints and semidefinite programming
references.venue: arXiv preprint arXiv:1903.01287,
references.authors: Mahyar Fazlyab Alexander Robey Hamed Hassani Manfred Morari George Pappas
references.title: Efficient and accurate estimation of lipschitz constants for deep neural networks
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Yaroslav Ganin Victor Lempitsky
references.title: Unsupervised domain adaptation by backpropagation
references.venue: In International conference on machine learning,
references.authors: Yaroslav Ganin Evgeniya Ustinova Hana Ajakan Pascal Germain Hugo Larochelle François Laviolette Mario Marchand Victor Lempitsky
references.title: Domain-adversarial training of neural networks
references.venue: The Journal of Machine Learning Research,
references.authors: Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: In Advances in neural information processing systems,
references.authors: Ian J Goodfellow Jonathon Shlens Christian Szegedy
references.title: Explaining and harnessing adversarial examples
references.venue: arXiv preprint arXiv:1412.6572,
references.authors: Sven Gowal Chongli Qin Po-Sen Huang Taylan Cemgil Krishnamurthy Dvijotham Timothy Mann Pushmeet Kohli
references.title: Achieving robustness in the wild via adversarial mixing with disentangled representations
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Dan Hendrycks Thomas Dietterich
references.title: Benchmarking neural network robustness to common corruptions and perturbations
references.venue: arXiv preprint arXiv:1903.12261,
references.authors: Dan Hendrycks Norman Mu Ekin D Cubuk Barret Zoph Justin Gilmer Balaji Lakshminarayanan
references.title: Augmix: A simple data processing method to improve robustness and uncertainty
references.venue: arXiv preprint arXiv:1912.02781,
references.authors: Dan Hendrycks Kevin Zhao Steven Basart Jacob Steinhardt Dawn Song
references.title: Natural adversarial examples
references.venue: arXiv preprint arXiv:1907.07174,
references.authors: Dan Hendrycks Steven Basart Norman Mu Saurav Kadavath Frank Wang Evan Dorundo Rahul Desai Tyler Zhu Samyak Parajuli Mike Guo
references.title: The many faces of robustness: A critical analysis of out-of-distribution generalization
references.venue: arXiv preprint arXiv:2006.16241,
references.authors: Hossein Hosseini Radha Poovendran
references.title: Semantic adversarial examples
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops,
references.authors: Xun Huang Ming-Yu Liu Serge Belongie Jan Kautz
references.title: Multimodal unsupervised image-toimage translation
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Jonathan J. Hull
references.title: A database for handwritten text recognition research
references.venue: IEEE Transactions on pattern analysis and machine intelligence,
references.authors: Ajil Jalal Andrew Ilyas Constantinos Daskalakis Alexandros G Dimakis
references.title: The robust manifold defense: Adversarial training using generative models
references.venue: arXiv preprint arXiv:1712.09196,
references.authors: Sandesh Kamath Amit Deshpande KV Subrahmanyam
references.title: Invariance vs. robustness of neural networks
references.venue: arXiv preprint arXiv:2002.11318,
references.authors: Nikolaos Karianakis Jingming Dong Stefano Soatto
references.title: An empirical evaluation of current convolutional architectures’ ability to manage nuisance location and scale variability
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: arXiv preprint arXiv:1412.6980,
references.authors: Jack Klys Jake Snell Richard Zemel
references.title: Learning latent subspaces in variational autoencoders
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Alexey Kurakin Ian Goodfellow Samy Bengio
references.title: Adversarial examples in the physical world
references.venue: arXiv preprint arXiv:1607.02533,
references.authors: Yann LeCun Corinna Cortes CJ Burges
references.title: Mnist handwritten digit database
references.venue: ATT Labs [Online]. Available: http://yann. lecun. com/exdb/mnist,
references.authors: Hsin-Ying Lee Hung-Yu Tseng Jia-Bin Huang Maneesh Singh Ming-Hsuan Yang
references.title: Diverse image-to-image translation via disentangled representations
references.venue: In Proceedings of the European conference on computer vision (ECCV),
references.authors: Hyeungill Lee Sungyeob Han Jungwoo Lee
references.title: Generative adversarial trainer: Defense to adversarial perturbations with gan
references.venue: arXiv preprint arXiv:1705.03387,
references.authors: Dan Li Dacheng Chen Baihong Jin Lei Shi Jonathan Goh See-Kiong Ng
references.title: Mad-gan: Multivariate anomaly detection for time series data with generative adversarial networks
references.venue: In International Conference on Artificial Neural Networks,
references.authors: Ming-Yu Liu Thomas Breuel Jan Kautz
references.title: Unsupervised image-to-image translation networks
references.venue: In Advances in neural information processing systems,
references.authors: Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras Adrian Vladu
references.title: Towards deep learning models resistant to adversarial attacks
references.venue: arXiv preprint arXiv:1706.06083,
references.authors: Qi Mao Hsin-Ying Lee Hung-Yu Tseng Siwei Ma Ming-Hsuan Yang
references.title: Mode seeking generative adversarial networks for diverse image synthesis
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Marco Melis Ambra Demontis Battista Biggio Gavin Brown Giorgio Fumera Fabio Roli
references.title: Is deep learning safe for robot vision? adversarial examples against the icub humanoid
references.venue: In Proceedings of the IEEE International Conference on Computer Vision Workshops,
references.authors: Yuval Netzer Tao Wang Adam Coates Alessandro Bissacco Bo Wu Andrew Y Ng
references.title: Reading digits in natural images with unsupervised feature learning
references.venue: NIPS Workshop on Deep Learning and Unsupervised Feature Learning,
references.authors: Konstantinos Papangelou Konstantinos Sechidis James Weatherall Gavin Brown
references.title: Toward an understanding of adversarial examples in clinical trials
references.venue: In Joint European Conference on Machine Learning and Knowledge Discovery in Databases,
references.authors: Kexin Pei Yinzhi Cao Junfeng Yang Suman Jana
references.title: Deepxplore: Automated whitebox testing of deep learning systems
references.venue: In proceedings of the 26th Symposium on Operating Systems Principles,
references.authors: Aditi Raghunathan Jacob Steinhardt Percy Liang
references.title: Certified defenses against adversarial examples
references.venue: arXiv preprint arXiv:1801.09344,
references.authors: Pouya Samangouei Maya Kabkab Rama Chellappa
references.title: Defense-gan: Protecting classifiers against adversarial attacks using generative models
references.venue: arXiv preprint arXiv:1805.06605,
references.authors: Yang Song Rui Shu Nate Kushman Stefano Ermon
references.title: Constructing unrestricted adversarial examples with generative models
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Johannes Stallkamp Marc Schlipsing Jan Salmen Christian Igel
references.title: The German Traffic Sign Recognition Benchmark: A multi-class classification competition
references.venue: In IEEE International Joint Conference on Neural Networks,
references.authors: Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna Dumitru Erhan Ian Goodfellow Rob Fergus
references.title: Intriguing properties of neural networks
references.venue: arXiv preprint arXiv:1312.6199,
references.authors: Rohan Taori Achal Dave Vaishaal Shankar Nicholas Carlini Benjamin Recht Ludwig Schmidt
references.title: Measuring robustness to natural distribution shifts in image classification
references.venue: arXiv preprint arXiv:2007.00644,
references.authors: Dogancan Temel Min-Hung Chen Ghassan AlRegib
references.title: Traffic sign detection under challenging conditions: A deeper look into performance variations and spectral characteristics
references.venue: IEEE Transactions on Intelligent Transportation Systems,
references.authors: Eric Tzeng Judy Hoffman Kate Saenko Trevor Darrell
references.title: Adversarial discriminative domain adaptation
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Simon Vandenhende Bert De Brabandere Davy Neven Luc Van Gool
references.title: A three-player gan: generating hard samples to improve classification networks
references.venue: In 2019 16th International Conference on Machine Vision Applications (MVA),
references.authors: Huaxia Wang Chun-Nam Yu
references.title: A direct approach to robust deep learning using adversarial networks
references.venue: arXiv preprint arXiv:1905.09591,
references.authors: Mei Wang Weihong Deng
references.title: Deep visual domain adaptation: A survey
references.authors: Ze Wang Xiuyuan Cheng Guillermo Sapiro Qiang Qiu
references.title: Stochastic conditional generative networks with basis decomposition
references.venue: arXiv preprint arXiv:1909.11286,
references.authors: Eric Wong J Zico Kolter
references.title: Provable Defenses Against Adversarial Examples Via the Convex Outer Adversarial Polytope
references.venue: arXiv preprint arXiv:1711.00851,
references.authors: Eric Wong J Zico Kolter
references.title: Learning perturbation sets for robust machine learning
references.venue: arXiv preprint arXiv:2007.08450,
references.authors: Chaowei Xiao Bo Li Jun-Yan Zhu Warren He Mingyan Liu Dawn Song
references.title: Generating adversarial examples with adversarial networks
references.venue: arXiv preprint arXiv:1801.02610,
references.authors: Chaowei Xiao Jun-Yan Zhu Bo Li Warren He Mingyan Liu Dawn Song
references.title: Spatially transformed adversarial examples
references.venue: arXiv preprint arXiv:1801.02612,
references.authors: Han Xiao Kashif Rasul Roland Vollgraf
references.title: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms
references.venue: arXiv preprint arXiv:1708.07747,
references.authors: Chhavi Yadav Léon Bottou
references.title: Cold case: The lost mnist digits
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Zili Yi Hao Zhang Ping Tan Minglun Gong
references.title: Dualgan: Unsupervised dual learning for imageto-image translation
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Matthew D Zeiler
references.title: Adadelta: an adaptive learning rate method
references.venue: arXiv preprint arXiv:1212.5701,
references.authors: Jun-Yan Zhu Taesung Park Phillip Isola Alexei A Efros
references.title: Unpaired image-to-image translation using cycle-consistent adversarial networks
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Jun-Yan Zhu Richard Zhang Deepak Pathak Trevor Darrell Alexei A Efros Oliver Wang Eli Shechtman
references.title: Toward multimodal image-to-image translation
references.venue: In Advances in neural information processing systems,
sections.heading: 1 INTRODUCTION
sections.text: The last decade has seen remarkable progress in deep learning (DL), which has prompted widescale integration of DL frameworks into myriad application domains (LeCun et al., 2015). In many of these applications, and in particular in safety-critical domains, it is essential that the DL systems are robust and trustworthy (Dreossi et al., 2019). However, it is now well-known that DL is fragile to seemingly innocuous changes to the input data (Szegedy et al., 2013). Indeed, well-documented examples of fragility to carefully-designed noise can be found in a variety of contexts, including image classification (Madry et al., 2017), clinical trials (Papangelou et al., 2018), and robotics (Melis et al., 2017). Accordingly, a number of adversarial training algorithms (Goodfellow et al., 2014b; Wong & Kolter, 2017) as well as certifiable defenses (Raghunathan et al., 2018; Fazlyab et al., 2019a) have recently been proposed, which have provided a rigorous framework for improving the robustness of DL against norm-bounded perturbations (Fazlyab et al., 2019b; Dobriban et al., 2020).
  Despite this encouraging progress, very recent papers have unanimously shown that DL is also fragile to unbounded shifts in the data-distribution due to a wide range of natural phenomena (Djolonga et al., 2020; Taori et al., 2020; Hendrycks et al., 2020; Hendrycks & Dietterich, 2019). For example, in image classification, such shifts include changes due to lighting, blurring, or weather conditions (Pei et al., 2017; Chernikova et al., 2019). However, there are remarkably few general, principled techniques that have been shown to provide robustness against these forms of out-of-distribution, natural variation (Hendrycks et al., 2019a). Furthermore, as these unseen distributional shifts are arguably more common in safety-critical domains, the task of designing algorithms that generalize to natural, out-of-distribution data is an important and novel challenge for the DL community.
  In this paper, we propose a paradigm shift from perturbation-based adversarial robustness to modelbased robust deep learning. Our goal is to provide principled, general algorithms that can be used to train neural networks to be robust against natural, out-of-distribution shifts in data. Our experiments show that across a variety of challenging, naturally-occurring conditions, such as variation in lighting, haze, rain, and snow, and across various datasets, including SVHN, GTSRB, CURE-TSR, and ImageNet, classifiers trained with our model-based algorithms significantly outperform standard DL baselines, adversarially-trained classifiers, and, when applicable, domain adaptation methods.
  Contributions. The contributions of this paper can be summarized as follows: • Paradigm shift. We propose a paradigm shift from perturbation-based robustness to model-based
  robust deep learning, where models of natural variation express changes due to natural conditions. • Optimization-based formulation. We formulate a novel model-based robust training problem by
  constructing a general robust optimization procedure to search for challenging natural variation. • Models of natural variation. For many challenging forms of natural variation, we use deep
  generative models to learn models of natural variation that are consistent with realistic conditions. • Novel algorithms. We propose a family of novel robust training algorithms that exploit models
  of natural variation to improve the robustness of DL against worst-case natural variation. • Out-of-distribution robustness. We show that our algorithms are the first to consistently provide
  robustness against natural, out-of-distribution shifts that frequently occur in real-world environments, including snow, rain, fog, and brightness on SVHN, GTSRB, CURE-TSR, and ImageNet.
  • ImageNet-c robustness. We show that our algorithms can significantly improve the robustness of classifiers trained on ImageNet and tested on ImageNet-c by as much as 30 percentage points.
  • Robustness to simultaneous distributional shifts. We show that our methods are composable and can improve robustness to multiple simultaneous sources of natural variation. To evaluate this feature, we curate several new datasets, each of which has two simultaneous distributional shifts.
  • Robustness to unseen domains. We show that models of natural variation can be reused on datasets that are entirely unseen during training to improve out-of-distribution generalization.
sections.heading: 2 PERTURBATION-BASED ROBUSTNESS: APPROACHES AND LIMITATIONS
sections.text: In this paper, we consider a standard classification task in which training data (x, y) ⇠ D is distributed according to a joint distribution D over instances x 2 Rd and labels y 2 [k] := {0, 1, . . . , k}. In this setting, given a finite training sample drawn i.i.d. from D, the goal of the learning problem is to obtain a classifier fw parameterized by weights w 2 Rp such that fw can correctly predict the labels y corresponding to new instances x drawn i.i.d. from D. In practice, one can learn fw by approximately solving the non-convex empirical risk-minimization (ERM) problem argminw E[`(x, y;w)] where ` is a suitable loss-function. However, neural networks trained using ERM are known to be susceptible to adversarial attacks. This means that given a datum x with a corresponding label y, one can find another datum xadv such that x is close to xadv in a given Euclidean norm and xadv is predicted by the learned classifier as belonging to a different class c 6= y. If such a datum xadv exists, it is called an adversarial example. This is illustrated in Figure 1a; although these pandas look identical, they were classified differently in (Goodfellow et al., 2014b).
  (a) Models take the form G(x, ), where is a nuisance parameter that describes how the output image x0 := G(x, ) is varied. (b) Input image x and corresponding generated images for a learned model of natural variation on ImageNet.
  Figure 2: In this paper, we introduce models of natural variation to describe natural transformations.
  The dominant paradigm toward improving robustness against adversarial examples relies on a robust optimization perspective wherein neural networks are trained to correctly classify worst-case perturbations of data (Madry et al., 2017; Wong & Kolter, 2017). This can be formulated as follows:
  argmin w E(x,y)⇠D h max 2 `(x+ , y;w) i
  (1)
  We can think of (1) as comprising two coupled optimization problems: an inner maximization problem in which we seek a challenging perturbation and an outer minimization problem in which we seek weights that lead to strong classification performance.
  Limitations of perturbation-based robustness. Despite remarkable progress toward improving the robustness of DL against norm-bounded perturbations, there are significant limitations to adversarial training. Notably, DL is known to be fragile to many forms of natural variation, which cannot be described by small perturbations x 7! x+ . In image classification, such natural variation includes changes in weather or background color (Eykholt et al., 2018; Hendrycks et al., 2019b; Hosseini & Poovendran, 2018), spatial transformations such as rotation or scaling (Xiao et al., 2018b; Karianakis et al., 2016), and sensor-based attacks (Kurakin et al., 2016). Because such transformations frequently arise in safety-critical domains, it is critically important for the DL community to develop algorithms that are robust against out-of-distribution, natural variation in data. In this paper, we specifically address this challenge by proposing a principled, optimization-based framework which can be used in general settings to provide robustness against arbitrary sources of natural variation.
sections.heading: 3 A NEW ROBUSTNESS PARADIGM: MODEL-BASED ROBUST DEEP LEARNING
sections.text: Underlying the task of improving the robustness of neural networks against natural, out-ofdistribution data are two fundamental challenges. Firstly, unlike in the adversarial robustness community, in real-world, safety-critical environments, data can vary in unknown and highly nonlinear ways. Thus, the first step toward building a robust training procedure must be to design a mechanism that accurately describes how data varies in such environments. Next, assuming a suitable model of natural variation, the second challenge is to formulate a training procedure that exploits this model toward improving robustness. In this section, we present novel solutions to each of these challenges.
sections.heading: 3.1 MODELS OF NATURAL VARIATION
sections.text: In order to effectively model sources of natural variation in a domain-agnostic setting, we will abstractly define models of natural variation. Concretely, a model of natural variation G(x, ) is a map that describes how an input datum x can be naturally varied by a nuisance parameter resulting in a new datum x0 := G(x, ). Ideally, for a fixed datum x, varying the nuisance parameter should vary the severity of the natural conditions in the generated datum x0. An example of such a model is shown in Figure 2, where an image x on the left (in this case, in sunny weather) can be naturally varied by and consequently transformed into the image x0 on the right (in snowy weather). In the remainder of this subsection, we consider cases in which (1) a model G is known a priori, and (2) a model G is unknown and therefore must be learned offline from data. In this second case in which models of natural variation must be learned, we propose a method for obtaining these models.
  Known models of natural variation. In many problems, a model G(x, ) is known a priori due to intrinsic geometric structure. For example, there is underlying structure that describes how data can
  be rotated, translated, or scaled; models for rotating an image can be characterized by G(x, ) = R( )x where R( ) is a rotation matrix and 2 := [0, 2⇡). In prior work, this idea has been used to train classifiers to be robust to rotation and scaling (Engstrom et al., 2017; Kamath et al., 2020).
  Learning models of natural variation from data. In many situations, models natural variation are not known a priori or are too costly to obtain. For example, consider Figure 2 in which a model G(x, ) takes an image x of a street sign in sunny weather and maps it to an image x0 := G(x, ) in snowy weather. Even though there is a relationship between the two images, obtaining a model G relating these two domains is extremely challenging if we resort to geometric structure. For such problems we advocate for learning the model G from data. To do so, we assume that we have access to two unpaired domains A and B that are drawn from a common distribution. Domain A contains the original data, such as the images with sunny weather, and domain B contains naturally transformed data, such as images with snow. Ideally, a model of natural variation should learn to map images from domain A to corresponding images with different levels of natural variation captured by the images of domain B. In our experiments section, we rely on the MUNIT framework (Huang et al., 2018), which combines two autoencoders and two generative adversarial networks (Goodfellow et al., 2014a), to learn mappings between domains A and B. Furthermore, we note that many choices unpaired, unconditional image-to-image translation networks satisfy our criteria for G, and in future work we plan to investigate the efficacy of these architectures. In Appendix A, we describe parallel experiments that we carried out with two other architectural choices for G, and we fully characterize the MUNIT architecture used in our experiments.
sections.heading: 3.2 MODEL-BASED ROBUST TRAINING FORMULATION
sections.text: The model-based robust training paradigm that we propose retains the basic elements of adversarial training described in Section 2. Our point of departure from the classical adversarial training formulation is in the choice of the so-called adversarial perturbation. In this paper, we assume that data can be transformed according to a model of natural variation G(x, ) by choosing different values of from a given nuisance space . The goal of the model-based approach is to train a classifier that achieves high accuracy both on a test set drawn i.i.d. from D and on more-challenging test data that has been subjected to the source of natural variation that G models. This perspective can be captured by the following robust optimization problem:
  min w E(x,y)⇠D h max 2 `(G(x, ), y;w) i . (2)
  In the inner maximization problem of this formulation, given an instance-label pair (x, y), we seek a vector ⇤ 2 that produces a corresponding instance x0 := G(x, ⇤) which gives rise to high loss values `(G(x, ⇤), y;w) under the current weight w. One can think of this vector ⇤ as characterizing the worst-case nuisance that can be generated by the model G(x, ⇤) for the original instance x. After solving this inner problem, we solve the outer minimization problem by finding weights w that minimize the risk against the challenging instance G(x, ⇤). By training the network to correctly classify this worst-case data, the goal is to become invariant to the model G(x, ) for any 2 .
sections.heading: 4 MODEL-BASED TRAINING ALGORITHMS
sections.text: We now assume that we have access to a suitable model of natural variation G(x, ) and shift our attention toward exploiting G in the development of novel robust training algorithms. In the empirical version of (2), rather than assuming access to the full joint distribution D, we assume that we are given given a set of i.i.d. samples Dn := {(xj , yj)}nj=1 drawn from D. Thus we have:
  w? 2 argmin w2Rp
  nX
  j=1
  h max 2 ` (G (xj , ) , yj ;w) i . (3)
  Note that when w parameterizes a neural network, (3) is a nonconvex-nonconcave min-max problem, which is difficult to solve exactly. Thus, we resort to approximate optimization techniques for solving (3). Specifically, we propose three algorithmic variants: Model-based Adversarial Training (MAT), Model-based Robust Training (MRT), and Model-based Data Augmentation (MDA). Pseudocode for MAT is given in Algorithm 1; pseudocode for MRT and MDA as well as further discussion of these algorithms is given in Appendix B. At a high level, each of these methods alternates between solving the outer minimization problem and the inner maximization problem. To this
  Algorithm 1 Model-based Adversarial Training (MAT) Input: Data sample Dn = {(xj , yj)}nj=1, number of steps k, step size ↵ Output: Learned weight w
  1: repeat 2: for minibatch Bm := {(x1, y1), (x2, y2), . . . , (xm, ym)} ⇢ Dn do 3: Initialize := ( 1, 2, . . . , m) (0q, 0q, . . . , 0q) 4: for k steps do 5: g r Pm j=1 `(G(xj , j), yj ;w) 6: ⇧ [ + ↵g] # ⇧ denotes projection onto the set 7: end for 8: g rw Pm j=1[`(G(xj , j), yj ;w) + · `(xj , yj ;w)]
  9: w Update(g, w) # Update can be SGD, Adam, Adadelta, etc. 10: end for 11: until convergence
  end, each of these algorithms uses SGD to solve the outer problem; the methods differ in how they seek solutions to the inner problem, and in what follows, we describe each of these procedures in more detail. In each algorithm, given a datum (x, y), the solution ? to the inner problem is used to create a new datum (G(x, ?), y) that is added to the training set before solving the outer problem.
  Model-based Adversarial Training. In MAT, we seek an exact solution to the inner problem by performing k steps of gradient ascent in on the objective `(G(x, ), y;w). The resulting nuisance parameter ? is one that causes `(G(x, ?), y;w) to have high loss under the current weight w.
  Model-based Robust Training. In MRT, we first randomly sample i 2 for i 2 [k]. We then select the i? 2 [k] such that `(G(x, i?), y;w) is maximized. In this way, rather than exactly solving the inner problem, MRT uses a sampling-based approach to finding challenging data G(x, i?).
  Model-based Data Augmentation. In MDA, rather than explicitly trying to solve the inner problem, we seek a diversity of naturally-varying data rather than the “worst-case.” In this way, MDA samples i 2 for i 2 [k] and then appends {G(x, i), y}ki=1 to the training dataset.
sections.heading: 5 EXPERIMENTS
sections.text: We present experiments in five different settings over twelve distinct datasets to demonstrate the broad applicability of MBRDL. First, in Sections 5.1-5.2, we show that our algorithms are the first to consistently provide out-of-distribution robustness across a range of challenging corruptions, including shifts in brightness, contrast, snow, fog, frost, and haze on CURE-TSR, ImageNet, and ImageNet-c. In Section 5.3, we curate several new datasets containing simultaneous sources of natural variation, and we then show that models of natural variation can be composed to provide robustness against these simultaneous shifts. In Section 5.4, we show that models of natural variation trained on a fixed dataset can be reused to provide robustness on datasets entirely unseen during training. Finally, in Section 5.5, we assume access to unlabeled data corresponding to a fixed domain shift, and we compare our algorithms to suitable baselines, including domain adaptation methods. Throughout these experiments, we use the notation “source (A!B)” to denote a distributional shift from domain A to domain B. For example, “contrast (low!high)” will denote a shift from low-contrast to high-contrast. Images from domains A and B for each of the shifts used in this paper are available in Appendix A. We note that our experiments contain domains with both natural and artificially-generated variation; details concerning how we extracted non-artificial variation can be found in Appendix D. Architecture and hyperparameter details are given in Appendix C.
sections.heading: 5.1 OUT-OF-DISTRIBUTION ROBUSTNESS
sections.text: In many applications, one might have data corresponding to low levels of natural variation, such as a dusting of snow in images of street signs. However, it is often difficult to collect data corresponding to high levels of natural variation, such as images taken during a blizzard. In such cases, we show that our algorithms can be used to provide significant out-of-distribution robustness against data with high levels of natural variation by training on data with relatively low levels of the same source of
  natural variation. To do so, we use data from the CURE-TSR dataset (Temel et al., 2019), which contains images of street signs divided into subsets according to various sources of natural variation and corresponding severity levels. For example, for images in the “snow” subset, level 0 corresponds to no snow, whereas level 5 corresponds to a full blizzard. Thus, for each row of Table 1, we use unlabeled data from levels 0 and 2 to learn a model of natural variation corresponding to a given source of natural variation in CURE-TSR. We then train classifiers using MDA with labeled level 0 data. We also train classifiers using ERM and PGD using the labeled data from levels 0 and 2. We then test all classifiers on data from levels 3, 4, and 5. Note that while this is an unfair comparison for our methods, given that the model-based algorithms are not given access to labeled level 2 data, our algorithms still outperform the baselines by as much as 20 percentage points on level 5 data.
sections.heading: 5.2 MODEL-BASED ROBUSTNESS ON THE SHIFT FROM IMAGENET TO IMAGENET-C
sections.text: To demonstrate the scalability of our approach, we perform experiments on ImageNet (Deng et al., 2009) and the recently-curated ImageNet-c dataset (Hendrycks & Dietterich, 2019). ImageNet-c contains images from the ImageNet test set that are corrupted according to artificial transformations, such as snow, rain, and fog, and are labeled from 1-5 depending on the severity of the corruption. For numerous challenging corruptions, we train models to map from the classes 0-9 of ImageNet to the corresponding classes of ImageNet-c. We then train all networks on classes 10-59 of ImageNet, and test on the corresponding classes for various subsets of ImageNet-c. Note that in this setting, the ImageNet classes used to train the model of natural variation are disjoint from those that are used to train the classifier, so many techniques, including most domain adaptation methods, do not apply; to offer a point of comparison, we include the accuracies of classifiers trained using AugMix, which is a recently proposed method that adds known transformations to the data (Hendrycks et al., 2019a).
sections.heading: 5.3 ROBUSTNESS TO SIMULTANEOUS DISTRIBUTIONAL SHIFTS
sections.text: In practice, it is common to encounter multiple simultaneous distributional shifts. For example, in image classification, there may be shifts in both brightness and contrast; yet while there may be examples corresponding to shifts in either brightness or contrast in the training data, there may not be any examples of both shifts occurring simultaneously. To address this robustness challenge,
  for each row of Table 3, we learn two models of natural variation G1 and G2 using unlabeled training data corresponding to two separate shifts, which map domains A1!B1 (e.g. low- to highbrightness) and A2!B2 (e.g. low- to high-contrast). We then compose these models to form a new model G(x, ) = G1(G2(x, ), ) which can be used to provide robustness against both shifts simultaneously. We then train classifiers on labeled data from A1\A2 and test on data from B1\B2. To create the data from B1 \ B2 for the ImageNet experiments, we apply pairs of transformations that were originally used to create the ImageNet-c datasets; more details are in Appendix D.
sections.heading: 5.4 TRANSFERABILITY OF MODEL-BASED ROBUSTNESS
sections.text: Because we learn models of natural variation offline before training a classifier, our paradigm can be applied to domains that are entirely unseen while training the model. In particular, we show that models can be reused on similar yet unseen datasets to provide robustness against a common source of natural variation. For example, one might have access to two domains corresponding to the shift from images of European street signs taken during the day to images taken at night. However, one might wish to provide robustness against the same shift from daytime to nighttime on a new dataset of American street signs without access to any nighttime images in this new dataset. Whereas many techniques, including most domain adaptation methods, do not apply in this scenario, in the MBRDL paradigm, we can simply learn a model corresponding to the changes in lighting for the European street signs and then apply this model to the dataset of the American signs. Table 4 shows several experiments of this stripe in which a model G is learned on one dataset D1 and then applied on another D2; we improve robustness on unseen domains by up to 40 percentage points.
sections.heading: 5.5 MODEL-BASED ROBUST DEEP LEARNING FOR UNSUPERVISED DOMAIN ADAPTATION
sections.text: While our approach does not require labeled data from domain B, when such data is available, it is of interest to evaluate how our approach compares to relevant methods such as domain adaptation. In Table 5, for each shift from domain A to B, we assume access to labeled data from domain A
  and unlabeled data from domain B. In each row, we use unlabeled data from both domains to train a model of natural variation. We then train classifiers using our algorithms, as well with ERM and PGD, using data from domain A and test on data from the test set for domain B. Furthermore, we compare to ADDA, which is a well-known domain adaptation method (Tzeng et al., 2017). In every scenario, our model-based algorithms significantly outperform the baselines, often by 10-20 percentage points. Note that while this is one of the most commonly studied settings in domain adaptation, it represents only one particular setting to which the MBRDL paradigm can be applied.
sections.heading: 6 RELATED WORK
sections.text: Aside from the algorithms we introduced in Section 4, we are not aware of any other algorithms that can be used to address out-of-distribution robustness across the diverse array of tasks presented in the previous section. However, several lines of research have sought to address this problem in constrained settings or under highly restrictive assumptions. In the domain adaptation literature, various methods have been proposed which rely on the restrictive assumption that unlabeled data corresponding to a fixed distributional shift is available during training (Tzeng et al., 2017; Ajakan et al., 2014; Ganin & Lempitsky, 2015). Unlike these approaches, our solution does not assume access to unlabeled data from a fixed shift and can be applied to datasets that are entirely unseen during training. Furthermore, several works have used generative models to create adversarial perturbations (Xiao et al., 2018a; Lee et al., 2017; Wang & Yu, 2019; Samangouei et al., 2018; Jalal et al., 2017) or perceptually-realistic images subject to relatively simple corruptions in specific application domains (Dunn et al., 2019; Song et al., 2018; Vandenhende et al., 2019; Arruda et al., 2019). On the other hand, our approach is broadly applicable to arbitrary and challenging sources of natural variation.
  Two concurrent works formulate robust training procedures assuming that data is corrupted according to a fixed generative architecture. Gowal et al. (2020) exploit properties specific to the StyleGAN architecture to formulate a training algorithm that provides robustness against color-based shifts on MNIST and CelebA. In our work, we propose a more general framework and three novel robust training algorithms that can exploit any suitable generative network, and we show improvements on more challenging, naturally-occurring shifts across twelve distinct datasets. Wong & Kolter (2020) use conditional VAEs to learn perturbation sets corresponding to simple corruptions from pairs of images. In our framework we improve robustness against more challenging, natural shifts by learning from unpaired datasets and we do not rely on class-conditioning to generate realistic images.
sections.heading: 7 CONCLUSION
sections.text: In this paper, we proposed a novel model-based robust training paradigm for deep learning that provides robustness with respect to models of natural variation. Our notion of robustness offers a departure from adversarial training with respect to norm-bounded data perturbations. In our experiments, we show that our paradigm can provide significant out-of-distribution robustness on many challenging distributional shifts. Furthermore, our paradigm can provide robustness against multiple simultaneous distribution shifts and on domains that are entirely unseen while training the model, and shows significant out-of-distribution robustness as datasets become more challenging.
title: MODEL-BASED ROBUST DEEP LEARNING: GENERAL-