summaries: This work proposes and studies two techniques to generate unadversarial images (images which cause a computer vision deep learning model to more reliably give accurate predictions). One technique is for creating unadversarial patches which can be overlaid on testing images to increase the robustness of a pre-trained classifier, the other is for creating unadversarial textures for 3D meshes which when applied to 3D meshes. The work provides strong empirical evidence for unadversarial patches on ImageNet and CIFAR, including results where images are corrupted after applying the patches. Evidence is provided for unadversarial textures on simulated data and a small scale real world experiment.

summaries: In this work, the authors leverage techniques from the adversarial examples literature to design patches and textures that aid in classifying an object *correctly.* In a fixed-model setting, they construct patches / textures using gradient-based methods and demonstrate effectiveness on clean CIFAR-10 and ImageNet and various robustness benchmarks such as ImageNet-C (common perturbations). The authors also compare their patches / textures to simple baselines such as QR codes, smaller-sized images from the training set, and predefined fixed patterns.

summaries: This work introduces and studies "unadversarial" input modifications (patches or object textures) designed to *increase* classifier accuracy, particularly in corrupted images. The work is inspired by research on adversarial examples (in particular "adversarial patches" which trick a classifier into predicting a particular class when the patch is present in the scene, regardless of what other class might be present).  Given access to a known, fixed, pre-trained network, a small patch (or the entire object texture) are optimized by gradient descent to increase prediction accuracy. When applied, the method is shown to be quite effective for increasing classification accuracy (and performance on a simple regression task) under a variety of simulated image corruptions (blur, fog, etc.). In addition, the patch method is shown to significantly aid classification even when printed and affixed to real physical objects.

summaries: This paper exploits the  known-sensitivity of deep models to perturbation in the input data in order to improve (rather than decrease) the performance of a pre-trained model in a classification task. Specifically, the authors propose to alter the input data (i.e. images)  by 1) adding a patch to the image 2) altering the texture of specific objects. The paper shows experimentally that this improves both in-domain performance and robustness to data corruptions.

summaries: The paper considers a scenario where a fixed computer vision system (such as CNN based classification or regression models) is given and one tries to adapt the design of objects in order to increase their chance of being correctly detected by the system. This setting is similar to the design of marker systems for computer vision with the difference that the detection algorithm is considered fixed.  Two approaches are presented to design (i) patches that can be printed and sticked onto objects and (ii) complete textures that cover the surface of 3D objects. Both approaches follow the design of adversarial perturbations with the difference that instead of maximizing the loss of the system, it is now minimized to improve its performance instead of weakening it.  Experiments evaluate how these approaches can improve the overall performance of the given system as well as its robustness against a variety of different corruptions. Experiments are performed on synthetic 2D and 3D data, as well as on real objects, where the designed patches are printed out and attached to physical objects. Overall, they demonstrate consistent and significant improvements over non-modified objects and simple baselines.

abstractText: We study a class of computer vision settings wherein one can modify the design of the objects being recognized. We develop a framework that leverages this capability—and deep networks’ unusual sensitivity to input perturbations—to design “robust objects,” i.e., objects that are explicitly optimized to be confidently classified. Our framework yields improved performance on standard benchmarks, a simulated robotics environment, and physical-world experiments.1

authors:
Syed Muhammad Abbas
Salman Aslam
Karsten Berns
Abubakr Muhammad

title: Analysis and Improvements in AprilTag Based State Estimation

authors:
Anish Athalye
Logan Engstrom
Andrew Ilyas
Kevin Kwok

title: Synthesizing Robust Adversarial Examples

venue: In: International Conference on Machine Learning (ICML)

authors:
Michael A Alcorn
Qi Li
Zhitao Gong
Chengfei Wang
Long Mai
Wei-Shinn Ku
Anh Nguyen

title: Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects

venue: In: Conference on Computer Vision and Pattern Recognition (CVPR)

authors:
Anurag Arnab
Ondrej Miksik
Philip HS Torr

title: On the robustness of semantic segmentation models to adversarial attacks

authors:
Battista Biggio
Igino Corona
Davide Maiorca
Blaine Nelson
Nedim Šrndić
Pavel Laskov
Giorgio Giacinto
Fabio Roli

title: Evasion attacks against machine learning at test time”. In: Joint European conference on machine learning and knowledge discovery in databases (ECML-KDD)

authors:
Adith Boloor
Xin He
Christopher Gill
Yevgeniy Vorobeychik
Xuan Zhang

title: Simple physical adversarial examples against end-to-end autonomous driving models

venue: IEEE International Conference on Embedded Software and Systems (ICESS)

authors:
Tom B. Brown
Dandelion Mané
Aurko Roy
Martín Abadi
Justin Gilmer

title: Adversarial Patch. 2018

authors:
Pin-Yu Chen
Huan Zhang
Yash Sharma
Jinfeng Yi
Cho-Jui Hsieh

title: Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models

venue: Workshop on Artificial Intelligence and Security

authors:
Kevin Eykholt
Ivan Evtimov
Earlence Fernandes
Bo Li
Amir Rahmati
Florian Tramer
Atul Prakash
Tadayoshi Kohno
Dawn Song

title: Physical Adversarial Examples for Object Detectors

authors:
Gamaleldin Elsayed
Ian Goodfellow
Jascha Sohl-Dickstein

title: Adversarial Reprogramming of Neural Networks

venue: In: International Conference on Learning Representations (ICLR)

authors:
Logan Engstrom
Andrew Ilyas
Shibani Santurkar
Dimitris Tsipras
Jacob Steinhardt
Aleksander Madry

title: Identifying Statistical Bias in Dataset Replication

authors:
Logan Engstrom
Brandon Tran
Dimitris Tsipras
Ludwig Schmidt
Aleksander Madry

title: Exploring the Landscape of Spatial Robustness

authors:
Alhussein Fawzi
Pascal Frossard

title: Manitest: Are classifiers really invariant?

venue: British Machine Vision Conference (BMVC)

title: ARTag, a fiducial marker system using digital techniques

venue: IEEE Computer Society Conference on Computer Vision and Pattern Recognition

authors:
Davide Falanga
Alessio Zanchettin
Alessandro Simovic
Jeffrey Delmerico
Davide Scaramuzza

title: Vision-based autonomous quadrotor landing on a moving platform

venue: IEEE International Symposium on Safety, Security and Rescue Robotics (SSRR)

authors:
Sergio Garrido-Jurado
Rafael Muñoz-Salinas
Francisco José Madrid-Cuevas
Manuel Jesús

title: Marin-Jiménez. “Automatic generation and detection of highly reliable fiducial markers under occlusion

venue: Pattern Recognition

authors:
Robert Geirhos
Patricia Rubisch
Claudio Michaelis
Matthias Bethge
Felix A. Wichmann
Wieland Brendel

title: ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.

authors:
Dan Hendrycks
Steven Basart
Norman Mu
Saurav Kadavath
Frank Wang
Evan Dorundo
Rahul Desai
Tyler Zhu
Samyak Parajuli
Mike Guo
Dawn Song
Jacob Steinhardt
Justin Gilmer

title: The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

authors:
Dan Hendrycks
Thomas G. Dietterich

title: Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations

authors:
Andrew Ilyas
Logan Engstrom
Anish Athalye
Jessy Lin

title: Black-box Adversarial Attacks with Limited Queries and Information

authors:
Andrew Ilyas
Logan Engstrom
Aleksander Madry

title: Prior Convictions: Blackbox Adversarial Attacks with Bandits and Priors

authors:
Andrew Ilyas
Shibani Santurkar
Dimitris Tsipras
Logan Engstrom
Brandon Tran
Aleksander Madry

title: Adversarial Examples Are Not Bugs, They Are Features

venue: Neural Information Processing Systems (NeurIPS)

authors:
Jorn-Henrik Jacobsen
Jens Behrmann
Richard Zemel
Matthias Bethge

title: Excessive Invariance Causes Adversarial Vulnerability

venue: In: International Contemporary on Learning Representations

authors:
Saumya Jetley
Nicholas Lord
Philip Torr

title: With friends like these, who needs adversaries?

venue: Advances in Neural Information Processing Systems (NeurIPS)

authors:
Alexey Kurakin
Ian J. Goodfellow
Samy Bengio

title: Adversarial Machine Learning at Scale

authors:
Can Kanbak
Seyed-Mohsen Moosavi-Dezfooli
Pascal Frossard

title: Geometric robustness of deep networks: analysis and improvement

venue: In: Conference on Computer Vision and Pattern Recognition

title: Learning Multiple Layers of Features from Tiny Images

venue: Technical report

authors:
Daniel Kang
Yi Sun
Dan Hendrycks
Tom Brown
Jacob Steinhardt

title: Testing Robustness Against Unforeseen Adversaries

authors:
Mark Lee
Zico Kolter

title: On physical adversarial patches for object detection

venue: arXiv preprint arXiv:1906.11897

authors:
Sven Lange
Niko Sunderhauf
Peter Protzel

title: A vision based onboard approach for landing and position control of an autonomous multirotor UAV in GPS-denied environments

venue: International Conference on Advanced Robotics

authors:
Xuancen Liu
Shifeng Zhang
Jiayi Tian
Longbin Liu

title: An Onboard VisionBased System for Autonomous Landing of a Low-Cost Quadrotor on a Novel Landing Pad

authors:
Marco Melis
Ambra Demontis
Battista Biggio
Gavin Brown
Giorgio Fumera
Fabio Roli

title: Is deep learning safe for robot vision? adversarial examples against the icub humanoid

venue: Proceedings of the IEEE International Conference on Computer Vision Workshops

authors:
Aleksander Madry
Aleksandar Makelov
Ludwig Schmidt
Dimitris Tsipras
Adrian Vladu

title: Towards deep learning models resistant to adversarial attacks

authors:
Phong Ha Nguyen
Muhammad Arsalan
Ja Hyung Koo
Rizwan Ali Naqvi
Noi Quang Truong
Kang Ryoung Park

title: LightDenseYOLO: A fast and accurate marker tracker for autonomous UAV landing by visible light camera sensor on drone

venue: Sensors

authors:
Paarth Neekhara
Shehzeen Hussain
Shlomo Dubnov
Farinaz Koushanfar

title: Adversarial Reprogramming of Text Classification Neural Networks

venue: Empirical Methods in Natural Language Processing (EMNLP)

authors:
Merlin Nimier-David
Delio Vicini
Tizian Zeltner
Wenzel Jakob

title: Mitsuba 2: A Retargetable Forward and Inverse Renderer

venue: In: Transactions on Graphics (Proceedings of SIGGRAPH Asia)

title: AprilTag: A robust and flexible visual fiducial system

venue: IEEE International Conference on Robotics and Automation

authors:
Nicolas Papernot
Patrick McDaniel
Ian Goodfellow
Somesh Jha
Z Berkay Celik
Ananthram Swami

title: Practical black-box attacks against machine learning

venue: Asia Conference on Computer and Communications

authors:
Nicolas Papernot
Patrick McDaniel
Ian Goodfellow

title: Transferability in Machine Learning: from Phenomena to Black-box Attacks using Adversarial Samples

venue: ArXiv preprint arXiv:1605.07277

authors:
Olga Russakovsky
Jia Deng
Hao Su
Jonathan Krause
Sanjeev Satheesh
Sean Ma
Zhiheng Huang
Andrej Karpathy
Aditya Khosla
Michael Bernstein
Alexander C. Berg
Li Fei-Fei

title: ImageNet Large Scale Visual Recognition Challenge

venue: In: International Journal of Computer Vision (IJCV)

authors:
Francisco J Romero-Ramire
Rafael Muñoz-Salinas
Rafael Medina-Carnicer

title: Fractal Markers: a new approach for long-range marker pose estimation under occlusion

venue: IEEE Access

authors:
Benjamin Recht
Rebecca Roelofs
Ludwig Schmidt
Vaishaal Shankar

title: Do ImageNet Classifiers Generalize to ImageNet?

authors:
Aditi Raghunathan
Sang Michael Xie
Fanny Yang
John C Duchi
Percy Liang

title: Adversarial Training Can Hurt Generalization

authors:
Mahmood Sharif
Sruti Bhagavatula
Lujo Bauer
Michael K. Reiter

title: Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition

venue: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications

authors:
Chawin Sitawarin
Arjun Nitin Bhagoji
Arsalan Mosenia
Mung Chiang
Prateek Mittal

title: Darts: Deceiving autonomous cars with toxic signs

authors:
Hadi Salman
Andrew Ilyas
Logan Engstrom
Ashish Kapoor
Aleksander Madry

title: Do Adversarially Robust ImageNet Models Transfer Better?

authors:
Srikanth Saripalli
James F Montgomery
Gaurav S Sukhatme

title: Visually guided landing of an unmanned aerial vehicle

venue: IEEE transactions on robotics and automation

authors:
Dong Su
Huan Zhang
Hongge Chen
Jinfeng Yi
Pin-Yu Chen
Yupeng Gao

title: Is Robustness the Cost of Accuracy? A Comprehensive Study on the Robustness of 18 Deep Image Classification Models

venue: European Conference on Computer Vision (ECCV)

authors:
Christian Szegedy
Wojciech Zaremba
Ilya Sutskever
Joan Bruna
Dumitru Erhan
Ian Goodfellow
Rob Fergus

title: Intriguing properties of neural networks

authors:
Rohan Taori
Achal Dave
Vaishaal Shankar
Nicholas Carlini
Benjamin Recht
Ludwig Schmidt

title: Measuring Robustness to Natural Distribution Shifts in Image Classification

authors:
Antonio Torralba
Alexei A Efros

title: Unbiased look at dataset bias

authors:
Noi Quang Truong
Phong Ha Nguyen
Se Hyun Nam
Kang Ryoung Park

title: Deep learning-based super-resolution reconstruction and marker detection for drone landing

authors:
Dimitris Tsipras
Shibani Santurkar
Logan Engstrom
Alexander Turner
Aleksander Madry

title: Robustness May Be at Odds with Accuracy

authors:
Simen Thys
Wiebe Van Ranst
Toon Goedemé

title: Fooling automated surveillance cameras: adversarial patches to attack person detection

venue: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops

authors:
John Wang
Edwin Olson

title: AprilTag 2: Efficient and robust fiducial detection

venue: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

authors:
D. Wang
D. Tseng
P. Li
Y. Jiang
M. Guo
M. Danielczuk
J. Mahler
J. Ichnowski
K. Goldberg

title: Adversarial Grasp Objects

venue: IEEE 15th International Conference on Automation Science and Engineering (CASE)

authors:
Xiaojun Xu
Xinyun Chen
Chang Liu
Anna Rohrbach
Trevor Darrell
Dawn Song

title: Fooling vision and language models despite localization and attention mechanism

authors:
Cihang Xie
Jianyu Wang
Zhishuai Zhang
Yuyin Zhou
Lingxi Xie
Alan Yuille

title: Adversarial examples for semantic segmentation and object detection

venue: Proceedings of the IEEE International Conference on Computer Vision

authors:
Shaowu Yang
Sebastian A Scherer
Konstantin Schauwecker
Andreas Zell

title: Autonomous landing of MAVs on an arbitrarily textured landing site using onboard monocular vision

venue: Journal of Intelligent & Robotic Systems

heading: 1 Introduction

text:
Performing reliably on unseen or shifting data distributions is a difficult challenge for modern computer vision systems. For example, slight rotations and translations of images suffice to reduce the accuracy of state-of-the-art classifiers [ETT+19; ALG+19; KMF18]. Similarly, models that attain near human-level performance on benchmarks exhibit significantly degraded performance when faced with even mild image corruptions and transformations [HD19; KSH+19]. In fact, when an adversary is allowed to modify inputs directly, standard vision models can be manipulated into predicting arbitrary outputs (cf. adversarial examples [BCM+13; SZS+14]). While robustness interventions and additional training data can improve out-of-distribution behavior, they do not fully close the gap between model performance on standard heldout data and on corrupted/otherwise unfamiliar data [TDS+20; HBM+20]. The situation is worse still when test-time distribution is underor mis-specified, which occurs commonly in practice.
How can we change this state of affairs? We propose a new approach to image recognition in the face of unforeseen corruptions or distribution shifts. This approach is rooted in a reconsideration of the problem setup itself. Specifically, we observe that in many situations, a system designer actually controls, to some extent, the inputs that are fed into that model. For example, a drone operator seeking to train a landing pad detector can modify the surface of the landing pad; and, a roboticist training a perception model to recognize a small set of custom objects can slightly alter the texture or design of these objects.
We find that such control over inputs can be leveraged to drastically improve our ability to tackle computer vision tasks. In particular, it allows us to turn the input-sensitivity of modern vision systems from a weakness into a strength. Instead of optimizing inputs to mislead models (e.g., as in adversarial examples), we can alter inputs to reinforce correct behavior, yielding what we refer to as “unadversarial examples.” Indeed, we show that even a simple gradient-based algorithm can
∗Equal contribution. †Work partially completed while at Microsoft Research. 1Our code is available at https://github.com/microsoft/unadversarial.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
successfully construct unadversarial examples in a variety of vision settings and demonstrate that, by optimizing objects for vision systems (rather than vice-versa), we can significantly improve both in-distribution performance and robustness to unforeseen data shifts and corruptions.
We demonstrate the efficacy of our methods on both standard benchmarks (CIFAR, ImageNet) and robustness-based benchmarks (ImageNet-C, CIFAR-C) while also comparing them to a broad set of baselines (e.g., QR codes or heuristically designed patches). To further highlight the practicality of our framework, we (a) extend our methods to designing the texture of three-dimensional objects (rather than patches); (b) deploy unadversarial examples in a simulated drone setting; and (c) ensure that the performance improvement yielded by the objects we design actually transfer to the physical world.

heading: 2 Motivation and approach

text:
While vision models tend to perform well on held-out data drawn from the same distribution as the training data, out-of-distribution inputs can severely degrade this performance. For example, models behave unreliably under distribution shifts induced by new data collection procedures [RRS+19; EIS+20; TE11], synthetic corruptions [HD19; KSH+19], spatial transformations [ETT+19; FF15], as well as under other types of shift.
Given a fixed type of distribution shift, a standard approach to increasing model robustness is to explicitly train on or regularize with data from the corresponding anticipated test distribution [KSH+19]. For example, Engstrom et al. [ETT+19] find that vision models trained on worst-case rotations and translations end up being fairly robust to rotation and translation-based distribution shifts. However, this approach is not without shortcomings—for example, Kang et al. [KSH+19] find that training CIFAR classification models that are robust to JPEG-compression in this manner requires a significant sacrifice in natural accuracy. Recent works make similar observations in the context of other distribution shift mechanisms like `p adversaries [TSE+19; SZC+18; RXY+19] or texture swapping [GRM+19].
These observations give rise to a more general question: given that performing reliably in the face of constrained, well-specified distribution shifts is already a difficult challenge, how can we attain robustness to broad, unforeseen distribution shifts?

heading: 2.1 Leveraging more controlled vision settings

text:
Consider the vision tasks of detecting a landing pad from a drone, or classifying manufacturing components from a factory robot. In both these tasks, reliable in-distribution performance is a necessity; still, a number of possible distribution shifts may occur at deployment time. For example, the drone might approach the landing pad at an atypical angle, or have a view obstructed by snow,
smoke, or rain. Similarly, the factory robot may encounter objects in unfamiliar poses, or could be equipped with only a low-quality/noisy camera.
At first glance, dealing with these issues seems to require tackling the difficult problem of general distribution shift robustness discussed earlier in this section. However, there is in fact a critical distinction between the scenarios considered above and vision tasks in their full generality. In particular, in these scenarios and many others, the system designer has control over the physical objects that the model operates on. For instance, the designer of the drone’s landing algorithm could paint the landing pad bright yellow. A machine learning model trained to detect this custom landing pad might then be more effective than a model trained to detect a standard grey pad, especially in lowvisibility conditions. Still, the particular choice to paint the landing pad yellow is rather ad hoc, and likely rooted in the way humans recognize objects. Meanwhile, an abundance of prior work (e.g., [JBZ+19; GRM+19; JLT18; IST+19]) demonstrates that humans and machine learning models tend to use different sets of features to make their decisions. This suggests that rather than relying on human priors, we should instead be asking: how can we build objects that are easily detectable by machine learning models?

heading: 2.2 Unadversarial examples

text:
The task of making inputs less recognizable by computer vision systems has been a focus of research in adversarial examples. Adversarial examples are small, carefully constructed perturbations to natural images that can induce arbitrary (mis)behavior from machine learning models [BCM+13; SZS+14]. These perturbations are typically constructed as the result of an optimization problem that maximizes the loss of a machine learning model with respect to the input, i.e., by solving the optimization problem
δadv = arg max δ∈∆ L(fθ(x+ δ), y), (1)
where fθ is a parameterized model (e.g., a neural network with weights θ); x is a natural input; y is the corresponding correct label; L is the loss function used to train θ (e.g., cross-entropy loss) and ∆ is a class of permissible perturbations (e.g., norm-bounded perturbations: ∆ = {δ : ‖δ‖p ≤ } for some small > 0). Adversarial perturbations are typically crafted via projected gradient descent (PGD) [Nes03] in input space, a standard iterative first-order optimization method—prior work in adversarial examples has shown that even a few iterations of PGD suffice to completely change the prediction of many state-of-the-art machine learning systems [MMS+18].
From adversarial examples to unadversarial objects. The goal of this work is to modify the design of objects so that they are more easily recognizable by computer vision systems. If we could specify every pixel of every image that a model encounters at test time, we could draw on the effectiveness of adversarial examples, and construct image perturbations (using PGD) that minimize the loss of the system, e.g.,
δunadv = arg min δ∈∆ L(θ;x+ δ, y). (2)
In our setting of interest, however, having such fine-grained access to the test inputs is unrealistic (presumably, if we had precise control over every pixel in the input, we could just directly encode the ground-truth label directly in it). Instead, we have limited control over some physical objects; these objects are in turn captured within images, affected by many signals that are out of our control, such as camera artifacts, weather effects, or background scenery.
It turns out that we can still draw on techniques from adversarial examples research in this limitedcontrol setting. Specifically, a recent line of work [KGB17; SBB+16; EEF+18a; AEI+18] concerns itself with constructing robust adversarial examples [AEI+18], i.e., physically realizable objects that act as adversarial examples when introduced into a scene in any one of a variety of ways. For example, Sharif et al. [SBB+16] design glasses frames that cause facial recognition models to misclassify faces, Athalye et al. [AEI+18] design custom-textured 3D models that are misclassified by state-of-the-art ImageNet classifiers from many angles and viewpoints, and [BMR+18] design adversarial patches: stickers that can be placed anywhere on objects causing them to be misclassified. In this paper, we leverage the techniques developed in the above line of work to construct robust un-adversarial objects—physically realizable objects optimized to minimize (rather than maximize) the loss of a target classifier. In the next section, we will more concretely discuss our methods for generating unadversarial objects, then outline our evaluation setup.

heading: 2.3 Constructing unadversarial objects

text:
In the previous section, we identified a class of scenarios where a system designer can, to some extent, control the objects that a machine learning system model operates on. In these settings, we motivated so-called unadversarial examples as a potential way to boost models’ overall performance and robustness to distribution shifts. In this section, we present and illustrate two concrete algorithms for constructing unadversarial examples: unadversarial patches and unadversarial textures. In the former, we design a sticker or “patch” [BMR+18] that can be placed on the object; in the latter, we design the 3D texture of the object (in a similar manner to the texture-based adversarial examples of Athalye et al. [AEI+18]). Example results from both techniques are shown in Figure 2. For simplicity, we will assume that the task being performed is image classification, but the techniques are directly applicable to other tasks as well. In all cases, we require access to a pre-trained model for the dataset of interest.
Unadversarial patches (cf. Algorithm 1 in Appendix A). To train unadversarial patches (cf. Figure 2a), in addition to the pre-trained model, we require sample access to image-label pairs from the dataset of interest. At each iteration, we sample an image-label pair (x, y) from a training set, and place the patch corresponding to class y onto the image with random orientation and position2. Since placing the patch is an affine transformation, after each iteration we can compute the gradient of the model’s loss with respect to the pixels in the patch, and take a negative gradient step on the patch parameters. The algorithm terminates when the model’s loss on sticker-boosted images plateaus, or after a fixed number of iterations.
Unadversarial textures (cf. Algorithm 2 in Appendix A). To train unadversarial textures (cf. Figure 2b), we do not require sample access to the dataset, but instead a set of 3D meshes for each class of objects that we would like to augment, as well as a set of background images that we can use to simulate sampling a scene (these can be images from the dataset of interest, solid-color backgrounds, random patterns, etc.).
For each 3D mesh, our goal is to optimize a 2D texture which improves classifier performance when mapped onto the mesh. At each iteration, we sample a mesh and a random background; we then use a 3D renderer (Mitsuba [NVZ+19]) to map the corresponding texture onto the mesh. We overlay the rendering onto a random background image, and then feed the resulting composed image into the pre-trained classifier, with the label being that of the sampled 3D mesh. Since rendering is typically non-differentiable, we use a linear approximation of the rendering process (cf. Athalye et al. [AEI+18]) in order to compute (this time approximate) gradients of the model’s loss with respect to the utilized texture. From there, we apply the same SGD algorithm as we did for the patch case.

heading: 3 Experimental evaluation

text:
In order to determine the effectiveness of our proposed framework, we evaluate against a suite of computer vision tasks. Below, we first provide some detail on the precise access model and baselines that will be considered. We then briefly outline the experimental setup of each task, and show that
2We allow the patch to be placed anywhere as a matter of convenience: ideally we would only be applying the patch onto the main object itself, but this would require bounding box data that we do not have for most classification datasets.
unadversarial objects consistently improve the performance and robustness of the vision systems tested. For a more detailed account of each experimental setup, see Appendix C.

heading: 3.1 Access model and baselines

text:
In many of the settings discussed thus far, a system designer can alter the objects being recognized but is not allowed to alter the the classifier itself. That is, we are optimizing unadversarial objects for a fixed (pre-trained) model. For instance, a road engineer may wish to design road signs that are easier to recognize for autonomous vehicles, without being able to train or alter the machine learning models that operate the vehicles. Similarly, a roboticist might want to design a landing pad that works better for a commercial (pre-trained) drone. We will refer to this setting as the fixedmodel setting. On the other hand, sometimes the same entity is able to train both the model and the transformations (the discussed factory robot example may fall into this category, for example). In this “free-model” setting, one may be able to boost performance by co-designing the machine learning model and the objects of interest.
In this work, we will focus on designing unadversarial examples in the fixed-model setting, for the sake of both simplicity and applicability. In particular, any valid algorithm under a fixed-model assumption is also a valid algorithm under the co-design assumption (but the converse is not true). This leaves the task of leveraging even more control in joint optimization settings as a potential avenue for future work.
Baselines. Since we consider the fixed-model setting throughout our paper, the only truly comparable baselines are those which do not alter the model being trained. Nonetheless, in order to fully contextualize our results, we will also consider a few baselines that fall outside of our intended access model (e.g., QR codes). A notable disadvantage of these baselines (that we do not explicitly
demonstrate below) is that if the “unadversarial signal” (e.g., the QR code) is occluded or removed, the entire system fails; this is in contrast to the fixed-model setting, where the pre-trained model is able to recognize objects without any unadversarial signals and is instead “boosted” by their presence.

heading: 3.2 Clean data and synthetic corruptions

text:
We first test whether unadversarial examples improve the performance of image classifiers on benchmark datasets. Using the algorithm described in Section 2.3, we construct unadversarial patches of varying size for pre-trained ResNet-50 classifiers on the CIFAR [Kri09] and ImageNet [RDS+15] datasets. For evaluation, we add these patches at random positions, scales, and orientations to validation set images (see Appendix C for the exact protocol). As shown in Figure 4a, the pre-trained ImageNet classifier is consistently more accurate on the augmented ImageNet images. For example, an unadversarial patch 20 times smaller than ImageNet images boosts accuracy by 26.3% (analogous results for CIFAR are given in Appendix D).
Robustness to synthetic corruptions. Next, we use the CIFAR-C and ImageNet-C datasets [HD19] (consisting of the CIFAR and ImageNet validation sets corrupted in 15 systematic ways) to see whether the addition of unadversarial patches to images confers any corruption robustness.
We use the same patches and evaluation protocol that we used when looking at clean data (to ensure a fair evaluation, we apply corruptions to boosted images only after the unadversarial patches have been applied). As a consequence, at test time neither model nor patch has been exposed to any image corruptions beyond standard data augmentation. As a result, this experiment tests the ability for unadversarially boosted images to withstand completely unforeseen corruptions; we also avoid any potential biases from training on (and thus “overfitting” to [KSH+19]) a specific type of corruption. The results (cf. Figure 4b for ImageNet and Appendix D for CIFAR) indicate that unadversarial patches do improve performance across corruption types; for example, applying an unadversarial patch 5% the size of a standard ImageNet image boosts accuracy by an average of 31.7% points across corruptions 3.
The model does not ignore the image in the presence of unadversarial patches.
Recall from our discussion of the fixed-model setting in Section 3.1 that an advantage of designing unadversarial objects without changing the model is that the model still works in the absence of the unadversarial signal. We now briefly explore the case where the model is exposed to an unadversarial signal for the wrong class. Ideally, we would want the patch to only assist/boost the signal from the original image—in particular, we do not want the patch to make the model totally ignore the contents of the image itself. Thus, in cases where the signal from the image and the patch conflict, we would like the classifier to predict according to the features present in the image more frequently than the class encoded in the unadversarial patch.
3Since the original corruption benchmarks proposed by [HD19] are only available as pre-computed JPEGs (for which we cannot apply a patch pre-corruption) or CPU-based Python image operations (which were prohibitively slow), we re-implemented all 15 corruptions as batched GPU operations; we verified that model accuracies on our corruptions mirrored the original CPU counterparts (i.e., within 1% accuracy). For more details about our reimplementation, see our code release.
Indeed, this turns out to be the case. Figure 5 shows that on CIFAR-10, when the signal from an unadversarial patch and the image itself conflict, the model predicts according to the patch only 31.2% of the time on average, and according to the image 60.3% of the time (the accuracy of the model when the patch and image agree is 98.93%.)
At first glance, this result may seem to be at odds with the near-perfect effectiveness of adversarial patches [BMR+18]. However, the phenomenon we observe here can be tied to the subtle difference between the way we train our unadversarial patches and the way one trains targeted adversarial patches. In the former, we overlay each patch exclusively onto images from its respective class— thus, unadversarial patches are never optimized to be effective when overlaid on a different class. In the latter, however, adversarial patches are optimized to maximize confidence in a particular class on all possible backgrounds, making the patch dominant even when overlaid on an image from a different class.
Baselines. We also compare our results to a variety of natural baselines; the most relevant of these is the “best loss image patch,” where we use the minimum-loss training image in place of a patch. We compare with this baseline to ensure that our method is doing something beyond this naive way to add signal to an image. The results are shown in Appendix D, along with comparisons to less sophisticated baselines, such as QR Codes and predefined random Gaussian noise patches.

heading: 3.3 Classification in 3D simulation

text:
We now test unadversarial examples in a more practical setting: recognizing 3D objects in a high-fidelity simulator. We collect meshes corresponding to four ImageNet classes: “warplane,” “minibus,” “container ship,” and “trailer truck,” from sketchfab.com. We generate a texture for each object using the unadversarial texture algorithm of Section 2.3, using the ImageNet validation set as the set of backgrounds for the algorithm, and a pre-trained ResNet-50 as the classifier.
To evaluate the resulting textures, we import each mesh into Microsoft AirSim, a high-fidelity threedimensional simulator; we then test pre-trained ImageNet models’ ability to recognize each object with and without the unadversarial texture applied in a variety of surroundings. We also test each texture’s robustness to more realistic weather corruptions (snow and fog) built directly into the simulator (rather than applied as a post-processing step). We provide further detail on AirSim and our usage of it in Appendix B. Examples of the images used to evaluate the unadversarial textures, as well as our main results for one of the meshes are shown in Figure 6. We find that in both standard and adverse weather conditions, the model consistently performs better on the unadversarial texture than on the original. We present similar results for the other three meshes in Figure 7.

heading: 3.4 Localization for (simulated) drone landing

text: We then assess whether unadversarial examples can help outside of the classification setting. Again using AirSim, we set up a drone landing task with a perception module that receives as input an axis-aligned aerial image of a landing pad, and is tasked with outputing an estimate of the camera’s (x, y)-position relative to the pad. While this task is quite basic, we are particularly interested in studying performance in the presence of heavy (simulated) weather-based corruptions. The drone is equipped with a pretrained regression model that localizes the landing pad (described in detail in Appendix B). We optimize an unadversarial texture for the surface of the landing pad to best help the drone’s regression model in localization. Figure 8 shows an example of the landing pad localization task, along with the performance of the unadversarial landing pad compared to the standard pad. The drone landing on the unadversarial pad consistently lands both more reliably.

heading: 3.5 Physical-world unadversarial examples

text: Finally, we move out of simulation and test whether the unadversarial patches that we generate can survive naturally-arising distribution shift from effects such as real lighting, camera artifacts, and printing imperfections. We use four household objects (a toy racecar, miniature plane, coffeepot, and eggnog container), and print out (on a standard InkJet printer) the adversarial patch corresponding to the label of each object. We take pictures of the toy with and without the patch taped on using an ordinary cellphone camera, and count the number of poses for which the toy is correctly classified by a pre-trained ImageNet classifier. Our results are in Table 9a, and examples of patches are in Figure 9b. Classifying both patched and unpatched images over a diverse set of poses, we find that the adversarial patches consistently improve performance even at uncommon object orientations.

heading: 4 Related work

text:
Here, we first highlight (and differentiate from) previous works using reference markers to improve recognition and localization. We will then discuss related work in adversarial robustness.
Improving computer vision with fiducial markers. In past research, vision-based precision landing was initially attempted with classical computer vision based tracking of reference designs
[SMS03; LSP09; YSS+14; FZS+17], or on arrangements of fiducial markers [LZT+19] (a fiducial marker is a fixed pattern or object that is placed in a scene as a reference point for location or measure). Several types of visual fiducial markers were also proposed for robust tracking and pose estimation applications through classical vision [Fia05; GMM+14; Ols11; RMM19]. While fiducial markers such as AprilTags [Ols11] were commonly used in robotics, they suffer from limitations such as orientation uncertainty, accuracy falloff depending on viewing angles, and short detection ranges [AAB+19; WO16], which motivated research into using convolutional neural networks for landing pad detection and pose estimation [NAK+18; TNN+19]. Our work attempts to unify these perspectives by leveraging the expressivity of neural network-based systems to design robust unadversarial examples. Another key difference between fiducial markers and unadversarial objects is that the former require vision systems to be aware of their presence; in contrast, the latter are designed on top of pre-trained systems. This means that we do not require any further non-standard model training, and that we do not depend as heavily on the unadversarial example being visible.
Adversarial robustness. Our design of unadversarial examples is motivated by the success of adversarial examples, i.e., minute perturbations to the inputs of machine learning models that can induce nearly arbitrary behavior. Adversarial examples were shown to be effective in seriously hampering machine vision related tasks such as classification [SZS+14; EEF+18a; AEI+18], object detection/segmentation [EEF+18b; AMT18; LK19; XWZ+17] and visual question-answering (VQA) [XCL+18]. Furthermore, prior work has shown that adversarial examples can be constructed even without direct access to the vision system being manipulated [CZS+17; PMG16; PMG+17; IEA+18; IEM18] Synthesized physical adversarial examples were shown to be effective in fooling person detection [TVG19], sign detection for autonomous driving [SBM+18; BHG+19]. Robotic platforms such as manipulators were also shown to be sensitive to vision based adversarial examples [MDB+17], and to specifically designed adversarial objects [WTL+19]. Additionally, recent work [EGS19; NHD+19] shows that one can “reprogram” neural networks using adversarial examples, e.g., one can construct a patch that causes a CIFAR-10 classifier to operate as an MNIST classifier.

heading: 5 Discussion and conclusions

text:
In this work, we demonstrated that it is possible to design object alternations that boost the corresponding classifiers’ performance, even under strong and unforeseen distribution shift. Indeed, such resulting unadversarial objects are robust to a broad range of data shifts and corruptions, even when these were never seen in training. We view our results as a promising route towards increasing out-of-distribution robustness of computer vision models.
Limitations and future work. One limitation of our method is that it requires differentiability with respect to properties of the input object of interest, or a close proxy (e.g. a differentiable simulator) that can mimick the object and the environment in which the object operates.
Furthermore, as discussed at length in Section 3.1, in this work we only consider the case where the system designer cannot alter the model being trained. While this carries some inherent advantages (e.g., increased applicability), future work should investigate whether attaining even more accuracy and robustness is possible by jointly optimizing models and unadversarial objects.
Finally, we have shown unadversarial examples to be effective only in classification and regression settings. However, the fact that unadversarial examples and adversarial examples share the same underlying generation technique is evidence that unadversarial examples could apply to any system that is vulnerable to adversarial examples. Thus, an avenue of future work could be to apply these examples to detection, segmentation, style transfer, and other domains.
Broader impact. On one hand, as with any improvement to the robustness and reliability of computer vision systems, unadversarial examples also carry the potential for misuse (e.g., as part of overreaching surveillance technology). On the other hand, unadversarial examples give end users of machine learning systems—who typically have little to no power over the behaviour of the ML systems they interact with—the power to improve the performance and robustness of these systems in their respective use cases. For example, in the future someone living in a remote area where roads or traffic signs may look “atypical” can slightly modify their signs to be better recognized by autonomous vehicles.

heading: Acknowledgements

text:
We are grateful to Ian Engstrom for helping take the photographs for the physical-world experiments.
Work supported in part by the NSF grants CCF-1553428 and CNS-1815221, Open Philanthropy, and the Microsoft Corporation. This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001120C0015.
Research was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

title: Unadversarial Examples: Designing Objects for Robust Vision


summaries: (Auxiliary Review only -- the authors need not respond to this).  This work proposes issues with data augmentation in RL and proposes a couple of techniques to address those issues. It applies augmentation only on the current state and not future state to address erroneous bootstrapping. It also proposes a Q objective that uses both the augmented and un-augmented data. Also, for actor critic experiments, the actor is optimised only with un-augmented data and learn a  policy that is general via sharing parameters between the networks. These are simple techniques that help the paper demonstrate good results in a range of tasks.  This is a clear paper with good set of experiments. The problem has been described clearly and the work proposes simple refinements to the data augmentation strategies used in RL. While I am convinced of the quality of the method itself and the experiments, this work could have benefitted from more rigour with the baselines to make a stronger case for the method. Hence, I am offering a score of 6 noting that the paper is marginally above acceptance threshold.

summaries: This paper studies the problem of high-variance Q-targets under data augmentation, and proposes a method to reduce the variance of the targets called SVEA. The key idea contains three parts: (1) Apply augmentation to the Q-value estimation of the current state and not the next state Q-targets for bootstrapping, and (2) modify the objective to include both the targets using augmented and original observations and (3) optimize the actor using only the unaugmented data for actor-critic algorithms, with parameter sharing.   The paper performed experiments in DeepMind control suite and its variants (with raw pixel inputs). Compared to other data augmentation methods (e.g. DrQ) their proposed SVEA method is robust to the type of data augmentation and is more sample efficient with a better final performance. Some ablations were also presented among the above components to indicate the usefulness of all components. Generalization performance was demonstrated in the DistractingCS and robotic manipulation environments. The method also appears to help when using a Vision Transformers architecture.

summaries: This work proposes a simple technique to improve the robustness and sample-efficiency of reinforcement learning. The technique can be applied to any algorithm via a simple modification of the input pipeline to said algorithm's Q-learning subroutine. The claim is that this modification reduces the variance of Q-targets and therefore stabilizes learning.

summaries: The paper addresses the question of zero-shot or fast generalization of value-based deep reinforcement learning agents using visual inputs to image transformation which do not change the (conceptual) state of the environment. The long term goal is therefore training agents which learn efficient, high return control policies both which are robust to irrelevant variations in high dimensional input observations. The ultimate goal seems to be generalization of learned skills to novel environments, but as far as I can tell this particular evaluation philosophy is implemented in experiments.  The paper reviews previous attempts to learn robust state representations to visual variations and formulates clear hypotheses for explaining the causes of difficulties which prior methods faced and were hindered by. The said hypotheses suggest a simplified approach (SVEA) which is empirically developed and evaluated in the remainder of the paper. Control tasks from widely used benchmarks are used to evaluate the method, somewhat less usual is the focus on vision-based high-dimensional versions of such tasks, instead of standard domains such as Atari games and benchmarks for generalization such as Natural Atari [1], CoinRun/ProcGen [2].

summaries: The paper presents a way of improving training stability of DrQ (an image-based RL algorithm that uses data augmentation) by applying image augmentation more carefully. The authors propose to input a mix of augmented and un-augmented observations into the critic, and stop augmenting the actor's inputs. On the considered benchmarks the proposed method demonstrates better generalization results.

abstractText: While agents trained by Reinforcement Learning (RL) can solve increasingly challenging tasks directly from visual observations, generalizing learned skills to novel environments remains very challenging. Extensive use of data augmentation is a promising technique for improving generalization in RL, but it is often found to decrease sample efficiency and can even lead to divergence. In this paper, we investigate causes of instability when using data augmentation in common off-policy RL algorithms. We identify two problems, both rooted in high-variance Q-targets. Based on our findings, we propose a simple yet effective technique for stabilizing this class of algorithms under augmentation. We perform extensive empirical evaluation of image-based RL using both ConvNets and Vision Transformers (ViT) on a family of benchmarks based on DeepMind Control Suite, as well as in robotic manipulation tasks. Our method greatly improves stability and sample efficiency of ConvNets under augmentation, and achieves generalization results competitive with state-of-the-art methods for image-based RL in environments with unseen visuals. We further show that our method scales to RL with ViT-based architectures, and that data augmentation may be especially important in this setting.†

authors:
Rishabh Agarwal
Marlos C. Machado
P.S. Castro
Marc G. Bellemare

title: Contrastive behavioral similarity embeddings for generalization in reinforcement

venue: learning. ArXiv,

title: A markovian decision process

venue: Journal of Mathematics and Mechanics,

authors:
Emily M. Bender
Timnit Gebru
Angelina McMillan-Major
Shmargaret Shmitchell

title: On the dangers of stochastic parrots: Can language models be too big

venue: In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. Association for Computing Machinery,

authors:
Christopher Berner
Greg Brockman
Brooke Chan
Vicki Cheung
Przemyslaw Debiak
Christy Dennison
David Farhi
Quirin Fischer

title: Dota 2 with large scale deep reinforcement

authors:
T. Brown
Benjamin Mann
Nick Ryder
Melanie Subbiah
J. Kaplan
Prafulla Dhariwal
Arvind Neelakantan
Pranav Shyam

title: Language models are few-shot

venue: learners. arXiv,

authors:
Yevgen Chebotar
A. Handa
Viktor Makoviychuk
M. Macklin
J. Issac
Nathan D. Ratliff
D. Fox

title: Closing the sim-to-real loop: Adapting simulation randomization with real world experience

venue: International Conference on Robotics and Automation (ICRA),

authors:
Ting Chen
Simon Kornblith
Mohammad Norouzi
Geoffrey Hinton

title: A simple framework for contrastive learning of visual representations, 2020

authors:
K. Cobbe
Oleg Klimov
Christopher Hesse
Taehoon Kim
John Schulman

title: Quantifying generalization in reinforcement learning

venue: In Icml,

authors:
Karl Cobbe
Oleg Klimov
Chris Hesse
Taehoon Kim
John Schulman

title: Quantifying generalization in reinforcement learning, 2018

authors:
A. Dosovitskiy
Lucas Beyer
Alexander Kolesnikov
Dirk Weissenborn
Xiaohua Zhai
Thomas Unterthiner
M. Dehghani
Matthias Minderer

title: An image is worth 16x16 words: Transformers for image recognition

authors:
Frederik Ebert
Chelsea Finn
Sudeep Dasari
Annie Xie
Alex Lee
Sergey Levine

title: Visual foresight: Model-based deep reinforcement learning for vision-based robotic control, 2018

authors:
Lasse Espeholt
Hubert Soyer
R. Munos
K. Simonyan
V. Mnih
Tom Ward
Yotam Doron
Vlad Firoiu

title: Impala: Scalable distributed deep-rl with importance weighted actor-learner

venue: architectures. ArXiv,

authors:
Jesse Farebrother
Marlos C. Machado
Michael H. Bowling

title: Generalization and regularization in dqn

venue: ArXiv,

authors:
Scott Fujimoto
H.V. Hoof
D. Meger

title: Addressing function approximation error in actor-critic methods

authors:
Spyros Gidaris
Praveer Singh
Nikos Komodakis

title: Unsupervised representation learning by predicting image rotations, 2018

authors:
R. Givan
T. Dean
M. Greig

title: Equivalence notions and model minimization in markov decision processes

venue: Artificial Intelligence,

authors:
Tuomas Haarnoja
Aurick Zhou
Pieter Abbeel
Sergey Levine

title: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

venue: arXiv preprint arXiv:1801.01290,

authors:
Danijar Hafner
T. Lillicrap
Ian S. Fischer
R. Villegas
David R Ha
H. Lee
James Davidson

title: Learning latent dynamics for planning

venue: pixels. ArXiv,

authors:
Danijar Hafner
T. Lillicrap
Jimmy Ba
Mohammad Norouzi

title: Dream to control: Learning behaviors by latent imagination

authors:
Nicklas Hansen
Xiaolong Wang

title: Generalization in reinforcement learning by soft data augmentation

venue: In International Conference on Robotics and Automation,

authors:
Nicklas Hansen
Rishabh Jangir
Yu Sun
Guillem Alenyà
Pieter Abbeel
Alexei A. Efros
Lerrel Pinto
Xiaolong Wang

title: Self-supervised policy adaptation during deployment

venue: In International Conference on Learning Representations,

authors:
H.V. Hasselt
A. Guez
Matteo Hessel
V. Mnih
D. Silver

title: Learning values across many orders of magnitude

venue: Nips,

authors:
H.V. Hasselt
A. Guez
D. Silver

title: Deep reinforcement learning with double q-learning

venue: In Aaai,

authors:
M. Hausknecht
P. Stone

title: Deep recurrent q-learning for partially observable mdps

venue: In AAAI Fall Symposia,

authors:
Kaiming He
X. Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Kaiming He
Haoqi Fan
Yuxin Wu
Saining Xie
Ross B. Girshick

title: Momentum contrast for unsupervised visual representation learning

venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Matteo Hessel
Joseph Modayil
H.V. Hasselt
Tom Schaul
Georg Ostrovski
Will Dabney
Dan Horgan
Bilal Piot

title: Rainbow: Combining improvements in deep reinforcement learning

venue: Aaai,

authors:
Max Jaderberg
Volodymyr Mnih
Wojciech Marian Czarnecki
Tom Schaul
Joel Z Leibo
David Silver
Koray Kavukcuoglu

title: Reinforcement learning with unsupervised auxiliary tasks, 2016

authors:
Leslie Pack Kaelbling
Michael L. Littman
Anthony R. Cassandra

title: Planning and acting in partially observable stochastic domains

authors:
D. Kalashnikov
A. Irpan
P. Pastor
J. Ibarz
A. Herzog
Eric Jang
Deirdre Quillen
Ethan Holly

title: Qtopt: Scalable deep reinforcement learning for vision-based robotic

venue: manipulation. ArXiv,

authors:
Diederik P. Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

authors:
Ilya Kostrikov
Denis Yarats
Rob Fergus

title: Image augmentation is all you need: Regularizing deep reinforcement learning from pixels

venue: International Conference on Learning Representations,

authors:
K.G. Larsen
A. Skou

title: Bisimulation through probabilistic testing (preliminary report)

venue: In Proceedings of the 16th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages,

authors:
Michael Laskin
Kimin Lee
Adam Stooke
Lerrel Pinto
Pieter Abbeel
Aravind Srinivas

title: Reinforcement learning with augmented data

venue: arXiv preprint arXiv:2004.14990,

authors:
Kimin Lee
Kibok Lee
Jinwoo Shin
Honglak Lee

title: A simple randomization technique for generalization in deep reinforcement

authors:
Sergey Levine
Chelsea Finn
Trevor Darrell
Pieter Abbeel

title: End-to-end training of deep visuomotor policies

venue: The Journal of Machine Learning Research,

authors:
T. Lillicrap
J. Hunt
A. Pritzel
N. Heess
T. Erez
Y. Tassa
D. Silver
Daan Wierstra

title: Continuous control with deep reinforcement learning

title: Self-improving reactive agents based on reinforcement learning, planning and teaching

venue: Machine Learning,

authors:
Xingyu Lin
Harjatin Singh Baweja
George Kantor
David Held

title: Adaptive auxiliary task weighting for reinforcement learning

venue: Advances in neural information processing systems,

authors:
Clare Lyle
Mark Rowland
Georg Ostrovski
Will Dabney

title: On the effect of auxiliary tasks on representation dynamics

venue: In Aistats,

authors:
P. Mirowski
Razvan Pascanu
Fabio Viola
Hubert Soyer
Andy Ballard
Andrea Banino
Misha Denil
R. Goroshin

title: Learning to navigate in complex

venue: environments. ArXiv,

authors:
Volodymyr Mnih
Koray Kavukcuoglu
David Silver
Alex Graves
Ioannis Antonoglou
Daan Wierstra
Martin Riedmiller

title: Playing atari with deep reinforcement learning

venue: arXiv preprint arXiv:1312.5602,

authors:
Ashvin V Nair
Vitchyr Pong
Murtaza Dalal
Shikhar Bahl
Steven Lin
Sergey Levine

title: Visual reinforcement learning with imagined goals

venue: In Advances in Neural Information Processing Systems,

authors:
Mehdi Noroozi
Paolo Favaro

title: Unsupervised learning of visual representations by solving jigsaw puzzles

venue: In European Conference on Computer Vision,

authors:
Deepak Pathak
Philipp Krahenbuhl
Jeff Donahue
Trevor Darrell
Alexei A Efros

title: Context encoders: Feature learning by inpainting

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Deepak Pathak
Dhiraj Gandhi
A. Gupta

title: Self-supervised exploration via disagreement

authors:
Xue Bin Peng
Marcin Andrychowicz
Wojciech Zaremba
Pieter Abbeel

title: Sim-to-real transfer of robotic control with dynamics randomization

venue: IEEE International Conference on Robotics and Automation (ICRA),

authors:
Lerrel Pinto
Abhinav Gupta

title: Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours

venue: In 2016 IEEE international conference on robotics and automation (ICRA),

authors:
Lerrel Pinto
Marcin Andrychowicz
Peter Welinder
Wojciech Zaremba
Pieter Abbeel

title: Asymmetric actor critic for image-based robot learning

venue: arXiv preprint arXiv:1710.06542,

authors:
Roberta Raileanu
M. Goldstein
Denis Yarats
Ilya Kostrikov
R. Fergus

title: Automatic data augmentation for generalization in deep reinforcement learning

authors:
Fabio Ramos
Rafael Possas
Dieter Fox

title: Bayessim: Adaptive domain randomization via probabilistic inference for robotics simulators

venue: Robotics: Science and Systems

authors:
Max Schwarzer
Ankesh Anand
Rishab Goel
R Devon Hjelm
Aaron Courville
Philip Bachman

title: Data-efficient reinforcement learning with self-predictive representations

venue: arXiv preprint arXiv:2007.05929,

authors:
Ramanan Sekar
Oleh Rybkin
Kostas Daniilidis
Pieter Abbeel
Danijar Hafner
Deepak Pathak

title: Planning to explore via self-supervised world models, 2020

authors:
Evan Shelhamer
Parsa Mahmoudieh
Max Argus
Trevor Darrell

title: Loss is its own reward: Selfsupervision for reinforcement

authors:
Connor Shorten
T. Khoshgoftaar

title: A survey on image data augmentation for deep learning

venue: Journal of Big Data,

authors:
Xingyou Song
Yiding Jiang
Stephen Tu
Yilun Du
Behnam Neyshabur

title: Observational overfitting in reinforcement learning

authors:
Aravind Srinivas
Michael Laskin
Pieter Abbeel

title: Curl: Contrastive unsupervised representations for reinforcement learning

venue: arXiv preprint arXiv:2004.04136,

authors:
Austin Stone
Oscar Ramirez
K. Konolige
Rico Jonschkowski

title: The distracting control suite - a challenging benchmark for reinforcement learning

authors:
Adam Stooke
Kimin Lee
Pieter Abbeel
Michael Laskin

title: Decoupling representation learning from reinforcement learning

title: Learning to predict by the methods of temporal differences

authors:
Richard S. Sutton
Andrew G. Barto

title: Reinforcement Learning: An Introduction

authors:
Yuval Tassa
Yotam Doron
Alistair Muldal
Tom Erez
Yazhe Li
Diego de Las Casas
David Budden
Abbas Abdolmaleki

title: Deepmind control suite

venue: Technical report,

authors:
Yonglong Tian
Dilip Krishnan
Phillip Isola

title: Contrastive multiview coding

venue: arXiv preprint arXiv:1906.05849,

authors:
Josh Tobin
Rachel Fong
Alex Ray
Jonas Schneider
Wojciech Zaremba
Pieter Abbeel

title: Domain randomization for transferring deep neural networks from simulation to the real world

venue: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),

authors:
Emanuel Todorov
Tom Erez
Yuval Tassa

title: Mujoco: A physics engine for model-based control

venue: In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems,

authors:
Aaron van den Oord
Yazhe Li
Oriol Vinyals

title: Representation learning with contrastive predictive coding, 2018

authors:
Ashish Vaswani
Noam M. Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N. Gomez
Lukasz Kaiser
Illia Polosukhin

title: Attention is all you

venue: need. arXiv,

authors:
Oriol Vinyals
I. Babuschkin
Wojciech Czarnecki
Micha "el Mathieu
Andrew Dudzik
J. Chung
D. Choi
Richard Powell

title: Grandmaster level in starcraft ii using multi-agent reinforcement

venue: learning. Nature,

authors:
K. Wang
Bingyi Kang
Jie Shao
Jiashi Feng

title: Improving generalization in reinforcement learning with mixture regularization

authors:
Xudong Wang
Long Lian
Stella X. Yu

title: Unsupervised visual attention and invariance for reinforcement learning

authors:
Ziyu Wang
Tom Schaul
Matteo Hessel
H.V. Hasselt
Marc Lanctot
N.D. Freitas

title: Dueling network architectures for deep reinforcement

authors:
Zhirong Wu
Yuanjun Xiong
Stella X Yu
Dahua Lin

title: Unsupervised feature learning via nonparametric instance discrimination

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Zhenlin Xu
Deyi Liu
Junlin Yang
M. Niethammer

title: Robust and generalizable visual representation learning via random convolutions

authors:
Denis Yarats
Amy Zhang
Ilya Kostrikov
Brandon Amos
Joelle Pineau
Rob Fergus

title: Improving sample efficiency in model-free reinforcement learning from images, 2019

authors:
Denis Yarats
Rob Fergus
Alessandro Lazaric
Lerrel Pinto

title: Reinforcement learning with prototypical representations

venue: arXiv preprint arXiv:2102.11271,

authors:
Sarah Young
Dhiraj Gandhi
Shubham Tulsiani
Abhinav Gupta
Pieter Abbeel
Lerrel Pinto

title: Visual imitation made easy

venue: CoRL,

authors:
Tianhe Yu
Saurabh Kumar
A. Gupta
Sergey Levine
Karol Hausman
Chelsea Finn

title: Gradient surgery for multi-task learning

authors:
A. Zhang
Rowan McAllister
R. Calandra
Y. Gal
Sergey Levine

title: Learning invariant representations for reinforcement learning without reconstruction

authors:
C. Zhang
Oriol Vinyals
R. Munos
S. Bengio

title: A study on overfitting in deep reinforcement

authors:
Richard Zhang
Phillip Isola
Alexei A Efros

title: Colorful image colorization

venue: In European conference on computer vision,

authors:
Yuke Zhu
Roozbeh Mottaghi
Eric Kolve
Joseph J Lim
Abhinav Gupta
Li Fei-Fei
Ali Farhadi

title: Target-driven visual navigation in indoor scenes using deep reinforcement learning

venue: In Icra,

authors:
Brian D Ziebart
Andrew Maas
J Andrew Bagnell
Anind K Dey

title: Maximum entropy inverse reinforcement learning

venue: In Proceedings of the 23rd National Conference on Artificial Intelligence,

heading: 1 Introduction

text:
Reinforcement Learning (RL) from visual observations has achieved tremendous success in various applications such as video-games [43, 4, 70], robotic manipulation [37], and autonomous navigation [42, 83]. However, it is still very challenging for current methods to generalize the learned skills to novel environments, and policies trained by RL can easily overfit to the training environment [81, 13], especially for high-dimensional observation spaces such as images [8, 58].
Increasing the variability in training data via domain randomization [66, 50] and data augmentation [57, 35, 33, 51] has demonstrated encouraging results for learning policies invariant to changes in environment observations. Specifically, recent works on data augmentation [35, 33] both show improvements in sample efficiency from simple cropping and translation augmentations, but the studies also conclude that additional data augmentation in fact decrease sample efficiency and even cause divergence. While these augmentations have the potential to improve generalization, the increasingly varied data makes the optimization more challenging and risks instability. Unlike supervised learning, balancing the trade-off between stability and generalization in RL requires substantial trial and error.
In this paper, we illuminate causes of instability when applying data augmentation to common off-policy RL algorithms [43, 38, 15, 18]. Based on our findings, we provide an intuitive method for stabilizing this class of algorithms under use of strong data augmentation. Specifically, we find two main causes of instability in previous work’s application of data augmentation: (i) indiscriminate application of data augmentation resulting in high-variance Q-targets; and (ii) that Q-value estimation strictly from augmented data results in over-regularization.
†Website and code is available at: https://nicklashansen.github.io/SVEA.
35th Conference on Neural Information Processing Systems (NeurIPS 2021)
To address these problems, we propose SVEA: Stabilized Q-Value Estimation under Augmentation, a simple yet effective framework for data augmentation in off-policy RL that greatly improves stability of Q-value estimation. Our method consists of the following three components: Firstly, by only applying augmentation in Q-value estimation of the current state, without augmenting Q-targets used for bootstrapping, SVEA circumvents erroneous bootstrapping caused by data augmentation; Secondly, we formulate a modified Q-objective that optimizes Q-value estimation jointly over both augmented and unaugmented copies of the observations; Lastly, for SVEA implemented with an actorcritic algorithm, we optimize the actor strictly on unaugmented data, and instead learn a generalizable policy indirectly through parameter-sharing. Our framework can be implemented efficiently without additional forward passes nor introducing additional learnable parameters.
We perform extensive empirical evaluation on the DeepMind Control Suite [64] and extensions of it, including the DMControl Generalization Benchmark [21] and the Distracting Control Suite [60], as well as a set of robotic manipulation tasks. Our method greatly improve Q-value estimation with ConvNets under a set of strong data augmentations, and achieves sample efficiency, asymptotic performance, and generalization that is competitive or better than previous state-of-the-art methods in all tasks considered, at a lower computational cost. Finally, we show that our method scales to RL with Vision Transformers (ViT) [10]. We find that ViT-based architectures are especially prone to overfitting, and data augmentation may therefore be a key component for large-scale RL.

heading: 2 Related Work

text:
Representation Learning. Learning visual invariances using data augmentation and self-supervised objectives has proven highly successful in computer vision [46, 45, 82, 74, 68, 65, 75, 27, 7]. For example, Chen et al. [7] perform an extensive study on data augmentation (e.g. random cropping and image distortions) for contrastive learning, and show that representations pre-trained with such transformations transfer effectively to downstream tasks. While our work also uses data augmentation for learning visual invariances, we leverage the Q-objective of deep Q-learning algorithms instead of auxiliary representation learning tasks.
Visual Learning for RL. Numerous methods have been proposed with the goal of improving sample efficiency [29, 56, 68, 76, 40, 59, 61, 54, 77] of image-based RL. Recently, using self-supervision to improve generalization in RL has also gained interest [80, 47, 55, 1, 22, 21, 72]. Notably, Zhang et al. [80] and Agarwal et al. [1] propose to learn behavioral similarity embeddings via auxiliary tasks (bisimulation metrics and contrastive learning, respectively), and Hansen et al. [21] learn visual invariances through an auxiliary prediction task. While these results are encouraging, it has also been shown in [29, 40, 22, 79, 41] that the best choice of auxiliary tasks depends on the particular RL task, and that joint optimization with sub-optimally chosen tasks can lead to gradient interference. We achieve competitive sample-efficiency and generalization results without the need for carefully chosen auxiliary tasks, and our method is therefore applicable to a larger variety of RL tasks.
Data Augmentation and Randomization for RL. Our work is directly inspired by previous work on generalization in RL by domain randomization [66, 50, 48, 52, 6] and data augmentation [36, 9, 71, 35, 33, 51, 61, 21]. For example, Tobin et al. [66] show that a neural network trained for object localization in a simulation with randomized visual augmentations improves real world generalization. Similarly, Lee et al.[36] show that application of a random convolutional layer to observations during training improve generalization in 3D navigation tasks. More recently, extensive studies on data augmentation [35, 33] have been conducted with RL, and conclude that, while small random crops and translations can improve sample efficiency, most data augmentations decrease sample efficiency and cause divergence. We illuminate main causes of instability, and propose a framework for data augmentation in deep Q-learning algorithms that drastically improves stability and generalization.
Improving Deep Q-Learning. While deep Q-learning algorithms such as Deep Q-Networks (DQN) [43] have achieved impressive results in image-based RL, the temporal difference objective is known to have inherent instabilities when used in conjunction with function approximation and off-policy data [63]. Therefore, a variety of algorithmic improvements have been proposed to improve convergence [24, 73, 25, 23, 53, 38, 15, 14, 28]. For example, Hasselt et al. [24] reduce overestimation of Q-values by decomposing the target Q-value estimation into action selection and action evaluation using separate networks. Lillicrap et al. [38] reduce target variance by defining the target Q-network as a slow-moving average of the online Q-network. Our method also improves Q-value estimation, but we specifically address the instability of deep Q-learning algorithms on augmented data.

heading: 3 Preliminaries

text:
Problem formulation. We formulate the interaction between environment and policy as a Markov Decision Process (MDP) [2] M = hS,A,P, r, i, where S is the state space, A is the action space, P : S ⇥ A 7! S is the state transition function that defines a conditional probability distribution P (·|st,at) over all possible next states given a state st 2 S and action at 2 A taken at time t, r : S⇥A 7! R is a reward function, and 2 [0, 1) is the discount factor. Because image observations only offer partial state observability [30], we define a state st as a sequence of k + 1 consecutive frames (ot,ot 1, . . . ,ot k), o 2 O, where O is the high-dimensional image space, as proposed in Mnih et al. [43]. The goal is then to learn a policy ⇡ : S 7! A that maximizes discounted return Rt = E ⇠⇡[ P T t=1 t r(st,at)] along a trajectory = (s0, s1, . . . , sT ) obtained by following policy ⇡ from an initial state s0 2 S to a state sT with state transitions sampled from P , and ⇡ is parameterized by a collection of learnable parameters ✓. For clarity, we hereon generically denote parameterization with subscript, e.g. ⇡✓. We further aim to learn parameters ✓ s.t. ⇡✓ generalizes well (i.e., obtains high discounted return) to unseen MDPs, which is generally unfeasible without further assumptions about the structure of the space of MDPs. In this work, we focus on generalization to MDPs M = hS,A,P, r, i, where states st 2 S are constructed from observations ot 2 O, O ✓ O of a perturbed observation space O (e.g. unseen visuals), and M ⇠M for a space of MDPs M. Deep Q-Learning. Common model-free off-policy RL algorithms aim to estimate an optimal stateaction value function Q⇤ : S ⇥ A 7! R as Q✓(s,a) ⇡ Q⇤(s,a) = max⇡✓ E [Rt|st = s,at = a] using function approximation. In practice, this is achieved by means of the single-step Bellman residual ⇣ r(st,at) + maxa0t Q tgt (st+1,a0t) ⌘ Q✓(st,at) [62], where parameterizes a target state-action value function Qtgt. We can choose to minimize this residual (also known as the temporal difference error) directly wrt ✓ using a mean squared error loss, which gives us the objective
LQ(✓, ) = Est,at,st+1⇠B
" 1
2
✓ r(st,at) + max
a0t Q
tgt (st+1,a 0 t ) ◆ Q✓(st,at) 2# , (1)
where B is a replay buffer with transitions collected by a behavioral policy [39]. From here, we can derive a greedy policy directly by selecting actions at = argmaxat Q✓(st,at). While Qtgt = Q and periodically setting ✓ exactly recovers the objective of DQN [43], several improvements have been proposed to improve stability of Eq. 1, such as Double Q-learning [24], Dueling Q-networks [73], updating target parameters using a slow-moving average of the online Q-network [38]:
n+1 (1 ⇣) n + ⇣✓n (2) for an iteration step n and a momentum coefficient ⇣ 2 (0, 1], and others [25, 23, 53, 14, 28]. As computing maxa0t Q tgt (st+1,a0t) in Eq. 1 is intractable for large and continuous action spaces, a number of prominent actor-critic algorithms that additionally learn a policy ⇡✓(st) ⇡ argmaxat Q✓(st,at) have therefore been proposed [38, 15, 18].
Soft Actor-Critic (SAC) [18] is an off-policy actor-critic algorithm that learns a state-action value function Q✓ and a stochastic policy ⇡✓ (and optionally a temperature parameter), where Q✓ is optimized using a variant of the objective in Eq. 1 and ⇡✓ is optimized using a -discounted maximum-entropy objective [84]. To improve stability, SAC is also commonly implemented using Double Q-learning and the slow-moving target parameters from Eq. 2. We will in the remainder of this work describe our method in the context of a generic off-policy RL algorithm that learns a parameterized state-action value function Q✓, while we in our experiments discussed in Section 6 evaluate of our method using SAC as base algorithm.
4 Pitfalls of Data Augmentation in Deep Q-Learning In this section, we aim to illuminate the main causes of instability from naïve application of data augmentation in Q-value estimation. Our goal is to learn a Q-function Q✓ for an MDP M that generalizes to novel MDPs M ⇠M with unseen visuals, and we leverage data augmentation as an optimality-invariant state transformation ⌧ to induce a bisimulation relation [34, 17] between a state s and its transformed (augmented) counterpart saug = ⌧(s, ⌫) with parameters ⌫ ⇠ V . Definition 1 (Optimality-Invariant State Transformation [33]). Given an MDP M, a state transformation ⌧ : S ⇥ V 7! S is an optimality-invariant state transformation if Q(s,a) = Q(⌧(s, ⌫),a) 8 s 2 S, a 2 A, ⌫ 2 V , where ⌫ 2 V parameterizes the transformation ⌧ .
Following our definitions of M,M from Section 3, we can further extend the concept of optimalityinvariant transformations to MDPs, noting that a change of state space (e.g. perturbed visuals) itself can be described as a transformation ⌧ : S ⇥ V 7! S with unknown parameters ⌫ 2 V . If we choose the set of parameters V of a state transformation ⌧ to be sufficiently large such that it intersects with V with high probability, we can therefore expect to improve generalization to state and observation spaces not seen during training. However, while naïve application of data augmentation as in previous work [35, 33, 61, 54] may potentially improve generalization, it can be harmful to Q-value estimation. We hypothesize that this is primarily because it dramatically increases the size of the observed state space, and consequently also increases variance Var [Q(⌧(s, ⌫))] Var [Q(s)] , ⌫ ⇠ V when V is large. Concretely, we identify the following two issues:
Pitfall 1: Non-deterministic Q-target. For deep Q-learning algorithms, previous work [35, 33, 61, 54] applies augmentation to both state saug
t , ⌧(st, ⌫) and successor state saugt+1 , ⌧(st+1, ⌫0) where
⌫, ⌫ 0 ⇠ V . Compared with DQN [43] that uses a deterministic (more precisely, periodically updated) Q-target, this practice introduces a non-deterministic Q-target r(st,at) + maxa0t Q tgt (saug t+1,a 0 t ) depending on the augmentation parameters ⌫0. As observed in the original DQN paper, high-variance target values are detrimental to Q-learning algorithms, and may cause divergence due to the “deadly triad” of function approximation, bootstrapping, and off-policy learning [63]. This motivates the work to introduce a slowly changing target network, and several other works have refined the Q-target update rule [38, 15] to further reduce volatility. However, because data augmentation is inherently non-deterministic, it can greatly increase variance in Q-target estimation and exacerbates the issue of volatility, as shown in Figure 1 (top). This is particularly troubling in actor-critic algorithms such as DDPG [38] and SAC [18], where the Q-target is estimated from (st+1,a0), a0 ⇠ ⇡(·|st+1), which introduces an additional source of error from ⇡ that is non-negligible especially when st+1 is augmented.
Pitfall 2: Over-regularization. Data augmentation was originally introduced in the supervised learning regime as a regularizer to prevent overfitting of high-capacity models. However, for RL, even learning a policy in the training environment is hard. While data augmentation may improve generalization, it greatly increases the difficulty of policy learning, i.e., optimizing ✓ for Q✓ and potentially a behavior network ⇡✓. Particularly, when the temporal difference loss from Eq. 1 cannot be well minimized, the large amount of augmented states dominate the gradient, which significantly impacts Q-value estimation of both augmented and unaugmented states. We refer to this issue as over-regularization by data augmentation. Figure 1 (bottom) shows the mean difference in Q-predictions made with augmented vs. unaugmented data in fully converged DrQ [33] agents trained with shift augmentation. Augmentations such as affine-jitter, random convolution, and random overlay incur large differences in estimated Q-values. While such difference can be reduced by regularizing the optimization with each individual augmentation, we emphasize that even the minimal shift augmentation used throughout training incurs non-zero difference. Since is commonly chosen to be a
moving average of ✓ as in Eq. 2, such differences caused by over-regularization affect Q✓ and Q tgt equally, and optimization may therefore still diverge depending on the choice of data augmentation. As such, there is an inherent trade-off between accurate Q-value estimation and generalization when using data augmentation. In the following section, we address these pitfalls.

heading: 5 Method

text: We propose SVEA: Stabilized Q-Value Estimation under Augmentation, a general framework for visual generalization in RL by use of data augmentation. SVEA applies data augmentation in a novel learning framework leveraging two data streams – with and without augmented data, respectively. Our method is compatible with any standard off-policy RL algorithm without changes to the underlying neural network that parameterizes the policy, and it requires no additional forward passes, auxiliary tasks, nor learnable parameters. While SVEA in principle does not make any assumptions about the structure of states st 2 S , we here describe our method in the context of image-based RL.

heading: 5.1 Architectural Overview

text:
An overview of the SVEA architecture is provided in Figure 2. Our method leverages properties of common neural network architectures used in off-policy RL without introducing additional learnable parameters. We subdivide the neural network layers and corresponding learnable parameters of a state-action value function into sub-networks f✓ (denoted the state encoder) and Q✓ (denoted the Q-function) s.t qt , Q✓(f✓(st),at) is the predicted Q-value corresponding to a given stateaction pair (st,at). We similarly define the target state-action value function s.t. q tgt t
, r(st,at) + maxa0t Q tgt (f tgt (st+1),a0) is the target Q-value for (st,at), and we define parameters as an exponential moving average of ✓ as in Eq. 2. Depending on the choice of underlying algorithm, we may choose to additionally learn a parameterized policy ⇡✓ that shares encoder parameters with Q✓ and selects actions at ⇠ ⇡✓(·|f✓(st)).
To circumvent erroneous bootstrapping from augmented data (as discussed in Section 4), we strictly apply data augmentation in Q-value estimation of the current state st, without applying data augmentation to the successor state st+1 used in Eq. 1 for bootstrapping with Q tgt
(and ⇡✓ if applicable), which addresses Pitfall 1. If ⇡✓ is learned (i.e., SVEA is implemented with an actor-critic algorithm), we also optimize it strictly from unaugmented data. To mitigate over-regularization in optimization of f✓ and Q✓ (Pitfall 2), we further employ a modified Q-objective that leverages both augmented and unaugmented data, which we introduce in the following section.

heading: 5.2 Learning Objective

text:
Our method redefines the temporal difference objective from Eq. 1 to better leverage data augmentation. First, recall that qtgt
t = r(st,at) + maxa0t Q
tgt (f tgt (st+1),a0). Instead of learning to predict
q tgt t
only from state st, we propose to minimize a nonnegative linear combination of LQ over two individual data streams, st and s aug t = ⌧(st, ⌫), ⌫ ⇠ V , which we define as the objective
L SVEA Q (✓, ) , ↵LQ st, q tgt t ; ✓,
+ LQ saug t , q tgt t ; ✓,
(3)
= Est,at,st+1⇠B h ↵ Q✓(f✓(st),at) qtgtt 2 2 +
Q✓(f✓(saugt ),at) q tgt t 2 2
i , (4)
where ↵, are constant coefficients that balance the ratio of the unaugmented and augmented data streams, respectively, and qtgt
t is computed strictly from unaugmented data. LSVEA Q (✓, ) serves as a
data-mixing strategy that oversamples unaugmented data as an implicit variance reduction technique.
As we will verify empirically in Section 6, data-mixing is a simple and effective technique for variance reduction that works well in tandem with our proposed modifications to bootstrapping. For ↵ = , the objective in Eq. 4 can be evaluated in a single, batched forward-pass by rewriting it as:
gt = [st, ⌧(st, ⌫)]N (5) ht = ⇥ q tgt t , q tgt t ⇤ N (6)
L SVEA Q (✓, ) = Est,at,st+1⇠B, ⌫⇠V h (↵+ ) kQ✓(f✓(gt),at) htk 2 2 i , (7)
where [·]N is a concatenation operator along the batch dimension N for st, s aug t 2 RN⇥C⇥H⇥W and qtgt t 2 RN⇥1, which is illustrated as ⌦ in Figure 2. Empirically, we find ↵ = 0.5, = 0.5 to be both effective and practical to implement, which we adopt in the majority of our experiments. However, more sophisticated schemes for selecting ↵, and/or varying them as training progresses could be interesting directions for future research. If the base algorithm of choice learns a policy ⇡✓, its objective L⇡(✓) is optimized solely on unaugmented states st without changes to the objective, and a stop-grad operation is applied after f✓ to prevent non-stationary gradients of L⇡(✓) from interfering with Q-value estimation, i.e., only the objective from Eq. 4 or optionally Eq. 7 updates f✓ using stochastic gradient descent. As described in Section 5.1, parameters are updated using an exponential moving average of ✓ and a stop-grad operation is therefore similarly applied after Qtgt .
We summarize our method for ↵ = applied to a generic off-policy algorithm in Algorithm 1.
Algorithm 1 Generic SVEA off-policy algorithm (I naïve augmentation, I our modifications) ✓, ✓⇡, : randomly initialized network parameters, ✓ B Initialize to be equal to ✓ ⌘, ⇣: learning rate and momentum coefficient ↵, : loss coefficients, default: (↵ = 0.5, = 0.5)
1: for timestep t = 1...T do act: 2: at ⇠ ⇡✓ (·|f✓(st)) B Sample action from policy 3: s0
t ⇠ P(·|st,at) B Sample transition from environment
4: B B [ (st,at, r(st,at), s0t) B Add transition to replay buffer update: 5: {si,ai, r(si,ai), s0i | i = 1...N} ⇠ B B Sample batch of transitions 6: si = ⌧(si, ⌫i), s0i = ⌧(s 0 i , ⌫ 0 i ), ⌫i, ⌫0i ⇠ V I Naïve application of data augmentation 7: for transition i = 1..N do 8: ✓⇡ ✓⇡ ⌘r✓⇡L⇡ (si; ✓⇡) (if applicable) B Optimize ⇡✓ with SGD 9: qtgt
i = r(si,ai) + maxa0i Q
tgt (f tgt (s0 i ),a0 i ) B Compute Q-target
10: saug i = ⌧(si, ⌫i), ⌫i ⇠ V I Apply stochastic data augmentation 11: gi = ⇥ si, s aug i ⇤ N , hi = ⇥ q tgt i , q tgt i ⇤ N I Pack data streams 12: ✓ ✓ ⌘r✓LSVEAQ (gi, hi; ✓, ) I Optimize f✓ and Q✓ with SGD 13: (1 ⇣) + ⇣✓ B Update using EMA of ✓
6 Experiments
We evaluate both sample efficiency, asymptotic performance, and generalization of our method and a set of strong baselines using both ConvNets and Vision Transformers (ViT) [10] in tasks from DeepMind Control Suite (DMControl) [64] as well as a set of robotic manipulation tasks. DMControl offers challenging and diverse continuous control tasks and is widely used as a benchmark for image-based
RL [19, 20, 76, 59, 35, 33]. To evaluate generalization of our method and baselines, we test methods under challenging distribution shifts (as illustrated in Figure 3) from the DMControl Generalization Benchmark (DMControl-GB) [21], the Distracting Control Suite (DistractingCS) [60], as well as distribution shifts unique to the robotic manipulation environment. Code is available at https://github.com/nicklashansen/dmcontrol-generalization-benchmark.
Setup. We implement our method and baselines using SAC [18] as base algorithm, and we apply random shift augmentation to all methods by default. This makes our base algorithm equivalent to DrQ [33] when K=1,M=1; we refer to the base algorithm as unaugmented and consider stability under additional data augmentation. We use the same network architecture and hyperparameters for all methods (whenever applicable), and adopt the setup from Hansen and Wang [21]. Observations are stacks of 3 RGB frames of size 84⇥ 84⇥ 3 (and 96⇥ 96⇥ 3 in ViT experiments). In the DMControlGB and DistractingCS benchmarks, all methods are trained for 500k frames and evaluated on all 5 tasks from DMControl-GB used in prior work, and we adopt the same experimental setup for robotic manipulation. See Appendix H for hyperparameters and further details on our experimental setup.
Baselines and data augmentations. We benchmark our method against the following strong baselines: (1) CURL [59], a contrastive learning method for RL; (2) RAD that applies a random crop; (3) DrQ that applies a random shift; (4) PAD [22] that adapts to test environments using self-supervision; (5) SODA [21] that applies data augmentation in auxiliary learning; as well as a number of ablations. We compare to the K=1,M=1 setting of DrQ by default, but also provide comparison to varying K,M . We experiment with a diverse set of data augmentations proposed in previous work on RL and computer vision, namely random shift [33], random convolution (denoted conv) [36], random overlay [21], random cutout [9], Gaussian blur, random affine-jitter, and random rotation [35, 16]. We provide samples for all data augmentations in Appendix C and test environments in Appendix E.

heading: 6.1 Stability and Generalization on DMControl

text:
Stability. We evaluate the stability of SVEA and DrQ under 6 common data augmentations; results are shown in Figure 4. While the sample efficiency of DrQ degrades substantially for most augmentations, SVEA is relatively unaffected by the choice of data augmentation and improves sample efficiency in 27 out of 30 instances. While the sample efficiency of DrQ can be improved by increasing its K,M parameters, we find that DrQ requires approx. 6x wall-time to match the sample efficiency of SVEA; see Figure 6 (left). We further ablate each component of SVEA and report both training and test curves in Figure 5; we find that both components are key to SVEA’s success. Because we empirically find the conv augmentation to be particularly difficult to optimize, we provide additional stability experiments in Section 6.2 and 6.3 using this augmentation. See Appendix A for additional ablations.
Generalization. We compare the test performance of SVEA to 5 recent state-of-the-art methods for image-based RL on the color_hard and video_easy benchmarks from DMControl-GB (results in Table 1), as well as the extremely challenging DistractingCS benchmark, where camera pose, background, and colors are continually changing throughout an episode (results in Figure 6 (right)). We here use conv and overlay augmentations for fair comparison to SODA, and we report additional results on the video_hard benchmark in Appendix F. SVEA outperforms all methods considered in 12 out of 15 instances on DMControl-GB, and at a lower computational cost than CURL, PAD, and SODA that all learn auxiliary tasks. On DistractingCS, we observe that SVEA improves generalization by 42% at low intensity, and its generalization degrades significantly slower than DrQ for high intensities. While generalization depends on the particular choice of data augmentation and test environments, this is an encouraging result considering that SVEA enables efficient policy learning with stronger augmentations than previous methods.
6.2 RL with Vision Transformers
Vision Transformers (ViT) [10] have recently achieved impressive results on downstream tasks in computer vision. We replace all convolutional layers from the previous experiments with a 4-layer ViT encoder that operates on raw pixels in 8⇥ 8 space-time patches, and evaluate our method using data augmentation in conjunction with ViT encoders. Importantly, we design the ViT architecture such that it roughly matches our CNN encoder in terms of learnable parameters. The ViT encoder is trained from scratch using RL, and we use the same experimental setup as in our ConvNet experiments. In particular, it is worth emphasizing that both our ViT and CNN encoders are trained using Adam [32] as optimizer and without weight decay. See Figure 7 (top) for an architectural overview, and refer to Appendix H for additional implementation details.
Our training and test results are shown in Figure 7 (bottom). We are, to the best of our knowledge, the first to successfully solve image-based RL tasks without CNNs. We observe that DrQ overfits significantly to the training environment compared to its CNN counterpart (94 test return on color_hard for DrQ with ViT vs. 569 with a ConvNet on the Walker, walk task). SVEA achieves comparable sample efficiency and improves generalization by 706% and 233% on Walker, walk and Cartpole, swingup, respectively, over DrQ, while DrQ + conv remains unstable. Interestingly, we observe that our ViT-based implementation of SVEA achieves a mean episode return of 877 on the color_hard benchmark of the challenging Walker, walk task (vs. 760 using CNNs). SVEA might therefore be a promising technique for fu-
ture research on RL with CNN-free architectures, where data augmentation appears to be especially important for generalization. We provide additional experiments with ViT encoders in Section 6.3 and make further comparison to ConvNet encoders in Appendix A.
6.3 Robotic Manipulation
We additionally consider a set of goal-conditioned robotic manipulation tasks using a simulated Kinova Gen3 arm: (i) reach, a task in which the robot needs to position its gripper above a goal indicated by a red mark; (ii) reach moving target, a task similar to (i) but where the robot needs to follow a red mark moving continuously in a zig-zag pattern at a random velocity; and (iii) push, a task in which the robot needs to push a cube to a red mark. The initial configuration of gripper, object, and goal is randomized, the agent uses 2D positional control, and policies are trained using dense rewards. Observations are stacks of RGB frames with no access to state information. Training and test environments are shown in Figure 8. See Appendix G for further details and environment samples.
Results are shown in Figure 9 and Figure 10. For both CNN and ViT encoders, SVEA trained with conv augmentation has similar sample efficiency and training performance as DrQ trained without augmentation, while DrQ + conv exhibits poor sample efficiency and fails to solve the push task. Generalization results are shown in Table 2. We find that naïve application of data augmentation has a higher success rate in test environments than DrQ, despite being less
successful in the training environment, which we conjecture is because it is optimized only from augmented data. Conversely, SVEA achieves high success rates during both training and testing.
Conclusion. SVEA is found to greatly improve both stability and sample efficiency under augmentation, while achieving competitive generalization results. Our experiments indicate that our method scales to ViT-based architectures, and it may therefore be a promising technique for large-scale RL experiments where data augmentation is expected to play an increasingly important role.
Broader Impact. While our contribution aims to reduce computational cost of image-based RL, we remain concerned about the growing ecological and economical footprint of deep learning – and RL in particular – with increasingly large models such as ViT; see Appendix J for further discussion.
Acknowledgments and Funding Transparency. This work was supported by grants from DARPA LwLL, NSF CCF-2112665 (TILOS), NSF 1730158 CI-New: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI), NSF ACI-1541349 CC*DNI Pacific Research Platform, NSF grant IIS-1763278, NSF CCF-2112665 (TILOS), as well as gifts from Qualcomm, TuSimple and Picsart.

title: Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation