summaries: Learning representations using self-supervision requires domain expertise to identify diverse transformations of the data samples that label preserving. This can be expensive and hard to obtain in many data modalities. The paper proposes to automate this by learning to generate transformations tailed to each modality and sample. Specifically, an adversarial strategy is applied to learning transformations that are close to the original view in the input space but hard to classify for the self-supervision encoder. The paper presents a generative model to automatically generate data that is needed for contrastive learning, with a focus on the SimCLR framework, while the method itself is general. Experiments were conducted across multiple modalities, including image, speech and wearable sensor data. The results demonstrate the effectiveness of the proposed model as compared to data augmentation methods relying on human domain knowledge.  The paper proposes a method for automatic generation of data views for contrastive self-supervised learning of representations. The method consists of learning an adversarial perturbation model that aims to maximize the distance between the original image and its perturbed views in the space of learned representations. To avoid collapse to a completely information-destroying perturbation model, authors propose to limit the perturbation strength in terms of the $l_p$ norm of the added noise. Authors apply their method on various image, speech and wearable sensor datasets where the proposed approach provides an improvement over other methods.
abstractText: Many recent methods for unsupervised representation learning train models to be invariant to different “views,” or distorted versions of an input. However, designing these views requires considerable trial and error by human experts, hindering widespread adoption of unsupervised representation learning methods across domains and modalities. To address this, we propose viewmaker networks: generative models that learn to produce useful views from a given input. Viewmakers are stochastic bounded adversaries: they produce views by generating and then adding an `p-bounded perturbation to the input, and are trained adversarially with respect to the main encoder network. Remarkably, when pretraining on CIFAR-10, our learned views enable comparable transfer accuracy to the welltuned SimCLR augmentations—despite not including transformations like cropping or color jitter. Furthermore, our learned views significantly outperform baseline augmentations on speech recordings (+9 points on average) and wearable sensor data (+17 points on average). Viewmaker views can also be combined with handcrafted views: they improve robustness to common image corruptions and can increase transfer performance in cases where handcrafted views are less explored. These results suggest that viewmakers may provide a path towards more general representation learning algorithms—reducing the domain expertise and effort needed to pretrain on a much wider set of domains. Code is available at https://github.com/alextamkin/viewmaker. Figure 1: Viewmaker networks generate complex and diverse input-dependent views for unsupervised learning. Examples shown are for CIFAR-10. Original image in center with pink border.
authors.name: Alex Tamkin
authors.name: Mike Wu
authors.name: Noah Goodman
references.authors: Nasir Ahmed T Natarajan Kamisetty R Rao
references.title: Discrete cosine transform
references.venue: IEEE transactions on Computers,
references.authors: Antreas Antoniou Amos Storkey Harrison Edwards
references.title: Data augmentation generative adversarial networks, 2017
references.authors: Philip Bachman R Devon Hjelm William Buchwalter
references.title: Learning representations by maximizing mutual information across views
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Suzanna Becker Geoffrey E Hinton
references.title: Self-organizing neural network that discovers surfaces in random-dot stereograms
references.venue: Nature, 355(6356):161–163,
references.authors: Sören Becker Marcel Ackermann Sebastian Lapuschkin Klaus-Robert Müller Wojciech Samek
references.title: Interpreting and explaining deep neural networks for classification of audio signals, 2018
references.authors: Yoshua Bengio Aaron Courville Pascal Vincent
references.title: Representation learning: A review and new perspectives
references.venue: IEEE transactions on pattern analysis and machine intelligence,
references.authors: Lukas Biewald
references.title: Experiment tracking with weights and biases, 2020
references.venue: URL https://www. wandb.com/. Software available from wandb.com
references.authors: Avishek Joey Bose Huan Ling Yanshuai Cao
references.title: Adversarial contrastive estimation
references.venue: arXiv preprint arXiv:1805.03642,
references.authors: Mathilde Caron Ishan Misra Julien Mairal Priya Goyal Piotr Bojanowski Armand Joulin
references.title: Unsupervised learning of visual features by contrasting cluster assignments
references.venue: arXiv preprint arXiv:2006.09882,
references.authors: Olivier Chapelle Jason Weston Léon Bottou Vladimir Vapnik
references.title: Vicinal risk minimization
references.venue: In Advances in neural information processing systems,
references.authors: Ting Chen Simon Kornblith Mohammad Norouzi Geoffrey Hinton
references.title: A simple framework for contrastive learning of visual representations
references.venue: arXiv preprint arXiv:2002.05709,
references.authors: Ting Chen Simon Kornblith Kevin Swersky Mohammad Norouzi Geoffrey Hinton
references.title: Big selfsupervised models are strong semi-supervised learners
references.venue: arXiv preprint arXiv:2006.10029,
references.authors: Xinlei Chen Haoqi Fan Ross Girshick Kaiming He
references.title: Improved baselines with momentum contrastive learning
references.venue: arXiv preprint arXiv:2003.04297,
references.authors: Ekin D Cubuk Barret Zoph Dandelion Mane Vijay Vasudevan Quoc V Le
references.title: Autoaugment: Learning augmentation policies from data
references.venue: arXiv preprint arXiv:1805.09501,
references.authors: Ekin D Cubuk Barret Zoph Jonathon Shlens Quoc V Le
references.title: Randaugment: Practical automated data augmentation with a reduced search space
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,
references.authors: Terrance DeVries Graham W. Taylor
references.title: Dataset augmentation in feature space, 2017
references.authors: Jeff Donahue Karen Simonyan
references.title: Large scale adversarial representation learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Jeff Donahue Philipp Krähenbühl Trevor Darrell
references.title: Adversarial feature learning
references.venue: arXiv preprint arXiv:1605.09782,
references.authors: Alexey Dosovitskiy Jost Tobias Springenberg Martin Riedmiller Thomas Brox
references.title: Discriminative unsupervised feature learning with convolutional neural networks. In Advances in neural information processing
references.authors: Raphael Gontijo-Lopes Sylvia J Smullin Ekin D Cubuk Ethan Dyer
references.title: Affinity and diversity: Quantifying mechanisms of data augmentation
references.venue: arXiv preprint arXiv:2002.08973,
references.authors: Ian Goodfellow
references.title: Nips 2016 tutorial: Generative adversarial networks
references.venue: arXiv preprint arXiv:1701.00160,
references.authors: Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: In Advances in neural information processing systems,
references.authors: Jean-Bastien Grill Florian Strub Florent Altché Corentin Tallec Pierre H Richemond Elena Buchatskaya Carl Doersch Bernardo Avila Pires Zhaohan Daniel Guo Mohammad Gheshlaghi Azar
references.title: Bootstrap your own latent: A new approach to self-supervised learning
references.venue: arXiv preprint arXiv:2006.07733,
references.authors: Raia Hadsell Sumit Chopra Yann LeCun
references.title: Dimensionality reduction by learning an invariant mapping
references.venue: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06),
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.authors: Kaiming He Haoqi Fan Yuxin Wu Saining Xie Ross Girshick
references.title: Momentum contrast for unsupervised visual representation learning
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Dan Hendrycks Thomas Dietterich
references.title: Benchmarking neural network robustness to common corruptions and perturbations
references.venue: arXiv preprint arXiv:1903.12261,
references.authors: Dan Hendrycks Steven Basart Norman Mu Saurav Kadavath Frank Wang Evan Dorundo Rahul Desai Tyler Zhu Samyak Parajuli Mike Guo
references.title: The many faces of robustness: A critical analysis of out-of-distribution generalization
references.venue: arXiv preprint arXiv:2006.16241,
references.authors: Daniel Ho Eric Liang Xi Chen Ion Stoica Pieter Abbeel
references.title: Population based augmentation: Efficient learning of augmentation policy schedules
references.venue: In International Conference on Machine Learning,
references.authors: Justin Johnson Alexandre Alahi Li Fei-Fei
references.title: Perceptual losses for real-time style transfer and super-resolution
references.venue: Computer Vision – ECCV
references.authors: Eugene Kharitonov Morgane Rivière Gabriel Synnaeve Lior Wolf Pierre-Emmanuel Mazaré Matthijs Douze Emmanuel Dupoux
references.title: Data augmenting contrastive learning of speech representations in the time domain
references.authors: Minseon Kim Jihoon Tack Sung Ju Hwang
references.title: Adversarial self-supervised contrastive learning, 2020
references.authors: György Kovács László Tóth Dirk Van Compernolle Sriram Ganapathy
references.title: Increasing the robustness of cnn acoustic models using autoregressive moving average spectrogram features and channel dropout
references.venue: Pattern Recognition Letters,
references.authors: Alex Krizhevsky
references.title: Learning multiple layers of features from tiny images
references.venue: Technical report,
references.authors: Oscar D Lara Miguel A Labrador
references.title: A survey on human activity recognition using wearable sensors
references.venue: IEEE communications surveys & tutorials,
references.authors: Siddique Latif Rajib Rana Sara Khalifa Raja Jurdak Junaid Qadir Björn W Schuller
references.title: Deep representation learning in speech processing: Challenges, recent advances, and future trends
references.authors: Yann LeCun Léon Bottou Yoshua Bengio Patrick Haffner
references.title: Gradient-based learning applied to document recognition
references.venue: Proceedings of the IEEE,
references.authors: Sungbin Lim Ildoo Kim Taesup Kim Chiheon Kim Sungwoong Kim
references.title: Fast autoaugment
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Tsung-Yi Lin Michael Maire Serge Belongie James Hays Pietro Perona Deva Ramanan Piotr Dollár C Lawrence Zitnick
references.title: Microsoft coco: Common objects in context
references.venue: In European conference on computer vision,
references.authors: Raphael Gontijo Lopes Dong Yin Ben Poole Justin Gilmer Ekin D Cubuk
references.title: Improving robustness without sacrificing accuracy with patch gaussian augmentation
references.authors: Loren Lugosch Mirco Ravanelli Patrick Ignoto Vikrant Singh Tomar Yoshua Bengio
references.title: Speech model pre-training for end-to-end spoken language understanding, 2019
references.authors: Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras Adrian Vladu
references.title: Towards deep learning models resistant to adversarial attacks
references.venue: arXiv preprint arXiv:1706.06083,
references.authors: Matthias Minderer Olivier Bachem Neil Houlsby Michael Tschannen
references.title: Automatic shortcut removal for self-supervised representation learning
references.venue: arXiv preprint arXiv:2002.08822,
references.authors: Ishan Misra Laurens van der Maaten
references.title: Self-supervised learning of pretext-invariant representations
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Takeru Miyato Shin-ichi Maeda Masanori Koyama Shin Ishii
references.title: Virtual adversarial training: a regularization method for supervised and semi-supervised learning
references.venue: IEEE transactions on pattern analysis and machine intelligence,
references.authors: Fernando Moya Rueda René Grzeszick Gernot A Fink Sascha Feldhorst Michael Ten Hompel
references.title: Convolutional neural networks for human activity recognition using body-worn sensors
references.venue: In Informatics,
references.authors: Arsha Nagrani Joon Son Chung Andrew Zisserman
references.title: Voxceleb: a large-scale speaker identification
references.authors: Aaron van den Oord Yazhe Li Oriol Vinyals
references.title: Representation learning with contrastive predictive coding
references.venue: arXiv preprint arXiv:1807.03748,
references.authors: Sinno Jialin Pan Qiang Yang
references.title: A survey on transfer learning
references.venue: IEEE Transactions on knowledge and data engineering,
references.authors: Vassil Panayotov Guoguo Chen Daniel Povey Sanjeev Khudanpur
references.title: Librispeech: An asr corpus based on public domain audio
references.venue: books. pp. 5206–5210,
references.authors: Daniel S Park William Chan Yu Zhang Chung-Cheng Chiu Barret Zoph Ekin D Cubuk Quoc V Le
references.title: Specaugment: A simple data augmentation method for automatic speech recognition
references.authors: Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury Gregory Chanan Trevor Killeen Zeming Lin Natalia Gimelshein Luca Antiga
references.title: Pytorch: An imperative style, highperformance deep learning library
references.venue: In Advances in neural information processing systems,
references.authors: Luis Perez Jason Wang
references.title: The effectiveness of data augmentation in image classification using deep learning
references.venue: arXiv preprint arXiv:1712.04621,
references.authors: Senthil Purushwalkam Abhinav Gupta
references.title: Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases
references.venue: arXiv preprint arXiv:2007.13916,
references.authors: Aditi Raghunathan Jacob Steinhardt Percy Liang
references.title: Certified defenses against adversarial examples
references.venue: arXiv preprint arXiv:1801.09344,
references.authors: Aditi Raghunathan Sang Michael Xie Fanny Yang John C Duchi Percy Liang
references.title: Adversarial training can hurt generalization
references.authors: Alexander J Ratner Henry Ehrenberg Zeshan Hussain Jared Dunnmon Christopher Ré
references.title: Learning to compose domain-specific transformations for data augmentation
references.venue: In Advances in neural information processing systems,
references.authors: Attila Reiss Didier Stricker
references.title: Introducing a new benchmarked dataset for activity monitoring
references.venue: In 2012 16th International Symposium on Wearable Computers,
references.authors: Olaf Ronneberger Philipp Fischer Thomas Brox
references.title: U-net: Convolutional networks for biomedical image segmentation
references.venue: In International Conference on Medical image computing and computerassisted intervention,
references.authors: Mehdi SM Sajjadi Giambattista Parascandolo Arash Mehrjou Bernhard Schölkopf
references.title: Tempered adversarial networks
references.venue: arXiv preprint arXiv:1802.04374,
references.authors: Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna Dumitru Erhan Ian Goodfellow Rob Fergus
references.title: Intriguing properties of neural networks
references.venue: arXiv preprint arXiv:1312.6199,
references.authors: Yonglong Tian Dilip Krishnan Phillip Isola
references.title: Contrastive multiview coding
references.venue: arXiv preprint arXiv:1906.05849,
references.authors: Yonglong Tian Chen Sun Ben Poole Dilip Krishnan Cordelia Schmid Phillip Isola
references.title: What makes for good views for contrastive learning
references.venue: arXiv preprint arXiv:2005.10243,
references.authors: László Tóth György Kovács Dirk Van Compernolle
references.title: A perceptually inspired data augmentation method for noise robust cnn acoustic models
references.venue: In International Conference on Speech and Computer,
references.authors: Toan Tran Trung Pham Gustavo Carneiro Lyle Palmer Ian Reid
references.title: A bayesian data augmentation approach for learning deep models, 2017
references.authors: Eleni Triantafillou Tyler Zhu Vincent Dumoulin Pascal Lamblin Utku Evci Kelvin Xu Ross Goroshin Carles Gelada Kevin Swersky Pierre-Antoine Manzagol Hugo Larochelle
references.title: Metadataset: A dataset of datasets for learning to learn from few examples, 2019
references.authors: Riccardo Volpi Hongseok Namkoong Ozan Sener John Duchi Vittorio Murino Silvio Savarese
references.title: Generalizing to unseen domains via adversarial data augmentation, 2018
references.authors: Pete Warden
references.title: Speech commands: A dataset for limited-vocabulary speech recognition, 2018
references.authors: Eric Wong J. Zico Kolter
references.title: Learning perturbation sets for robust machine learning, 2020
references.authors: Mike Wu Chengxu Zhuang Milan Mosse Daniel Yamins Noah Goodman
references.title: On mutual information in contrastive learning for visual representations
references.venue: arXiv preprint arXiv:2005.13149,
references.authors: Zhirong Wu Yuanjun Xiong Stella Yu Dahua Lin
references.title: Unsupervised feature learning via nonparametric instance-level discrimination
references.venue: arXiv preprint arXiv:1805.01978,
references.authors: Chaowei Xiao Bo Li Jun-Yan Zhu Warren He Mingyan Liu Dawn Song
references.title: Generating adversarial examples with adversarial networks, 2018
references.authors: Han Xiao Kashif Rasul Roland Vollgraf
references.title: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms
references.venue: arXiv preprint arXiv:1708.07747,
references.authors: Sangdoo Yun Dongyoon Han Seong Joon Oh Sanghyuk Chun Junsuk Choe Youngjoon Yoo
references.title: Cutmix: Regularization strategy to train strong classifiers with localizable features
references.venue: In Proceedings of the IEEE International Conference on Computer Vision,
references.authors: Hongyi Zhang Moustapha Cisse Yann N Dauphin David Lopez-Paz
references.title: mixup: Beyond empirical risk minimization
references.venue: arXiv preprint arXiv:1710.09412,
references.authors: Xinyu Zhang Qiang Wang Jian Zhang Zhao Zhong
references.title: Adversarial autoaugment
references.venue: arXiv preprint arXiv:1912.11188,
references.authors: Chengxu Zhuang Alex Lin Zhai Daniel Yamins
references.title: Local aggregation for unsupervised learning of visual embeddings
references.venue: In Proceedings of the IEEE International Conference on Computer Vision,
sections.heading: 1 INTRODUCTION
sections.text: Unsupervised representation learning has made significant recent strides, including in computer vision, where view-based methods have enabled strong performance on benchmark tasks (Wu et al., 2018; Oord et al., 2018; Bachman et al., 2019; Zhuang et al., 2019; Misra & Maaten, 2020; He et al., 2020; Chen et al., 2020a). Views here refer to human-defined data transformations, which target capabilities or invariances thought to be useful for transfer tasks. In particular, in contrastive learning of visual representations, models are trained to maximize the mutual information between different views of an image, including crops, blurs, noise, and changes to color and contrast (Bachman et al.,
  2019; Chen et al., 2020a). Much work has investigated the space of possible image views (and their compositions) and understanding their effects on transfer learning (Chen et al., 2020a; Wu et al., 2020; Tian et al., 2019; Purushwalkam & Gupta, 2020).
  The fact that views must be hand designed is a significant limitation. While views for image classification have been refined over many years, new views must be developed from scratch for new modalities. Making matters worse, even within a modality, different domains may have different optimal views (Purushwalkam & Gupta, 2020). Previous studies have investigated the properties of good views through the lens of mutual information (Tian et al., 2020; Wu et al., 2020), but a broadly-applicable approach for learning views remains unstudied.
  In this work, we present a general method for learning diverse and useful views for contrastive learning. Rather than searching through possible compositions of existing view functions (Cubuk et al., 2018; Lim et al., 2019), which may not be available for many modalities, our approach produces views with a generative model, called the viewmaker network, trained jointly with the encoder network. This flexibility enables learning a broad set of possible view functions, including input-dependent views, without resorting to hand-crafting or expert domain knowledge. The viewmaker network is trained adversarially to create views which increase the contrastive loss of the encoder network. Rather than directly outputting views for an image, the viewmaker instead outputs a stochastic perturbation that is added to the input. This perturbation is projected onto an `p sphere, controlling the effective strength of the view, similar to methods in adversarial robustness. This constrained adversarial training method enables the model to reduce the mutual information between different views while preserving useful input features for the encoder to learn from.
  In summary, we contribute:
  1. Viewmaker networks: to our knowledge the first modality-agnostic method to learn views for unsupervised representation learning
  2. On image data, where expert-designed views have been extensively optimized, our viewmaker-models achieve comparable transfer performance to state of the art contrastive methods while being more robust to common corruptions.
  3. On speech data, our method significantly outperforms existing human-defined views on a range of speech recognition transfer tasks.
  4. On time-series data from wearable sensors, our model significantly outperforms baseline views on the task of human activity recognition (e.g., cycling, running, jumping rope).
sections.heading: 2 RELATED WORK
sections.text: Unsupervised representation learning Learning useful representations from unlabeled data is a fundamental problem in machine learning (Pan & Yang, 2009; Bengio et al., 2013). A recently successful framework for unsupervised representation learning for images involves training a model to be invariant to various data transformations (Bachman et al., 2019; Misra & Maaten, 2020), although the idea has much earlier roots (Becker & Hinton, 1992; Hadsell et al., 2006; Dosovitskiy et al., 2014). This idea has been expanded by a number of contrastive learning approaches which push embeddings of different views, or transformed inputs, closer together, while pushing other pairs apart (Tian et al., 2019; He et al., 2020; Chen et al., 2020a;b;c), as well as non-contrastive approaches which do not explicitly push apart unmatched views (Grill et al., 2020; Caron et al., 2020). Related but more limited setups have been explored for speech, where data augmentation strategies are less explored (Oord et al., 2018; Kharitonov et al., 2020).
  Understanding and designing views Several works have studied the role of views in contrastive learning, including from a mutual-information perspective (Wu et al., 2020), in relation to specific transfer tasks (Tian et al., 2019), with respect to different kinds of invariances (Purushwalkam & Gupta, 2020), or via careful empirical studies (Chen et al., 2020a). Outside of a contrastive learning framework, Gontijo-Lopes et al. (2020) study how data augmentation aids generalization in vision models. Much work has explored different handcrafted data augmentation methods for supervised learning of images (Hendrycks et al., 2020; Lopes et al., 2019; Perez & Wang, 2017; Yun et al., 2019; Zhang et al., 2017), speech (Park et al., 2019; Kovács et al., 2017; Tóth et al., 2018; Kharitonov et al., 2020), or in feature space (DeVries & Taylor, 2017).
  Adversarial methods Our work is related to and inspired by work on adversarial methods, including the `p balls studied in adversarial robustness (Szegedy et al., 2013; Madry et al., 2017; Raghunathan et al., 2018) and training networks with adversarial objectives (Goodfellow et al., 2014; Xiao et al., 2018). Our work is also connected to the vicinal risk minimization principle (Chapelle et al., 2001) and can be interpreted as producing amortized virtual adversarial examples (Miyato et al., 2018). Previous adversarial view-based pretraining methods add adversarial noise on top of existing handcrafted views (Kim et al., 2020) or require access to specific transfer tasks during pretraining (Tian et al., 2020). In contrast, our method is more general: it is neither specialized to a particular downstream task, nor requires neither human-defined view families. Outside of multi-view learning paradigms, adversarial methods have also seen use for representation learning in GANs (Donahue et al., 2016; Donahue & Simonyan, 2019) or in choosing harder negative samples (Bose et al., 2018), as well as for data augmentation (Antoniou et al., 2017; Volpi et al., 2018; Bowles et al., 2018). Adversarial networks that perturb inputs have also been investigated to improve GAN training (Sajjadi et al., 2018) and to remove “shortcut” features (e.g., watermarks) for self-supervised pretext tasks (Minderer et al., 2020).
  Learning views Outside of adversarial approaches, our work is related to other studies that seek to learn data augmentation strategies by composing existing human-designed augmentations (Ratner et al., 2017; Cubuk et al., 2018; Zhang et al., 2019; Ho et al., 2019; Lim et al., 2019; Cubuk et al., 2020) or by modeling variations specific to the data distribution (Tran et al., 2017; Wong & Kolter, 2020). By contrast, our method requires no human-defined view functions, does not require first pretraining a generative model, and can generate perturbations beyond naturally-occurring variation observed in the training data (e.g. brightness or contrast), potentially conferring robustness benefits, as we explore in Section 4.3.
sections.heading: 3 METHOD
sections.text: In contrastive learning, the objective is to push embeddings of positive views (derived from the same input) close together, while pushing away embeddings of negative views (derived from different inputs). We focus mainly on the simple, yet performant, SimCLR contrastive learning algorithm (Chen et al., 2020a), but we also consider a memory bank-based algorithm (Wu et al., 2018) in Section 4. As our method is agnostic to the specific pretraining loss used, it is naturally compatible with other view-based algorithms such as MoCo (He et al., 2020), BYOL (Grill et al., 2020), and SwAV (Caron et al., 2020) by similarly substituting the data transformation pipeline with a viewmaker network.
  Formally, given a batch of N pairs of positive views (i, j) the SimCLR loss is
  L = 1 2N
  NX
  k=1 [`(2k 1, 2k) + `(2k, 2k 1)] where `(i, j) = log exp(si,j/⌧)P2N k=1 [k 6=i] exp(si,k/⌧)
  and sa,b is the cosine similarity of the embeddings of views a and b.
  We generate views by perturbing examples with a viewmaker network V , trained jointly with the main encoder network M . There are three attributes desirable for useful perturbations, each of which motivates an aspect of our method:
  1. Challenging: The perturbations should be complex and strong enough that an encoder must develop useful representations to perform the self-supervised task. We accomplish this by generating perturbations with a neural network that is trained adversarially to increase the loss of the encoder network. Specifically, we use a neural network that ingests the input X and outputs a view X + V (X).
  2. Faithful: The perturbations must not make the encoder task impossible, being so strong that they destroy all features of the input. For example, perturbations should not be able to zero out the input, making learning impossible. We accomplish this by constraining the perturbations to an `p sphere around the original input. `p constraints are common in the adversarial robustness literature where perturbations are expected to be indistinguishable. In our experiments, we find the best results are achieved with an `1 sphere, which grants the viewmaker a distortion budget that it can spend on a small perturbation for a large part of the input or a more extreme perturbation for a smaller portion.
  3. Stochastic: The method should be able to generate a variety of perturbations for a single input, as the encoder objective requires contrasting two different views of an input against each other. To do this, we inject random noise into the viewmaker, such that the model can learn a stochastic function that produces a different perturbed input each forward pass.
  Figure 2 summarizes our method. The encoder and viewmaker are optimized in alternating steps to minimize and maximize L, respectively. We use an image-to-image neural network as our viewmaker network, with an architecture adapted from work on style transfer (Johnson et al., 2016). See the Appendix for more details. This network ingests the input image and outputs a perturbation that is constrained to an `1 sphere. The sphere’s radius is determined by the volume of the input tensor times a hyperparameter ✏, the distortion budget, which determines the strength of the applied perturbation. This perturbation is added to the input image and optionally clamped in the case of images to ensure all pixels are in [0, 1]. Algorithm 1 describes this process precisely.
  Algorithm 1: Generating viewmaker views Input: Viewmaker network V , C ⇥W ⇥H image X, `1 distortion budget ✏, noise Output: Perturbed C ⇥W ⇥H image X P V (X, ) // generate perturbation
sections.heading: P ✏CWH|P |1 P // project to `1 sphere
sections.text: X X + P // apply perturbation X clamp(X, 0, 1) // clamp (images only)
sections.heading: 4 IMAGES
sections.text: We begin by applying the viewmaker to contrastive learning for images. In addition to SimCLR (Chen et al., 2020a), we also consider a memory bank-based instance discrimination framework (Wu et al., 2018, henceforth InstDisc).
  We pretrain ResNet-18 (He et al., 2015) models on CIFAR-10 (Krizhevsky, 2009) for 200 epochs with a batch size of 256. We train a viewmaker-encoder system with a distortion budget of ✏ = 0.05. We tried distortion budgets ✏ 2 {0.1, 0.05, 0.02} and found 0.05 to work best; however, we anticipate that further tuning would yield additional gains. As we can see in Figure 1, the learned views are diverse, consisting of qualitatively different kinds of perturbations and affecting different parts of the input. We compare the resulting encoder representations with a model trained with the expert views used for SimCLR, comprised of many human-defined transformations targeting different kinds of invariances useful for image classification: cropping-and-resizing, blurring, horizontal flipping, color dropping, and shifts in brightness, contrast, saturation, and hue (Chen et al., 2020a).
sections.heading: 4.1 TRANSFER RESULTS ON IMAGE CLASSIFICATION TASKS
sections.text: We evaluate our models on CIFAR-10, as well as eleven transfer tasks including MetaDataset (Triantafillou et al., 2019), MSCOCO (Lin et al., 2014), MNIST (LeCun et al., 1998), and FashionMNIST (Xiao et al., 2017). We use the standard linear evaluation protocol, which trains a logistic
  regression on top of representations from a frozen model. We apply the same views as in pretraining, freezing the final viewmaker when using learned views; we apply no views during validation. Table 1 shows our results, indicating comparable overall performance with SimCLR and InstDisc, all without the use of human-crafted view functions. This performance is noteworthy as our `1 views cannot implement cropping-and-rescaling, which was shown to be the most important view function in Chen et al. (2020a). We speculate that the ability of the viewmaker to implement partial masking of an image may enable a similar kind of spatial information ablation as cropping.
sections.heading: 4.1.1 COMPARISON TO RANDOM `1 NOISE
sections.text: Is random noise sufficient to produce domain-agnostic views? To assess how important adversarial training is to the quality of the learned representations, we perform an ablation where we generate views by adding Gaussian noise normalized to the same ✏ = 0.05 budget as used in the previous section. Transfer accuracy on CIFAR-10 is significantly hurt by this ablation, reaching 52.01% for a SimCLR model trained with random noise views compared to 84.50% for our method, demonstrating the importance of adversarial training to our method.
sections.heading: 4.1.2 THE IMPORTANCE OF INTER-PATCH MUTUAL INFORMATION AND CROPPING VIEWS
sections.text: Cropping-and-resizing has been identified as a crucial view function when pretraining on ImageNet (Chen et al., 2020a). However, what properties of a pretraining dataset make cropping useful? We hypothesize that such a dataset must have images whose patches have high mutual information. In other words, there must be some way for the model to identify that different patches of the same image come from the same image. While this may be true for many object or scene recognition datasets, it may be false for other important pretraining datasets, including medical or satellite imagery, where features of interest are isolated to particular parts of the image.
  To investigate this hypothesis, we modify the CIFAR-10 dataset to reduce the inter-patch mutual information by replacing each 16x16 corner of the image with the corner from another image in the training dataset (see Figure 3 for an example). Thus, random crops on this dataset, which we call CIFAR-10-Corners, will often contain completely unrelated information. When pretrained on CIFAR-10-Corners, expert views achieve 63.3% linear evaluation accuracy on the original CIFAR10 dataset, while viewmaker views achieve 68.8%. This gap suggests that viewmaker views are less reliant on inter-patch mutual information than the expert views.
sections.heading: 4.2 COMBINING VIEWMAKER AND HANDCRAFTED VIEWS
sections.text: Can viewmakers improve performance in cases where some useful handcrafted views have already been identified? Chen et al. (2020a) show that views produced through cropping are significantly improved by a suite of color-based augmentations, which they argue prevents the network from relying solely on color statistics to perform the contrastive task. Here, we show that viewmaker networks also enable strong gains when added on top of cropping and horizontal flipping views when pretraining on CIFAR-10—without any domain-specific knowledge. Alone, this subset of
  handcrafted augmentations achieves 73.2% linear evaluation accuracy on CIFAR-10. Combining these views with learned viewmaker perturbations (✏ = 0.05) achieves 83.1%.1 This suggests that viewmakers can significantly improve representation learning even in cases where some domainspecific views have already been developed.
sections.heading: 4.3 ROBUSTNESS TO COMMON CORRUPTIONS
sections.text: Image classification systems should behave robustly even when the data distribution is slightly different from that seen during training. Does using a viewmaker improve robustness against common types of corruptions not experienced at train time? To answer this, we evaluate both learned views, expert views, and their composition on the CIFAR-10-C dataset (Hendrycks & Dietterich, 2019), which assesses robustness to corruptions like snow, pixelation, and blurring. In this setting, corruptions are applied only at test time, evaluating whether the classification system is robust to some types of corruptions to which humans are robust.
  When considering methods in isolation, SimCLR augmentations result in less of an accuracy drop from clean to corrupted data compared to our learned views, as shown in Table 4a. This gap is expected, as the expert views overlap significantly with the CIFAR-10-C corruptions: both include blurring, brightness, and contrast transformations. Interestingly, however, when we train a viewmaker network while also applying expert augmentations (“Combined,” Table 4a), we can further improve the robust accuracy, with notable gains on noise and glass blur corruptions (Figure 4b). This is noteworthy, as our learned views have no explicit overlap with the CIFAR-10-C corruptions, unlike the expert augmentations.2 In the Combined setting, we use a distortion budget of ✏ = 0.01,
  1We did not see additional gains from using viewmakers on top of the full, well-optimized set of SimCLR augmentations.
  2We do notice a smaller decline in contrast corruption accuracy, possibly due to interactions between changing pixel magnitudes and the `p constraint.
  which we find works better than ✏ = 0.05, likely because combining the two augmentations at their full strength would make the learning task too difficult.
  These results suggest that learned views are a promising avenue for improving robustness in selfsupervised learning models.
sections.heading: 5 SPEECH
sections.text: Representation learning on speech data is an emerging and important research area, given the large amount of available unlabeled data and the increasing prevalence of speech-based human-computer interaction (Latif et al., 2020). However, compared to images, there is considerably less work on self-supervised learning and data augmentations for speech data. Thus, it is a compelling setting to investigate whether viewmaker augmentations are broadly applicable across modalities.
sections.heading: 5.1 SELF-SUPERVISED LEARNING SETUP
sections.text: We adapt the contrastive learning setup from SimCLR (Chen et al., 2020a). Training proceeds largely the same as for images, but the inputs are 2D log mel spectrograms. We consider both view functions applied in the time-domain before the STFT, including noise, reverb, pitch shifts, and changes in loudness (Kharitonov et al., 2020), as well as spectral views, which involve masking or noising different parts of the spectrogram (Park et al., 2019). To generate learned views, we pass the spectrogram as input to the viewmaker. We normalize the spectrogram to mean zero and variance one before passing it through the viewmaker, and do not clamp the resulting perturbed spectrogram. See the Appendix for more details. We train on the Librispeech dataset (Panayotov et al., 2015) for 200 epochs, and display some examples of learned views in the Appendix.
sections.heading: 5.2 SPEECH CLASSIFICATION RESULTS
sections.text: We evaluate on three speech classification datasets: Fluent Speech Commands (Lugosch et al., 2019), Google Speech Commands (Warden, 2018), and spoken digit classification (Becker et al., 2018), as well as speaker classification on VoxCeleb (Nagrani et al., 2017) and Librispeech (Panayotov et al., 2015), all using the linear evaluation protocol for 100 epochs. In Table 2, we report results with both the same distortion budget ✏ = 0.05 as in the image domain, as well as a larger ✏ = 0.1, for comparison. Both versions significantly outperform the preexisting waveform and spectral augmentations, with a +9 percentage point improvement on average for the ResNet-18 (✏ = 0.05) viewmaker model over the best expert views. The gains for real-world tasks such as command identification are compelling. One notable exception is the task of LibriSpeech speaker identification. Since LibriSpeech is the same dataset the model was pretrained on, and this effect is not replicated on VoxCeleb1, the other speaker classification dataset, we suspect the model may be picking up on dataset-specific artifacts (e.g. background noise, microphone type) which may make the speaker
  ID task artificially easier. An interesting possibility is that the worse performance of viewmaker views may result from the model being able to identify and ablate such spurious correlations in the spectrograms.
sections.heading: 6 WEARABLE SENSOR DATA
sections.text: To further validate that our method for learning views is useful across different modalities, we consider time-series data from wearable sensors. Wearable sensor data has a broad range of applications, including health care, entertainment, and education (Lara & Labrador, 2012). We specifically consider whether viewmaker views improve representation learning for the task of human activity recognition (HAR), for example identifying whether a user is jumping rope, running, or cycling.
sections.heading: 6.1 SELF-SUPERVISED LEARNING SETUP
sections.text: We consider the Pamap2 dataset (Reiss & Stricker, 2012), a dataset of 12 different activities performed by 9 participants. Each activity contains 52 different time series, including heart rate, accelerometer, gyroscope, and magnetometer data collected from sensors on the ankle, hand, and chest (all sampled at 100Hz, except heart rate, which is sampled at approximately 9Hz). We linearly interpolate missing data, then take random 10s windows from subject recordings, using the same train/validation/test splits as prior work (Moya Rueda et al., 2018). To create inputs for our model, we generate a multi-channel image composed of one 32x32 log spectrogram for each sensor timeseries window. Unlike speech data, we do not use the mel scale when generating the spectrogram. We then normalize the training and validation datasets by subtracting the mean and then dividing by the standard deviation of the training dataset.
  We train with both our learned views and the spectral views (Park et al., 2019) that were most successful in the speech domain (for multi-channel spectral masking, we apply the same randomly chosen mask to all channels). We also compare against a variant of these views with spectrogram noise removed, which we find improves this baseline’s performance.
sections.heading: 6.2 SENSOR-BASED ACTIVITY RECOGNITION RESULTS
sections.text: We train a linear classifier on the frozen encoder representations for 50 epochs, reporting accuracy on the validation set. We sample 10k examples for each training epoch and 50k examples for validation. Our views significantly outperform spectral masking by 12.8 percentage points when using the same ✏ = 0.05 as image and speech, and by 16.7 points when using a larger ✏ = 0.5 (Table 3). We also find that a broad range of distortion budgets produces useful representations, although overly-aggressive budgets prevent learning (Table 3). These results provide further evidence that our method for learning views has broad applicability across different domains.
sections.heading: 6.3 SEMI-SUPERVISED EXPERIMENTS
sections.text: An especially important setting for self-supervised learning is domains where labeled data is scarce or costly to acquire. Here, we show that our method can enable strong performance when labels for only a single participant (Participant 1) out of seven are available. We compare simple supervised learning on Participant 1’s labels against linear evaluation of our best pretrained model, which was
  trained on unlabeled data from all 7 participants. The model architectures and training procedures are otherwise identical to the previous section. As Figure 4 shows, pretraining with our method on unlabeled data enables significant gains over pure supervised learning when data is scarce, and even slightly outperforms the hand-crafted views trained on all 7 participants (cf. Table 3).
sections.heading: 7 CONCLUSION
sections.text: We introduce a method for learning views for unsupervised learning, demonstrating its effectiveness through strong performance on image, speech, and wearable sensor modalities. Our novel generative model—viewmaker networks—enables us to efficiently learn views as part of the representation learning process, as opposed to relying on domain-specific knowledge or costly trial and error. There are many interesting avenues for future work. For example, while the `1 constraint is simple by design, there may be other kinds of constraints that enable richer spaces of views and better performance. In addition, viewmaker networks may find use in supervised learning, for the purposes of data augmentation or improving robustness. Finally, it is interesting to consider what happens as the viewmaker networks increase in size: do we see performance gains or robustnessaccuracy trade-offs (Raghunathan et al., 2019)? Ultimately, our work is a step towards more general self-supervised algorithms capable of pretraining on arbitrary data and domains.
sections.heading: ACKNOWLEDGEMENTS
sections.text: We would like to thank Dan Yamins, Chengxu Zhuang, Shyamal Buch, Jesse Mu, Jared Davis, Aditi Raghunathan, Pranav Rajpurkar, Margalit Glasgow, and Jesse Michel for useful discussions and comments on drafts. AT is supported by an Open Phil AI Fellowship. MW is supported by the Stanford Interdisciplinary Graduate Fellowship as the Karr Family Fellow.