text:
\section{Introduction and Motivation}
\label{sec:intro}

Self-supervised learning (SSL) is on the rise across machine learning (ML), with notable recent successes in computer vision \citep{Chen2020ASF, He2020MomentumCF, Grill2020BootstrapYO}, natural language processing (NLP), \citep{Devlin2019BERTPO, Yang2019XLNetGA, Clark2020ELECTRAPT} and speech processing \citep{Oord2018RepresentationLW, Baevski2020wav2vec2A}. SSL enables a model to acquire useful capabilities from unlabeled data; these capabilities can then be leveraged to drastically reduce the amount of labeled data needed to achieve high performance in a domain---a crucial advance given the time and expense needed to annotate datasets of millions of labels.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{img/tail.pdf}
    \caption{\textbf{Domain-agnostic SSL could reduce the need for labeled data across a long tail of domains and application areas.} Currently, developing an SSL algorithm requires considerable domain-specific trial-and-error, limiting it to domains with the most active ML communities. Advances in domain-agnostic methods could make SSL available to all domains, as well as provide scientific insights into principles underlying the success of SSL across modalities. (Figure is illustrative, not based on real data.)}
    \label{fig:longtail}
\end{figure}

However, the potential impact of SSL is arguably greatest outside of the fields where it has currently seen the most success. Medical and scientific domains, for example, are rich in unlabeled data, yet the time and expertise needed for annotation far exceeds that for computer vision or NLP. This means that methods which reduce the need for labeled data are especially impactful in these settings.

Unfortunately, the most popular SSL methods are currently domain-specific---for example, the color jitter distortions used in SimCLR \citep{Chen2020ASF} are inapplicable to black-and-white chest x-rays, and the masked language modeling task used in BERT \citep{Devlin2019BERTPO} is not directly applicable to spoken language, which is untokenized. Furthermore, these algorithms are challenging to develop, requiring costly trial-and-error by ML experts \citep{Chen2020ASF}. Unfortunately, while a great number of domains may benefit from SSL, this distribution exhibits a long tail where the vast majority of domains lack the ML expertise and resources to develop custom SSL solutions.

We argue that an appealing alternative to developing domain-specific SSL methods is to develop domain-agnostic techniques which work across a wide range of settings without extensive modification. Such domain-agnostic SSL algorithms could benefit the field in multiple ways:

\begin{enumerate}
    \item \textbf{Making SSL work out-of-the box.} The most important impact of domain-agnostic SSL would be turning SSL into an out-of-the-box technology capable of being used in any domain of interest without significant ML expertise  (Figure \ref{fig:longtail}, right). Aside from medical and scientific domains, this would also benefit the combinatorial number of multimodal settings which currently require novel algorithms to learn the relationships between modalities \citep{Lu2019ViLBERTPT, Chen2020UNITERUI, Tan2019LXMERTLC}.
    \item \textbf{Improving handcrafted SSL methods.} Several works have investigated how more general SSL methods can be combined with domain-specific knowledge (e.g. image augmentations) to provide gains  \citep{Lee2021iMIXAD, Tamkin2020ViewmakerNL, Verma2020TowardsDC}. This suggests that advances in domain-agnostic SSL could benefit popular ML domains as well  (Figure \ref{fig:longtail}, left), through combination with domain-specific methods.
    \item \textbf{Uncovering fundamental principles of SSL across domains.} Communities such as computer vision and NLP currently have relatively disjoint investigations into SSL methods; this may obscure common scientific principles underlying the success of algorithms across modalities. Research on domain-agnostic methods may discover these general principles, which could benefit all domains.
\end{enumerate}

However, despite the promise of domain-agnostic SSL, there has been no standardized way to evaluate or drive progress in a cross-cutting way among different communities. To fill this need, we propose DABS, a \textbf{D}omain \textbf{A}gnostic \textbf{B}enchmark for \textbf{S}elf-supervised learning. DABS measures how well a single SSL algorithm works on many different domains, as opposed to just one. The benchmark is comprised of seven domains representing different kinds of data:  natural images, English text, speech, chest x-rays, multichannel sensor data, multilingual text, and images with text descriptions. Each domain contains one unlabeled dataset for self-supervised learning, and at least one labeled dataset to assess \emph{transfer}: how well the SSL model can adapt its abilities to downstream tasks. Models are assessed by their average transfer learning performance on downstream tasks across domains.

We anticipate a few common questions about DABS:

\paragraph{Why do we need a benchmark for domain-agnostic SSL?}

Benchmarks catalyze progress by providing a common set of tasks, rules, and evaluation criteria for research towards a particular goal.  In this case, DABS provides a standardized way to evaluate the performance of domain-agnostic methods. Fixing the choice and preprocessing of datasets allows for clean comparisons over a range of diverse domains, enabling researchers to pinpoint what specific changes contribute to the success of different methods. Furthermore, the provided infrastructure for data processing, training, transfer, and evaluation significantly reduces the barrier to entry for other researchers interested in these questions. Without a low-friction way to evaluate algorithms in a standard way, many researchers may not bother with the significant effort needed to gather and process 25+ different datasets across distinct domains, impeding cumulative progress as a field towards domain-agnostic SSL.

\paragraph{Why might we expect there to be a good domain-agnostic method?}

Many kinds of naturally-occurring and artificial data exhibit structure which humans can exploit to learn transferrable skills  \citep{Carey2004OnTO, spelke2007core, Bonawitz2011TheDS, Dubey2018InvestigatingHP}. Human-relevant data (as opposed to white noise) is often generated by some complex generative process. For example, the PAMAP2 wearable sensor dataset \citep{Reiss2012IntroducingAN} is produced by a cascade of latent factors including human interpretation of an activity command, the bodily mechanics of the activity's execution, and the physical properties of different kinds of sensors that produce measurements. Domain-agnostic pretraining objectives may enable models to capture these latent factors if they are useful for compressing the data (e.g. via density estimation objectives like language modeling \citep{Shin2006AMT}) or distinguishing examples from one another (e.g. via contrastive learning objectives \citep{Hadsell2006DimensionalityRB}).  Furthermore, studies on transfer learning of deep networks suggests there exist useful and general "subroutines" learned by SSL models which enable the model to transfer well to new datasets \citep{Yosinski2014HowTA, Tamkin2020InvestigatingTI}.  Empirically, the recent progress of existing domain-agnostic methods \citep{Tamkin2020ViewmakerNL, Lee2021iMIXAD, Verma2020TowardsDC} is cause for optimism about the future success of this research direction.

\paragraph{What does domain-agnostic mean?}
The goal of DABS is to catalyze the creation of SSL algorithms which are useful out-of-the-box across different domains. We operationalize this goal by evaluating algorithms on a suite of seven diverse domains crossing many different fields where machine learning is used. We also propose several constraints on submissions, described in Section \ref{sec:rules}, to prevent ``overfitting'' to these domains. For example, algorithms must use a set of provided dataloaders and keep their architecture and pretraining objective constant across domains (Section \ref{sec:rules}). However, we also rely on a degree of pragmatism and collaborative ethos from users of DABS to abide by the spirit of the benchmark; for example, a ``domain agnostic'' algorithm that uses an if-statement to select domain-specific methods for each domain would likely not generalize to new domains. To this end, we will add new domains in the future as an ultimate test of the generalizability of proposed algorithms.

To summarize, our \textbf{contributions} are:
\begin{enumerate}
    \item We propose and motivate the task of domain-agnostic self-supervised learning.
    \item We present a benchmark for measuring domain-agnostic self-supervised learning, including standardized data loaders and rules for ensuring fair comparisons across submissions
    \item We present two domain-agnostic baseline algorithms and evaluate them on our benchmark, showing relatively modest improvements over baselines that were not pretrained. This suggests ample room for future methods to drive progress.
\end{enumerate}

\section{Related Work}
\paragraph{Single-domain transfer learning benchmarks}
Several works have created benchmarks from multiple datasets in a single domain, often with the aim of measuring the general understanding capabilities of a single model by measuring its performance across those tasks.  Such datasets have been developed in  natural language processing \citep{Wang2018GLUEAM, Wang2019SuperGLUEAS}, computer vision \citep{triantafillou2019meta, Zhai2019ALS, Zamir2018TaskonomyDT}, speech processing \citep{Shor2020TowardsLA, Yang2021SUPERBSP}, molecular machine learning \citep{Wu2017MoleculeNetAB}, robotics \citep{Yu2019MetaWorldAB}, graphs \citep{Hu2020OpenGB}, and reinforcement learning \citep{bellemare2013arcade}, among others.

While these datasets often focus on how a single model can adapt to multiple downstream tasks in a domain, they are typically agnostic to the specifics of the pretraining process---encouraging a ``no holds barred'' setting where larger models, datasets, and domain-specific assumptions are all utilized to increase downstream accuracy. By contrast, our goal here is to develop general techniques that can be used out-of-the-box for acquiring transferrable capabilities from unlabeled data in any domain. Thus, we hold the pretraining data fixed, allowing researchers to improve only the (domain-agnostic) pretraining algorithm, model architecture, and transfer procedure.

\paragraph{Modality-agnostic architectures}
In order for an SSL method to be usable out-of-the-box, the model architecture must be applicable in new domains without much customization. Transformers \citep{Vaswani2017AttentionIA}, originally developed for text, have recently shown promise as a more general architecture for SSL through successful extensions to computer vision \citep{Dosovitskiy2020AnII}, molecular data \citep{Schwaller2019MolecularTA, rothchild2021c5t5}, speech processing \citep{Gong2021ASTAS}, and multimodal data \citep{Lu2019ViLBERTPT, Tan2019LXMERTLC, Chen2020UNITERUI}. These approaches typically use locality assumptions about continuous data (e.g. breaking the input into patches) to map the data into a sequence of embeddings, which are then processed by the transformer. Our baseline algorithms, e-Mix and ShED, leverage similar ideas to train transformer models across all seven of our domains, however we expect and encourage future work to explore other flexible architectures, such as Perceiver \citep{Jaegle2021PerceiverGP} which relaxes these locality assumptions at the cost of increased computational demands.

\paragraph{Domain-agnostic self-supervised algorithms}
Several streams of work have recently developed more general SSL methods. Recent work in contrastive learning has sought to reduce the reliance of the objective on domain-specific augmentation functions. The most common approach seeks to find heuristic augmentations which are applicable across a wider range of domains \citep{Lee2021iMIXAD, Verma2020TowardsDC, You2020GraphCL}, while other work seeks to develop generative models which learn data-dependent distortions during training with a suitable objective \citep{Tamkin2020ViewmakerNL}. Outside of contrastive learning, masked language modeling \citep{Devlin2019BERTPO} or replaced token detection \citep{Clark2020ELECTRAPT}, have been applied to other kinds of tokenized or discrete data \citep{Iida2021TABBIEPR, Hu2020StrategiesFP, Choromanski2020MaskedLM}, but require modification when applied to continuous domains \citep{Dosovitskiy2020AnII, Liu2020MockingjayUS}. However, none of these algorithms are applicable out-of-the-box on the DABS datasets, so in this work we propose simple domain-agnostic extensions of algorithms in both of these families.

\paragraph{Transfer learning methods} Evaluating a self-supervised learning algorithm requires a method to transfer model's abilities to other tasks of interest \citep{AbuMostafa1990LearningFH, Caruana1994LearningMR, Bengio2012DeepLO}. These strategies are typically quite domain-agnostic, but involve tradeoffs between various properties, including complexity, downstream performance, and the degree to which they modify the original model. The two most common transfer strategies have historically been training simple linear classifiers on activations extracted from these pretrained models \citep{Donahue2014DeCAFAD, McCann2017LearnedIT, Peters2018DeepCW}, or finetuning, where one can often achieve higher performance by specializing the entire model to the downstream task via end-to-end training \citep{Sermanet2013PedestrianDW, Girshick2014RichFH, Radford2018ImprovingLU, Devlin2019BERTPO}. However, recently, other transfer methods have shown initial success in capturing the benefits of both these extremes, including directly specifying the task in natural language \citep{Brown2020LanguageMA}, as well as approaches that train only a small subset of parameters in the original model \citep{Frankle2020TrainingBA, BenZaken2020BitFitSP} or that inject trainable features into the input \citep{Elsayed2019AdversarialRO,Liu2021GPTUT, Lester2021ThePO} or hidden states \citep{Li2021PrefixTuningOC} of the model.
We permit and encourage users of DABS to investigate different domain-agnostic transfer methods in order to understand their tradeoffs and performance across different domains.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{img/datasets.png}
    \caption{\textbf{Different domains and datasets used in this work.} Examples from pretraining datasets of different domains (clockwise from top-left): CIFAR-10, LibriSpeech, MS COCO, PAMAP2, CheXpert, and WikiText-103.  For sensor data, each line is a reading from a different sensor.}
    \label{fig:domains}
\end{figure}

\section{Evaluating Domain-Agnostic SSL Algorithms with DABS}
\label{sec:rules}

How should we evaluate a domain-agnostic SSL method? In DABS, the ultimate goal is to produce a general out-of-the-box solution for SSL across domains---one that generalizes without much modification to arbitrary desired applications. However, one challenge is that SSL methods are comprised of many factors, including the data, pretraining objective, model architecture, and transfer method. Here we describe how the rules of DABS ensure fair comparisons across each of these:

\paragraph{Datasets} DABS consists of multiple datasets spread across seven domains (detailed in Section \ref{sec:domains}). Each domain contains an unlabeled dataset used for pretraining, and one or more labeled datasets to evaluate transfer. These transfer datasets include both labeled subsets of the pretraining dataset as well as different labeled datasets in the same domain. To establish fair comparisons across algorithms, we standardize the data loading process, ensuring the same train/test splits, resolutions, tokenizers, and other details. As our primary aim is to measure the performance of methods in a domain-agnostic setting, as opposed to competing with domain-specific methods, we also prohibit the use of data augmentations which vary between domains (e.g. cropping-and-resizing used in natural images). While this may result in lower performance on transfer metrics relative to domain-specific approaches, past work suggests that many domain-agnostic methods can be combined with domain-specific techniques to provide gains \citep{Tamkin2020ViewmakerNL, Lee2021iMIXAD, Verma2020TowardsDC}.

\paragraph{Pretraining method} The goal of an SSL objective is to enable a model to acquire general capabilities from unlabeled data. To evaluate this, a single pretraining method is used to train a model on each pretraining dataset (Figure \ref{fig:dabs}). Crucially, this method may not be changed by hand between modalities (e.g. adding auxiliary losses for text). However, we do allow adaptive methods that alter the pretraining task in a general way based on the model's interaction with the unlabeled data, e.g. by learning a generative model to produce input-dependent distortions \citep{Tamkin2020ViewmakerNL}.

\paragraph{Model architecture} While the main model architecture should be kept constant, a key challenge is that different datasets have different input types and dimensions. Some recent works, such as \citet{Jaegle2021PerceiverGP}, attempt to build more data-agnostic model architectures capable of handling different kinds of inputs, however this comes at a compute cost. We take a more permissive stance, allowing different types of data to have specialized embedding modules that convert an example from the dataset into a series of vectors. These vectors then serve as the input to a model which is otherwise held constant across the datasets.

We provide a starter set of embedding modules compatible with sequence modeling architectures such as transformers \citep{Vaswani2017AttentionIA}. However, we encourage development of other general input modules as long as their tradeoffs are made clear when comparing against other methods---including running ablation studies to isolate the effects of changing the embedding module from the effects of proposed training strategies. We also emphasize that these embedding modules should encode as few assumptions about the data as possible: the goal is to produce a general, out-of-the-box strategy for SSL. For continuous 1D and 2D inputs, we use patch embeddings \citep{Dosovitskiy2020AnII} with standardized patch sizes (see Table \ref{table:datasets}), and for text we use a standard token embedding lookup table, where the text is tokenized using a standard tokenizer.\footnote{We use the common HuggingFace \href{https://huggingface.co/transformers/model_doc/bert.html}{BertTokenizer} and \href{https://huggingface.co/transformers/model_doc/xlmroberta.html}{XLMRobertaTokenizer} for English and multilingual text, respectively.} For multimodal data, we apply the appropriate embedding modules to each input, then concatenate the resulting embedding sequences in the same order each time. These embedding modules are the only parts of the model which differ across domains, enabling the main architecture to operate identically on each embedding sequence.

\paragraph{Transfer method} The ultimate measure of an SSL model is how well it performs when its capabilities are adapted to new tasks. Crucially, transfer methods are distinct from pretraining objectives, and must be compared in their own right as first-class components of an SSL algorithm. Like pretraining techniques, transfer methods exhibit tradeoffs beyond task performance: for example, finetuning a model may produce high accuracy, but requires a separate copy of the model for each use case. Other methods, such as linear evaluation \citep{Zhang2016ColorfulIC}, in-context learning \citep{Brown2020LanguageMA}, and p-, prefix-, and prompt tuning \citep{Liu2021GPTUT, Lester2021ThePO, Li2021PrefixTuningOC} enable the same model to be reused across tasks, but may achieve worse performance in some settings. We allow any transfer method as long as it is held constant across domains and downstream tasks.\footnote{Note that a transfer method does not presuppose a particular loss function, which may in general vary across tasks. For example, one can finetune a model for both regression and classification tasks.}
\paragraph{Final evaluation metric} There are many metrics one might use to compare SSL algorithms, including downstream accuracy, speed, fairness, and cost \citep{Ethayarajh2020UtilityII}. In this work, we focus on absolute performance of the model on the given data, for a given number of pretraining steps. However, participants may also be interested in other factors, including compute or data efficiency, or the scaling coefficient of techniques \citep{Kaplan2020ScalingLF, Henighan2020ScalingLF}. We encourage users of the benchmark to consider any of these, as long as they make clear what previous work is comparable and perform ablations to identify which specific changes impacted the metric being measured.

\section{Domains and Datasets}
\label{sec:domains}

Here, we describe the domains and datasets that comprise DABS. Domains were chosen to span a mix of impactful areas, including domains with both large ML research communities (natural images, text, speech) as well as domains where methods are more nascent (medical imaging, sensor recordings, vision-and-language). Dataloading and preprocessing within each dataset has been standardized to ensure fair comparisons; more information about data processing for each modality may be found in the Appendix.

\paragraph{Natural images} Two-dimensional color images of the natural world is a deeply-studied domain in machine learning. For an unlabeled pretraining dataset, we use ImageNet \citep{Russakovsky2015ImageNetLS}, a pervasive image classification benchmark in machine learning consisting of 1.3M images from 1000 classes, including meercat, streetcar, and chocolate sauce. We measure the average transfer accuracy on several image recognition tasks commonly used to assess transfer of pretrained vision models \citep{Chen2020ASF, Grill2020BootstrapYO}:
the FGVC-Aircraft dataset \citep{maji13fine-grained},
the Caltech-UCSD Birds Dataset \citep{WelinderEtal2010},
the German Traffic Sign Recognition Benchmark \citep{Houben-IJCNN-2013},
the Describable Textures Dataset \citep[][DTD]{cimpoi14describing}, the VGG Flower Dataset \citep{Nilsback2008AutomatedFC}, and the CIFAR-10 dataset \citep{Krizhevsky2009LearningML}.

\paragraph{Speech recordings} Speech processing is another large community with significant ML presence. We pretrain using the LibriSpeech corpus \citep{Panayotov2015LibrispeechAA}, a large English-language audiobook corpus commonly used for pretraining. We evaluate transfer to several datasets, including the VoxCeleb \citep{Nagrani17} and LibriSpeech \citep{Panayotov2015LibrispeechAA} speaker recognition datasets, and the Fluent Speech Commands (action, object, and location classification) \citep[]{Lugosch2019SpeechMP}, Google Speech Commands \citep{speechcommandsv2}, and AudioMNIST \citep{becker2018interpreting} utterance classification tasks. To prepare inputs for models, we preprocess examples into log-mel spectrograms---a format which differs significantly from natural image data, and thus may pose challenges for natural image-specific SSL approaches.

\paragraph{Monolingual English Text} The discrete, tokenized nature of text data makes it very different in form from the two previous continuous domains. Historically, monolingual English text has been a dominant focus shaping the development of self-supervised pretraining in NLP \citep{Dai2015SemisupervisedSL, Howard2018UniversalLM, Peters2018DeepCW, Radford2018ImprovingLU, Devlin2019BERTPO, Brown2020LanguageMA}. To assess whether domain-agnostic approaches can match the performance of methods tailored to this well-studied domain, we consider pretraining on WikiText-103 \citep{Merity2017PointerSM}, a 100-million token English-language dataset collected from the set of \textit{Good} and \textit{Featured} Wikipedia articles. For transfer, we evaluate on the GLUE benchmark \citep{Wang2018GLUEAM}, a suite of English language tasks including natural language inference, sentiment classification, and paraphrase classification, commonly used to measure transfer of pretrained models.\footnote{The GLUE benchmark tasks are CoLA \citep{warstadt2018neural}, SST-2 \citep{Socher2013RecursiveDM}, MRPC \citep{Dolan2005AutomaticallyCA}, QQP \citep{WinNT}, STS-B \citep{cer2017semeval}, MNLI \citep{N18-1101, bowman2015large}, QNLI \citep{rajpurkar2016squad, Wang2018GLUEAM}, RTE \citep{dagan2005pascal, bar2006second, giampiccolo2007third, bentivogli2009fifth}, and WNLI \citep{levesque2012winograd}.}

\paragraph{Multilingual Text} Unfortunately, machine learning learning approaches designed with only an individual language in mind are unlikely to perform equally well across the broader range of human languages \citep{Bender2009LinguisticallyN, Bender2011OnAA, bender2019rule, ruder2020beyondenglish, Wu2020AreAL}. To assess the generality of pretraining approaches on multilingual, typologically diverse data, we consider the mC4 dataset \citep{2019t5}, a filtered multilingual  web crawl corpus.\footnote{However, we note that multilingual data is not sufficient to ensure inclusion of multicultural perspectives, and we encourage future work conducting deeper analysis and documentation of mC4, similar to \citet{Dodge2021DocumentingTE}.} For pretraining, we interleave the mC4 subsets for English, Spanish, French, German, Chinese, Korean, and Japanese, meaning that the fraction of examples seen for each language during pretraining is constant (though the kind and number of unique tokens for each dataset may differ---reflecting the heterogeneity of data availability across languages). For transfer, we evaluate on the PAWS-X tasks \citep{pawsx2019emnlp}, a set of seven adversarial paraphrase identification datasets in English, Spanish, French, German, Chinese, Korean, and Japanese.

\paragraph{Medical imaging} Medical image understanding encompasses a rich set of domains which often possess ample unlabeled data yet limited labeled data, making them ideal targets for SSL. However, the statistics of medical images can differ significantly from natural images, including lower variation across many inputs and subtler task-relevant features that indicate presence of a pathology \citep{Raghu2019TransfusionUT}. Medical imaging boasts less of an ML presence than natural images, despite the fact that many techniques developed for the former may not apply---e.g. color transformations for black-and-white scans. We focus on chest x-rays as a representative medical imaging domain. We pretrain on the large CheXpert \citep{Irvin2019CheXpertAL} dataset of chest x-rays, and assess how well the pretrained model adapts to binary multiclass classification of five observations: atelectasis, cardiomegaly, consolidation, edema, and pleural effusion.\footnote{These are known as the CheXpert ``competition tasks'' \citep{Irvin2019CheXpertAL}.} To assess transfer to other chest x-ray datasets, we also measure multiclass binary classification of eight observations in the ChestX-ray8 \citep{Wang2017ChestXRay8HC} dataset: atelectasis, cardiomegaly, effusion, infiltration, mass, nodule, pneumonia, pneumothorax.

\paragraph{Multi-channel sensor data} Many scientific applications are data-rich and show significant promise for SSL. However, many such domains have a very scarce ML presence compared to domains like natural images. As an example, we consider multi-channel sensor data from wearable devices. We use the PAMAP2 dataset \citep{Reiss2012IntroducingAN}, consisting of 52-channel sensor recordings (including accelerometer, gyroscope, and magnetometer data) recorded from different body parts as participants perform varied physical activities. We measure transfer to the labeled PAMAP2 task of human-activity recognition: classifying the activity captured in a given recording snippet (e.g. cycling or walking). Examples are arranged into 1-dimensional time series of 52-channel measurements. Thus, this modality contributes to the benchmark's coverage both in terms of shape and content.

\paragraph{Images with text descriptions} Multimodal models are an increasingly important area in machine learning. An an example domain with two modalities, we consider natural images with paired English-language text descriptions. We pretrain on image-description pairs from the COCO dataset \citep{Lin2014MicrosoftCC}. We then assess the model's ability to adapt to 1) detecting whether an MS COCO image-text pair is matching or mismatched, and 2) the Visual Question Answering task \citep{Agrawal2015VQAVQ}, reformulated as a binary task to predict whether a question-answer pair correctly describes an image.

\section{Domain-Agnostic Baseline Algorithms}
\label{sec:baselines}

A domain agnostic self-supervised algorithm is comprised of a domain-agnostic encoder, pretraining objective, and transfer method for learning downstream tasks. However, to the best of our knowledge no previously-proposed method is compatible off-the-shelf with all of the domains in DABS. To establish some baseline approaches, we propose two simple, domain-agnostic algorithms that we evaluate on DABS, which we describe below and hope will serve as useful starting points for future research on domain-agnostic SSL. The core idea behind these algorithms is simple: use a small set of domain-specific embeddings modules to map inputs into an embedding space, and then define the pretraining task directly on those embeddings as opposed to the original inputs.

\subsection{Transformer Architecture}
Our algorithms use a generalized architecture based on transformers \citep{Vaswani2017AttentionIA}. These transformers take as input the sequence of embeddings obtained from the DABS embedding modules, then process them through a series of self-attention and feed-forward layers. We use a 12-layer transformer with hidden size 256, 8 attention heads, and dropout with probability 0.1. To obtain a feature vector for the input, the activations from the final layer are averaged and projected to a 128-dimensional vector. The sequence lengths vary in length depending on the dataset input dimensions and patch sizes, listed in Table \ref{table:datasets}. The same Transformer architecture is used across all experiments and is optimized with the AdamW optimizer with learning rate 1e-4 and weight decay 1e-4.

\subsection{Pretraining Objectives}
Given this common architecture, the models are then optimized with respect to a pretraining objective, which enables them to learn useful capabilities and representations from the data. We propose two baseline domain-agnostic SSL objectives, which generalize existing domain-specific methods:

\subsubsection{e-Mix: A Contrastive Embedding-Mixup Objective}
Contrastive learning and other view-matching objectives have made great strides establishing themselves as competitive or even superior alternatives to supervised pretraining in computer vision \citep{Wu2018UnsupervisedFL, Chen2020ASF, He2020MomentumCF, Grill2020BootstrapYO}. Several works have identified the reliance of these algorithms on hand-designed augmentation or ``view'' functions as a crucial impediment to applying these methods in a more domain-agnostic method. These works either \emph{learn} these view functions from scratch using a generative model \citep{Tamkin2020ViewmakerNL}, or rely on handcrafted augmentations which are more domain-general such as mixup \citep{Lee2021iMIXAD, Verma2020TowardsDC}. However, these works make assumptions about the structure of the inputs (e.g. that they are continuous) and use domain-specific encoders, leaving room for a more general solution.

We propose \emph{e-Mix}, a generalization of the i-Mix approach proposed in \citet{Lee2021iMIXAD}. In i-Mix, a batch of inputs $x_{1 \ldots N}$ is perturbed with mixup noise \citep{Zhang2018mixupBE}, where examples are additively mixed with other examples in the dataset. This produces mixed inputs $\tilde x_{1 \ldots N}$, where $\tilde x_{i} = \lambda x_i + (1 - \lambda) x_{\pi(i)}$ for some random permutation $\pi : \{1 \ldots N\} \rightarrow \{1 \ldots N\}$ and mixing coefficient $\lambda \sim \textrm{Uniform}(0.5, 1)$, chosen for each example. Then, the task is to learn an encoder $f$ such that the vector $f(\tilde x_i)$ is close to both $f(x_i)$ and $f(x_{\pi(i)})$ in proportion to their respective mixing coefficients. Formally, the loss is:
\begin{align}
    \ell_{\textrm{i-Mix}}(x, \pi, \lambda) =
    - \sum_{n=1}^{N} v_{i,n} \log \frac
    {\exp(\text{sim}(f(x_i), f(\tilde x_n)) / \tau))}
    {\sum_{k=1}^N \exp(\text{sim}(f(x_i),  f(\tilde x_k)) / \tau)}
\end{align}
where $\text{sim}$ denotes the cosine similarity, $\tau > 0$ is the temperature, and $v_{i,n}$ is a virtual label given by \begin{equation}
    v_{i,n}=
    \begin{cases}
      \lambda, & \text{if } n=i \\
      1-\lambda, & \text{if } n=\pi(i) \\
      0, & \text{otherwise}
    \end{cases}
  \end{equation}
 The main generalization provided by e-Mix is that the mixup noise is applied to the outputs of the embedding modules (i.e. the patch or token embeddings), as opposed to the inputs directly, which may not in general be continuous. This enables e-Mix to be applied without changes to each of the seven domains in the benchmark. To obtain $f(x_i)$ for a given example $x_i$, we process an input with the transformer, then mean pool the outputs along the sequence length dimension, and finally pass the resulting vector through a fully-connected layer with output size 128.

\subsubsection{SHED: A Shuffled Embedding Prediction Objective}

In contrast to the contrastive objectives that have become common in continous domains such as images and speech, token-level objectives have been more common in natural language processing. Perhaps the most paradigmatic example is masked language modeling \citep{Devlin2019BERTPO}, where random tokens in the input are either redacted or modified and the goal of the main network is to denoise the input. While this approach has been successfully applied to text, as well as other domains such as images \citep{Dosovitskiy2020AnII} and audio \citep{Liu2020MockingjayUS}, domain-specific modules are still needed to predict the inputs, which may be downsampled, averaged, or otherwise processed to improve performance.

To avoid this domain-specific complexity, we generalize another family of objectives based on the ELECTRA \citep{Clark2020ELECTRAPT} method for pretraining on text. Rather than reconstruct noised tokens as BERT does \citep{Devlin2019BERTPO}, ELECTRA involves replacing a subset of tokens in the input with substitutes, then training a detector network to predict which tokens were replaced. Substitute tokens can be chosen randomly, or generated by a BERT network. Similar replacement-detection methods have also recently been applied successfully to tabular data \citep{Iida2021TABBIEPR}, suggesting this objective is not text-specific.

We generalize ELECTRA by applying replacements at the embedding level, instead of the level of input tokens. This enables us to apply the method equally across all modalities, without domain-specific adjustments. To perform the replacements, we select 15\% of the embedding positions per input, then shuffle those embeddings among each other according to a random permutation. See the Appendix for more details. The task of the network is then to predict which of the embeddings were shuffled; this is instantiated as a binary prediction task performed by passing each output embedding through a fully-connected layer. We call this method ShED: Shuffled Embedding Detection.

\subsection{Linear Classification}

We transfer our trained models to downstream tasks with linear classifiers, a simple approach which enables the same base model to be adapted to many downstream tasks without storing a separate copy of the model for each task. We use the Adam optimizer \citep{kingma2014adam} with learning rate of 1e-4, $\beta_1 = 0.9, \beta_2 = 0.999$ for 100 epochs. We also compare against a randomly-initialized model which has not undergone training, to quantify the gains attributable to pretraining.

\subsection{Results}
We report average metrics by domain in Table \ref{table:results}, and full results for each transfer task in Table \ref{table:results-all}.  Our pretrained models broadly show gains over models that were not pretrained, although the gains are uneven and often quite modest compared to state-of-the-art domain-specific approaches. While the gains from pretraining are clear across transfer tasks in natural images, speech, text, sensors, and medical imaging, pretraining appears to hurt in the multimodal image-text domain, leaving a clear need for future work.  Interestingly, the relative gains for these algorithms also seems to reflect their communities of origin: e-Mix performs best on natural images, while ShED performs better on text-based tasks. Investigating the principles underlying these differences is an interesting avenue for future work, as is discovering methods that work better across all domians.

\begin{table}
\centering
\begin{tabular}{ccccccc}
\toprule
Pretraining & Natural Images & Text & Speech & Sensors & Med. Imaging & Images \& Text \\
\midrule
None   & 10.1 & 42.3 & 24.9 & 69.8 & 68.1 & \textbf{57.5}  \\
e-Mix  & \textbf{27.9} & 44.1 & \textbf{41.8} & 79.5 & 72.4 & 48.9  \\
ShED   & 20.9 & \textbf{48.4} & 36.5 & \textbf{88.7} & \textbf{74.5} & 54.3  \\
\bottomrule
\end{tabular}
\vspace{.2cm}
\caption{\textbf{Downstream linear classifier performance of baseline domain-agnostic methods across domains.} Reported numbers are average evaluation metrics across transfer tasks within a domain. Metrics are percent accuracy, with the exception of Medical Imaging (average percent AUC across the five pathologies), and two Text tasks: CoLA (Pearson corrleation) and STS-B (Spearman correlation). ``None'' refers to a randomly-reinitialized model that has not been trained.}
\label{table:results}
\end{table}

\section{Limitations and Conclusion}
\label{sec:conclusion}

DABS also has limitations. For example, a tradeoff exists between keeping a benchmark reasonably compact so it can be run easily and representing the full range of domains one might care about. Our choice of seven diverse domains represents a middle ground, but DABS is also a ``living benchmark,'' and we plan in the future to introduce domains spanning an even broader range of fields, data types, and applications to drive further progress towards domain-general SSL methods.

In addition, DABS does not capture how well domain-agnostic methods can be combined with domain-specific methods in a hybrid manner, which may be of greater relevance to domains like natural images where many domain-specific augmentations have already been developed. This is an important yet challenging-to-frame problem, and we encourage future work in this direction.

We have presented DABS: a Domain-Agnostic Benchmark for Self-Supervised Learning. Algorithms that perform well on DABS may have significant practical impact, unlocking the benefits of pretraining for a wide array of domains without a significant ML presence. We also hope DABS enables researchers to better understand the general principles underlying self-supervised learning across different domains, especially as the technology matures and becomes more broadly deployed.

\section*{Acknowledgments and Funding Disclosures}
We would like to thank Shyamal Buch, Jesse Mu, Jared Davis, Daniel Rothchild, and Mike Wu for useful discussions and feedback. AT is supported by an Open Phil AI Fellowship.

\bibliographystyle{plainnat}

redpajama_set_name: RedPajamaArXiv


text:
Home Breaking "Withdrawn to reimburse, each family 5,000 rubles": what were the conditions of...
"Withdrawn to reimburse, each family 5,000 rubles": what were the conditions of resettlement of Caucasians in 1943
History 03/02/20 "Withdrawn to reimburse, each family 5,000 rubles": what were the conditions of resettlement of Caucasians in 1943
the Great Patriotic war became a real tragedy for the Soviet people and a disaster for Nations that are due to individual representatives of the leadership of the Soviet Union was branded "collaborators." The Kalmyks, Ingush, Chechens, Balkars, Karachai, and many others had to endure not only the burden of protracted armed conflict, but also of deportation, often in the most remote corners of the country and not on the best terms.
From autonomy to deportation
the National policy of the early Soviet Union it's fair to assume not even tolerant, and encouraging diversity of people . By 1933 there were more than five thousand national councils – the administrative-territorial units, in which concentrated minorities. In Leningrad were printed Newspapers in 40 languages and has conducted radio programmes in Finnish.
However, in the mid-1930s, whether on the background of strengthening of the state and lack of need to go on about the groups with a small effect, or by disturbing the expectations of a looming global conflict, national policy of the Soviet leadership began to change. Political and even cultural autonomy of minorities is gradually eliminated, the arrests began on a national basis.
the Climax was the resettlement of people from areas of traditional residence. Deported Estonians, Latvians and Lithuanians, poles, Finns and Germans. Then went to the Koreans. And in each case, such action was motivated by just the border of the country it was necessary to clean it from the "unreliable elements".
the beginning of the great Patriotic wave of deportations gained momentum. And controversial events in the war only gave rise to the Soviet authorities continue the repression. In 1943-1944 years, the country held mass deportations of immigrants from the Northth Caucasus: Balkars, Ingush, Kalmyks, Karachays, Crimean Tatars, Nogais and others. And compared with pre-war stock relocation were already the character of "retaliation": the whole people the fault of individuals were accused of collaborating with the Nazis and in fact were expelled from their homeland.
"I Propose to deport…."
of Course, it is undeniable that the collaborative movement in the Caucasus was extremely strong. Former prisoners of war in the Wehrmacht was created by the Azerbaijan, Armenian, Georgian and North Caucasian legions. Not few were those who voluntarily took the side of Germans and organized the anti-Soviet uprising in the North Caucasus, and after their defeat of the partisans in the mountains. One only territory of the Chechen-Ingush Republic since the beginning of the war to January 1944, was eliminated the 55 gangs, killing 973 militants were arrested in 1901 people.
However, according to the NKVD, the gangs consisted only 0.5% of the population. So how fair is it to punish whole Nations for the sin of the individual "elements"? From a historical point of view, this issue is closed: the majority of repressed citizens were rehabilitated. Even though the benefits that they are now laid, will not do the hardships that they had endured during the deportations.
in addition, the relocation of North Caucasians was waves, which in theory can explain not only strategic, but also political implications of deportation. Many historians still believe that the real motives for the expulsion of entire peoples was not so much anti-Soviet activities during the war, many prewar conflicts of representatives of the deported Nations and government, ethnic prejudices of the Soviet leadership and the personal whim of Joseph Stalin, as well as intrigue the party bodies.
it is Documented that in most cases the decision on deportation, the Soviet leader took "the offer" of one or another party member. Most often it was the people's Commissar of internal Affairs Lavrenty Beria or the Minister of foreign Affairs Vyacheslav Molotov. Signed by the last was adopted a document regulating the expulsion of North Caucasus and the compensation for their losses in the relocation.
Beautifully on paper
According to the decree of Council of people's Commissars of the USSR of October 14, 1943, on the admission of migrants from the North Caucasus property, livestock and agricultural products, and conditions of partial reimbursement of the property in the places of resettlement of the deported were allowed to take some items: clothing, household equipment and food. The total weight of things on the ODA family could not exceed 500 kilograms. All the cattle, grain, houses and other buildings needed to pass, and they went primarily to cover state debts. Only the remaining part was due to the natural compensation to the new location.
the Resettlement of deportees, as a rule, carried out whole farms in vacant spaces in the existing collective and state farms in the Kazakh, Kirghiz SSR, Siberia and the far East. Those who drove less were placed in insulated tents. At the same time, the settlers were able to build mud houses for themselves or adapted to fit your needs vacant housing. Moreover, in special cases, they provided the construction loan within five thousand rubles per family for a period of seven years.
However, "beautiful" often it was only on paper. The eviction took place in the framework of special operations with the participation of thousands of NKVD, NKGB and the military, often in turmoil and confusion over three days could easily deport more than 100 thousand people. Often sometimes I make mistakes: for example, in the expulsion of the Chechens and Ingush to Kazakhstan were deported along with about 500 other people, mostly Avars.
the relocation and adaptation also took place hard. Had to ride in crowded cars and on the street it was cold. In 14 days the way many died from hunger and cold, and those who survived the road, we had to hold PErvoe time in a new place. After all, there still was nothing, and to heat the property – nothing.
"my younger sister went on one side of a wooden plank, to warm my feet, recalled in an interview with "Komsomolskaya Pravda" Karachai Bilal Baichorov. — She — left I — right. Other brother and sister folded up her different sides. Once I woke up this morning and think why mom's feet are extremely cold? Began to bother her to Wake, but she did not respond. And we suddenly realized the horrifying truth — our own, beloved NAFISAT no more in the world…"
And there are stories – hundreds of thousands. Because the total number of deportees from the North Caucasus people almost exceeded one million. And, unfortunately, not every one of them lived to rehabilitation.
Ivan Resepi
© Russian Seven
see also: editor's choice, "Russian Seven""Black storks": how to fight the most brutal "spooks" against the Soviet ariiotima snipers at the end of the great Patriotic was not wearing a distinctive snaiperskaya Nikolai Gogol: why is there a version of lethargic sleep pisatelyami Black devil in Khakassia the most mysterious place in Sibirtelekoma statesalaska… also Listen to the podcast "the Russian Seven". Share: Comments Comments to the article "the"Removed — to compensate, every family on 5,000 rubles": what were the conditions of resettlement of Caucasians in 1943″ Please log in to leave a comment! br>
Previous articleRace "Formula E" in China cancelled due to coronavirus
Next article"Peace": the threat is the largest diamond mine in the world

redpajama_set_name: RedPajamaCommonCrawl


As the year has come to a close I've been looking back on what I've achieved and where I was 12 months ago. Last year we were in the same place we had been for several years, in a constant patch and release cycle for code provided by our developers. We would continually have to fix software ourselves, sometimes bad by design sometimes just day to day bugs that we find. We also had to add new functionality ourselves as new requirements came through from the business. Often this doesn't get merged upstream.
Puppet came along early in the year and this enabled us to centralise and standardise the configuration for much of our infrastructure, so we could more quickly and easily roll out changes. This was a huge improvement for us, but it didn't really decrease the number and severity of software and configuration bugs.
The fact was our Quality Assurance processes were duplicated across the dev team and our team. The Dev team would do their own manual set of testing and we would run our own. As the year went on we started to included much more rigorous testing, as we had to test for bugs we had previously picked up but that had not been fixed upstream as well as functionality we'd implemented. But this was all still manual, time consuming and error prone. So how did we streamline this approach and improve quality in the end product at the same time?
Automation was key. We had automated, somewhat, the release mechanism, so I looked at going backwards and automating the deployment mechanism. We already had some degree of automation with deploying new Virtual Machines, but it was clunky and had far out grown the original scope. This was achieved by implementing vCenter Orchestrator, as I've discussed in my previously blogs.
So next on my list was automating tests. I came away from a Puppet Camp in Melbourne in November charged and ready to tackle this problem, as I'd heard two case studies where clients were using cucumber to automate testing. For testing puppet code the default testing framework is rspec, but cucumber seems to have gained quite a following in recent years for its simplistic language of test steps (implemented in a language called Gherkin that uses Given, When and Then keywords to develop test cases).
I had a lot of trouble initially wrapping my head around how I would complete all the testing, as we needed to do a mix of white-box and black-box testing. The puppet code we had needed to be white-box tested to ensure the internal logic was sound. But we also needed to test the proprietary products we use, for which we cannot inspect the internal state. This requires a different mechanism of testing called black-box testing, where we would need to simply throw information into a lab environment and inspect the result to ensure it matched our requirements.
Eventually, after a discussion with a friend who is a software developer who told me to just jump right in, I started to build my tests for puppet first.
So I implemented rspec testing for our puppet modules and after a few days of bashing my head against a brick wall I was able to confidently eliminate a small number of our manual tests. What frustrated me about rspec though was it's insistence on unit tests, testing just the functionality of a single module and nothing else. This seemed pointless to me as a System Administrator as I wasn't concerned with if a single module had an issue, but if the end to end system functioned.
I hacked away and implemented some integration testing that compiled a puppet catalog of almost an entire server and tested all the pieces existed, and all the variables and logic worked end to end.
At this point I decided to give cucumber a try, to see what all the hype was about, although I was initially hesitant to implement my "testing code" and "test steps" in two different files. Cucumber-puppet is sporadically maintained but still gives the same functionality as rspec. Another few days later and I started to fall in love with the Gherkin syntax that allowed me to define my test code once and write almost human readable tests that eventually consisted of 127 individual tests. Not only that it insisted on a more top down approach, testing all my modules together rather than individual functionality. But the end test steps still are not readable by management, they contain various phrases like "should have resource File[server.xml]" as well as "file should contain content matching <host>db01<\/host>" which really isn't meaningful.
Then I stumbled upon beaker and the solution to how to do black-box testing became apparent. Beaker is a wrapper of various shell tools and hypervisor APIs that allows you to build/configure a VM, kick off a puppet run with code from your branch and run some rspec tests. It's exactly what I needed, end-to-end testing (called acceptance testing).
There's still some work to do, such as integrating with vCenter Orchestrator which I've already invested heavily in to leverage automated provisioning (checkout my branch here), but I'm almost there. Also I need to fit it in with our existing release workflow, but our designated "testing engineer" should be happy. Once completed I believe I might well have reached Nirvāṇa.
I know this isn't really a new concept within the Software Development community, but too many businesses still fail at the basics. Especially the many companies that spent many many years just "integrating" products that various companies provided without doing real software development. In the Software Defined World, every company is a software company.


Engineers have the unique ability to see solutions where others see challenges. Through innovative thinking, extreme attention to detail, and carefully calculated actions, engineers of all kinds design and create things that make our lives better. Unfortunately, because of the extremely complex nature of their jobs, engineers also often find themselves on the wrong side of professional liability claims.
Because a small mistake by an engineer contractor can have significant consequences for a project, clients are often quick to point fingers — even when no mistake has been made. That is why it is essential for Ontario engineer contractors to have Errors and Omissions insurance in place before signing on to any project.
When you work in a highly specialized field, it is easy to run into miscommunications and suspicion of wrongdoing. Most people simply don't speak engineer, so whether an actual mistake has occurred or a client simply wants to place blame, claims arise against engineering contractors almost every day.
Your client believes he incurred a loss due to engineering drafts and blueprints containing an oversight.
Your client believes he must repair one of your submitted designs due to an error you have made in it.
Your client believes you did not stick to the agreed upon budget, resulting in cost overages or unfinished work.
Your client believes that engineering submissions are causing a bottleneck on the project due to their delivery time.
Of course, the complexity of an engineer's day-to-day work means that the above are just a small sampling of the near countless reasons clients can — and have — lawsuits for professional negligence.
Bottom line: if you work in the engineering field and have direct or indirect interactions with clients, you likely are at risk of being accused of wrongdoing.
Whether you work independently or within a larger firm, it is important you keep yourself protected. Your skillset is among the most coveted in Ontario, so don't let one mistake or false claim put your career at risk. Get protected today.