EVA: Exploring the Limits of Masked Visual Representation Learning at Scale

We launch EVA, a vision-centric foundation model to explore the limits of visual representation at scale using only publicly accessible data. EVA is a vanilla ViT pre-trained to reconstruct the masked out image-text aligned vision features conditioned on visible image patches. Via this pretext task, we can efficiently scale up EVA to one billion parameters, and sets new records on a broad range of representative vision downstream tasks, such as image recognition, video action recognition, object detection, instance segmentation and semantic segmentation without heavy supervised training. Moreover, we observe quantitative changes in scaling EVA result in qualitative changes in transfer learning performance that are not present in other models. For instance, EVA takes a great leap in the challenging large vocabulary instance segmentation task: our model achieves almost the same state-of-the-art performance on LVISv1.0 dataset with over a thousand categories and COCO dataset with only eighty categories. Beyond a pure vision encoder, EVA can also serve as a vision-centric, multi-modal pivot to connect images and text. We find initializing the vision tower of a giant CLIP from EVA can greatly stabilize the training and outperform the training from scratch counterpart with much fewer samples and less compute, providing a new direction for scaling up and accelerating the costly training of multi-modal foundation models. To facilitate future research, we release all the code and models at https://github.com/baaivision/EVA.

Introduction
Scaling up pre-trained language models (PLMs) [9,63,76] has revolutionized natural language processing (NLP) in the past few years. The key to this success lies in the simple and scalable self-supervised learning task of masked signal prediction [31,74], with which Transformer models [101] † Interns at Beijing Academy of Artificial Intelligence.  could be scaled up to billions of parameters using nearly unlimited unlabelled data, and generalize well to a wide range of downstream tasks with little tuning. With further scaling on compute, data, and model sizes, PLMs have led to not only continuous performance improvements [51,75,76], but also a surprising emergence of in-context learning capability [9,25,107,108].
pure MIM pre-trained model could not achieve favorable performance at billion-scale model sizes [117]. We regard this gap stems from the fact that natural images are raw and information-sparse. Meanwhile, an ideal vision pretext task needs the abstraction of not only the low-level geometry & structure information, but also high-level semantics, which is hardly captured by pixel-level recovery tasks [115].
In this work, we seek a suitable MIM pretext task for large scale vision representation learning and explore its limits at the scale of one billion parameters with tens of millions of unlabeled data. Recently, there are a few trials leveraging the semantic information from image-image or image-text contrastive learning [13,22,73] for MIM pretraining [44,109,130], which perform fairly well in vision downstream tasks. However, there remains a debate that (i) tokenized semantic features could provide better supervision signal for masked modeling in vision [5,70,104], and (ii) good performances could be also achieved via a simple postdistillation process without masked prediction tasks [110]. Through a pilot empirical study, we find that simply using image-text aligned (i.e., CLIP [73]) vision features as the prediction targets in MIM scales up well and achieves satisfactory performances on a broad range of downstream benchmarks. This pre-training task draws the benefits from both the high-level semantic abstraction of image-text contrastive learning as well as the good capture of geometry & structure in masked image modeling, which typically covers the information needed for most visual perception tasks. Via this MIM pretext task, we can efficiently scale up a vanilla ViT encoder [33], dubbed EVA, to one billion parameters with strong visual representations that transfers well to a wide range of downstream tasks (Fig. 1) Notably, different from other state-of-the-art billion-scale vision foundation models that demand tens of millions of or even billions of labeled images, such as SwinV2-G using ImageNet-21K-ext-70M [64] and ViT-g/G using JFT-3B [123], EVA does not need a costly supervised training stage and only leverage images from open-sourced datasets for academic reproducibility.
Moreover, we observe quantitative changes in scaling EVA result in qualitative changes in transfer learning performance that are not observed in other smaller-scale models, e.g., EVA makes a significant breakthrough in the challenging large vocabulary object-level recognition task: our model achieves almost the same performance on LVISv1.0 [39] (55.0 AP mask on val), an instance segmentation benchmark with more than 1,200 categories, as COCO [61] (   but with only 80 categories annotated. This emergent ability well matches the expectation of model scaling [108], that larger capability of model results in not only predictable performance improvements on standard benchmarks, but also unpredictable phenomenons and capabilities for resolving more challenging tasks. Going beyond a pure vision encoder, EVA can also serve as a vision-centric, multi-modal pivot that builds a bridge between vision and language. We show that initializing the image encoder via pre-trained EVA in a 1.1 billion parameters CLIP model can outperform the training from scratch counterpart on a broad range of zero-shot image / video classification benchmarks with much fewer samples and less compute. Moreover, EVA can greatly stabilize the giant CLIP's training & optimization process. Since large CLIP models usually suffer from training instability and inefficiency issues [2, 48], we hope our solution opens up a new direction for scaling up and accelerating the costly training of multi-modal foundation models.
By scaling up vision-centric foundation models with MIM pre-training to achieve strong performance on broad downstream tasks, we hope EVA would bridge the gap between vision and language with masked signal modeling, and contributes to the big convergence across different modalities.

Fly EVA to the Moon
We first conduct a series of pilot experiments for choosing an ideal vision pretext task in §2.1, then we scale up EVA pre-training via the chosen pre-training objective in §2.2. Finally, we evaluate the pre-trained representation on various downstream tasks in §2.3. Detailed experimental settings and configurations are in Appendix A.

The Feature Instrumentality Project
In this section, we seek a MIM vision pretext task with compelling transfer performance. Based on previous literature on vision pre-training, we study two promising candidates: (i) recovering the masked out tokenized semantic vision features [5,70,104], and (ii) feature distillation from strong pre-trained representation as in [110]. Both of them exploit pre-trained image-text aligned vision features (i.e., CLIP [73] vision features). Via a series of pilot experiments shown in Table 2, we find that: (i) the (additional) CLIP feature tokenization process is unnecessary for achieving good downstream performance (ii) feature distillation fails to provide consistent performance gain as the pre-training becomes longer. Instead, we find that simply reconstructing the masked out CLIP vision features conditioned on visible image patches is highly performant, which is chosen for scaling up EVA.
We clarify that this MIM pretext task is not originally proposed by us. Regressing the masked out image-text aligned vision features for MIM pre-training has been studied in MVP [109] and recently has been revisited by MILAN [44]. In this work, we show that this pretext task can scale up to billion-scale parameters and tens of millions of unlabeled images for vision-centric representation learning without (i) semantic feature quantization / tokenization [5,70], and (ii) explicitly using image-text paired pre-training data and large corpora as in BEiT-3 [104].

Pre-training
Architecture. The architecture configurations of EVA are in Table 3a. EVA is a vanilla ViT [33] with 1.0B parameters. The shape of her follows ViT giant [123] and the vision encoder of BEiT-3 [104]. We do not use relative positional embeddings [89] and layer-scale [99] during pre-training.
Pre-training objective. EVA is pre-trained to reconstruct the masked out image-text aligned vision features conditioned on visible image patches. We corrupt the input patches with [MASK] tokens, and we use block-wise masking with a masking ratio of 40% following [5,70,104]. The target for MIM pre-training is from the publicly available 1 Ope-nAI CLIP-L/14 vision tower trained on 224×224 pixel images [73]. The output feature of EVA is first normalized [3] and then projected to the same dimension as the CLIP feature via a linear layer. We use negative cosine similarity as the loss function.
Pre-training data. The data we used for pre-training EVA are summarized in Table 3b. For CC12M [16] and CC3M [88] datasets, we only use the image data without captions. For COCO [61] and ADE20K [129] datasets, we only use the train set data. ImageNet-21K [30] and Ob-ject365 [87] image data are also used. All these data are publicly accessible. The merged dataset for pre-training has 29.6 million images in total. model #param. extra labeled data image size top-1 acc.  The CLIP features we used as MIM prediction targets are trained on a 400 million image-text dataset in a selfsupervised manner. So during pre-training EVA also implicitly exploits the knowledge from this dataset to some extent. Meanwhile, these CLIP features is also widely used in other state-of-the-art representation learning & pre-training works such as the BEiT family [70,104], AI generated content [78,83,84] and large scale dataset filtering [10,85,86]. Table 3c, EVA is optimized via Adam [53] with decoupled weight decay [67] of 0.05. The peak learning rate is 1e-3 and decays according to a cosine learning rate schedule. We employed stochastic depth [46] with a rate of 0.1 for regularization and RandResizeCrop (0.2, 1) for data augmentation. Color jitter is not used.

Pre-training settings & hyper-parameters. As shown in
Pre-training infrastructure and statistics. Some basic pre-training statistics are available in Table 3d. The GPU we use is NVIDIA A100-SXM4-40GB. Pre-training code is based on BEiT [5] written in PyTorch [69]. We also adopt DeepSpeed optimization library [80] with ZeRO stage-1 optimizer [77] to save memory. We find using fp16 format with dynamic loss scaling is stable enough during the whole course of pre-training while using bfloat16 format is unnecessary. Since we use fp16 precision, EVA can also be pre-trained using 16× NVIDIA 24GB (32GB) GPUs with (without) gradient checkpointing [20].

Evaluation on Downstream Tasks
In this section, we extensively evaluate pre-trained EVA on several representative benchmarks, such as image classification ( §2.3.1), video action recognition ( §2.3.2), object detection & instance segmentation ( §2.3.3), semantic segmentation ( §2.3.4), and contrastive image-text pre-training with zero-shot evaluation ( §2.3.5). EVA achieves state-ofthe-art performance on a broad range of downstream tasks.
Results. Table 4 compares EVA with some state-of-the-art models on ImageNet-1K validation set. EVA achieves 89.6% top-1 accuracy with 336 2 inputs, comparable to BEiT-3. Using a larger image resolution of 560 2 can further boost the top-1 accuracy to 89.7%. Notice that BEiT-3 treats image classification as an image-to-text retrieval task. Therefore they leverage an additional one billion parameters pre-trained language encoder along with 35 million image-text data (21M pairs from CC12M, CC3M, SBU, COCO, VG and 14M pairs from ImageNet-21K) as well as 160GB text data in total. Meanwhile, we simply use a linear classifier on top of EVA with only ImageNet-21K image-tag data used for additional finetuning. With only publicly available data, EVA creates a new state-of-the-art image classification result on ImageNet-1K with a much neater architecture.
Robustness & generalization ability evaluation. We evaluate the robustness and generalization capability of EVA trained with an image size of 336 2 on 6 different ImageNet-1K validation set variants. In Table 5, we compare EVA with some top open-sourced models collected by the timm library 2 [111]. Following the evaluation procedure in [40], all these models are first fine-tuned on the original ImageNet-1K training set and then evaluated on different validation sets using the same fine-tuned model without further hyper- parameter selection and specialized fine-tuning. As shown in Table 5, EVA is the most competitive one in terms of absolute top-1 accuracies. However, these model various in pre-train data (from ImageNet-1K, ImageNet-21K to JFT-300M), input resolutions (from 224 2 to 800 2 ), model sizes (from hundreds of millions to one billion parameters) as well as architectures (ConvNets, vanilla & hierarchical ViTs), etc. Therefore their absolute accuracies are not directly comparable. Instead, we are more interested in the gap between the averaged top-1 accuracy on 6 validation set variants and the original ImageNet-1K validation set top-1 accuracy (the lower the better), i.e., we care about whether a model along with its training settings biases towards the original validation set and generalize well on other variants. From this perspective, EVA not only achieves the highest averaged accuracy, but also has the smallest performance gap, which reflects the excellent robustness and generalization ability of EVA.

Video Action Recognition
Datasets. For video action recognition, we evaluate EVA on Kinetics-400 (K-400) [52], Kinetics-600 (K-600) [14] and Kinetics-700 (K-700) [15] benchmarks. We first conduct intermediate fine-tuning on a merged dataset coined Kinetics-722 (K-722) that integrates videos from K-400, K-600 and K-700. We remove leaked as well as repeated videos in both training and validation sets. After this data de-duplicating process, K-722 has 0.63M training videos in total with 722 action classes.
Training & evaluation settings. EVA processes video data simply via spatial-temporal attention as [35,97] with no specific architectural adaptation for video related tasks. We first train EVA using K-722 training set for 40 epochs with 8 frames and 224 2 resolution, then we fine-tune EVA on each dataset for only 1 or 2 epochs. We set frame×crop×clip to 16×3×4 for fine-tuning and evaluation for all datasets. The  Table 6. Video action recognition. With only publicly available K-400, K-600 and K-700 as video pre-training data, EVA is also quite performant in video action recognition tasks.
Results. As shown in Table 6, EVA achieves better performance compared with some recent video-specific or large foundation models in video recognition. For reference, directly adapting image-only pre-trained EVA to K-400 without K-722 intermediate fine-tuning can also achieve a very competitive top-1 accuracy of 88.4%.

Object Detection & Instance Segmentation
Datasets. We evaluate the object detection and instance segmentation performance of EVA on both COCO [61] and LVISv1.0 [39] benchmark. COCO is a widely used objectlevel recognition benchmark consisting of 118k train, 5k val, and 20k test-dev images respectively, with 80 common object categories. LVISv1.0 is an emerging largevocabulary object-level recognition benchmark, which has more than 1,200 object categories as well as more than 2 million high quality instance segmentation masks (nearly 2× of COCO instance masks).
Notably, COCO and LVISv1.0 almost use the same set of images, and both train and val split of LVISv1.0 have a huge overlap with COCO train and val split. Meanwhile, COCO has much fewer object categories than LVISv1.0 (i.e., 80 v.s.1,200+). Therefore it is interesting and meaningful to evaluate a model's performance on both COCO and LVIS.
Metrics. For COCO, we report the standard box AP (AP box ) and mask AP (AP mask ) on both val and test-dev split. For LVISv1.0, we evaluate EVA using AP box , AP mask and AP mask rare defined in [39] on the v1.0 val set. We also report the experimental "boundary, fixed AP"(AP boundary in Table 8a) used in the LVIS 2021 challenge 3 for reference.
Training & evaluation settings. EVA uses Cascade Mask R-CNN [12] as the detector and adopts the training settings (e.g., large scale jitter data augmentation [37]) & architecture configurations (e.g., interleaved window & global (a) Results of object detection and instance segmentation on LVISv1.0 val set. "AP box , AP mask , AP mask rare " are evaluated using the rule defined in [39]. We also report the experimental "boundary, fixed AP"(AP boundary ) used in the LVIS 2021 challenge for reference.  Table 8. Object detection and instance segmentation performance on LVISv1.0 dataset (Table 8a). EVA not only achieves the state-ofthe-art results on both LVISv1.0 and COCO with the same set of architectures and settings, but also largely closes the performance gap between them (Table 8b). (timestamp: Nov 10, 2022) attention) of ViTDet [59]. Following the common practice [64,104,126], we first conduct intermediate fine-tuning for the whole detector using Objects365 [87] dataset with a resolution of 1024 2 , then we fine-tune the detector on COCO and LVISv1.0 train split respectively with 1280 2 inputs.
We report single-scale evaluation and multi-scale evaluation / test-time augmentation (tta) results of EVA for comparison. For COCO, Soft-NMS [8] is also applied. For instance segmentation task, the classification score is calibrated [47] via maskness [105].
The model architecture as well as the hyper-parameters for COCO and LVISv1.0 are almost the same (i.e., the hyperparameters are nearly "zero-shot" transferred from COCO to LVISv1.0), expect we use federated loss [131] and repeat factor sampling [39] following ViTDet on LVISv1.0.
COCO Results. Perhaps COCO is the most fierce vision benchmark. Table 7 [126], EVA demonstrates that with sufficient model size, data and pre-training, better performance can be also achieved via the classic R-CNN framework [38]. On the other hand, FocalNet and Group DETRv2 are incapable of instance segmentation due to using DINO.
Compared with SwinV2-Giant [64] and FD-SwinV2-Giant [110] that also adopt a (stronger HTC++ [17]) detector from the R-CNN family but with ∼3× model size of EVA, our approach streamlines the pre-training processes and pulls off a "Giant-killing" act via better representations.
Compared with BEiT-3, EVA shows that is possible to build a state-of-the-art object-level recognition system without exploiting (i) semantic feature quantization / tokenization [5,70], and (ii) image-text paired pre-training data and large corpora during pre-training.
LVIS Results. Table 8a summarizes the results on LVISv1.0 val set. EVA achieves state-of-the-art performance under all metrics with single-scale evaluation, outperforming the previous best approaches by a large margin.
Analysis the LVIS-COCO gap: more is different. It is interesting and valuable to evaluate and analyze a model on both COCO and LVISv1.0 benchmarks, as they share almost the same image set but with different numbers of annotated object categories. Compared with COCO which has only 80 annotated categories, LVIS annotates more than 1,200 object categories and thus naturally features a longtail distribution, which is more close to the challenging real world scenario [39]. In general, LVIS is considered to be a much harder benchmark than COCO in object-level recognition, and conventional methods usually suffer from a large performance drop on LVIS compared with COCO.
In Table 8b, we analyze the performance gap between LVISv1.0 and COCO benchmark of EVA and other stateof-the-art approaches. For previous leading methods such as ViTDet, the performance gap between AP box is around 8, and the gap between AP mask is around 5. However, using the same detector (Cascade Mask R-CNN) and almost the same settings as ViTDet pre-trained via MAE-Huge (ViTDet-H), EVA not only achieves the state-of-the-art results on both LVIS & COCO benchmarks simultaneously, but also largely closes the performance gap between them, especially for the instance segmentation task, that EVA achieves the same performance on LVIS and COCO with single-scale evaluation. Compared with ViTDet-H, we show a little larger model and stronger representations can take a great leap in the challenging large vocabulary instance segmentation benchmark-with one caveat described next.
Notice that it is inaccurate to say EVA "solves" the LVIS large vocabulary instance segmentation task based on "zero AP mask gap", and it is possible to achieve even higher AP mask on LVIS than COCO (e.g., LVIS has higher quality mask annotations for both training and evaluation). Although the AP mask rare of EVA on LVIS is much better than previous approaches, there is still a big gap between the rare categories and common / frequent categories (i.e., AP mask rare = 48.3 v.s. AP mask comm = 55.5 / AP mask freq = 57.4). From another perspective, the gap of AP box between LVIS and COCO (1.9 for EVA) may better reflect the overall gap between the large vocabulary and common vocabulary object-level recognition, since this metric eliminates the confounder of instance mask annotation quality.
Nevertheless, EVA makes a significant breakthrough in the challenging large vocabulary object-level recognition task. We believe other large vision foundation models such as SwinV2-G and BEiT-3 also have similar properties on COCO and LVISv1.0 benchmark. We hope our efforts can encourage the community to pay more attention to the relationships between different tasks while scaling up models and chasing the state-of-the-art in each individual task.  Table 9. Semantic segmentation performance on ADE20K and COCO-Stuff-164K dataset. "mIoU ss ": mIoU of single-scale evaluation, "mIoU ms ": mIoU using multi-scale evaluation.

Semantic Segmentation
Dataset. We evaluate EVA on ADE20K [129] and COCO-Stuff-164K [11]  Training & evaluation settings. We follow the task transfer pipelines of ViT-Adapter [23]+mask2former [24] but with a weakened model adaptation processes due to GPU memory limitation (40GB of VRAM): (i) relative position biases [90] are not applied. (ii) We use 8× decoders in mask2former segmentation head instead of 9×. (iii) The feature dimension in mask2former head is ∼0.6× of EVA encoder.
Results. We compare EVA with other leading semantic segmentation methods in Table 9. EVA achieves strong results in both ADE20K and COCO-Stuff-164K datasets.
On the other hand, the segmentation performance of EVA is slightly lower compared with BEiT-3 on ADE20K, we suspect this is partially due to our weakened architectural configurations.

Contrastive Language-Image Pre-training with Zero-shot Classification Evaluation
CLIP (Contrastive Language-Image Pre-training) [48,50,72,73] is a type of multi-modal foundation model that connects vision and language via contrastive image-text pre-training. CLIP can be applied to any image classification benchmark by simply providing the names of the visual categories to be recognized [1]. Thus the introduction of CLIP essentially reshapes the landscape of visual recognition. Meanwhile, CLIP features also play a central role in representation leaning [70,104], AI generated content [78,83,84] and large dataset filtering [10,85,86] Table 10. EVA as a vision-centric, multi-modal pivot. We evaluate a billion-scale contrastive language-image pre-trained (CLIP) model with the vision tower initialized from pre-trained EVA, which largely accelerates the contrastive training efficiency and shows promising zero-shot classification performance across a wide range of image / video benchmarks. The statistics & performance of EVA's MIM teacher (OpenAI CLIP-L) are also presented for reference. A thorough evaluation of EVA CLIP on 35 zero-shot benchmarks (including 27 zero-shot image classification benchmarks + 4 zero-shot video classification benchmarks + 2×2 zero-shot retrieval benchmarks) is available at link1 and link2.
In this section and Table 10, we show that EVA is not only a strong encoder for a wide range of vision downstream tasks, but also a multi-modal pivot that builds a bridge between vision and language. To demonstrate that, we train & evaluate EVA as a billion-scale CLIP's vision tower in various zero-shot image / video classification benchmarks.
Baselines and major challenges in CLIP model scaling. We compare our CLIP (dubbed EVA CLIP) with other opensourced strong CLIP competitors that exploit publicly accessible data / academic resources only. Model configurations and statistics are detailed in Table 10a.
There are two well-known major challenges of CLIP model training and scaling: (i) Large-scale Open CLIP models (e.g., Open CLIP-H & Open CLIP-g [2, 48]) usually suffer from severe training instability issues [2] and have to use bfloat16 format for optimization. (ii) The training efficiency is low, which may hinder model scaling and downstream performance. For instance, Open CLIP-g is heavily under-trained due to its large compute requirement, and its performance is even worse than the sufficiently-trained Open CLIP-H with a smaller model size.
Compared with our CLIP model, Open CLIP-H & -g are trained from scratch with much more image-text pairs (∼2.9× and ∼1.1× of ours) sampled from a much larger dataset (∼5× of ours) on ∼3× of GPUs. While by leveraging EVA, billionscale CLIP model training can be accelerated with improved zero-shot classification performance, described next.
Training settings. For our CLIP model, we initialize the vision encoder via pre-trained EVA and the language encoder from OpenAI CLIP-L. The pre-training implementation is based on Open CLIP [48]. We also adopt DeepSpeed optimization library [80] with ZeRO stage-1 optimizer [77] to save memory. We find using fp16 format with dynamic loss scaling is stable enough during the whole course of training while using bfloat16 format is unnecessary. These modifications allow us to train a 1.1B CLIP with a batch size of 41k on 256× NVIDIA A100 40GB GPUs.
Evaluation settings. We evaluate zero-shot image / video classification performance of each CLIP model on 12 benchmarks and report top-1 accuracy for comparisons.
Results. Table 10b shows the comparison. Our EVA CLIP achieves the highest averaged accuracy, and performs the best in 10 out of 12 zero-shot classification benchmarks. Notably, the ImageNet-1K validation zero-shot top-1 accuracy is 78.2% without using any of its training set labels, matching the original ResNet-101 [41]. Moreover, our model is quite robust and suffers from the smallest performance drop when facing natural distribution shifts in ImageNet.
At last, in Table 11 we provide zero-shot, linear probing & end-to-end fine-tuning top-1 accuracy of EVA-CLIP on ImageNet-1K validation set for reference. Our approach creates the new state-of-the-art results among all existing self-supervised learning methods. Notice that EVA CLIP's vision branch learns from Ope-nAI CLIP-L, while language branch initialized from the same CLIP-L model. Therefore, starting from a CLIP-L with only 430M parameters, we progressively scale up a 1.1B EVA CLIP-g with large performance improvements. This implies that interleaved MIM & image-text contrastive pre-training could be an efficient and scalable CLIP training approach. To our knowledge, EVA CLIP-g is the largest performant CLIP model trained via publicly accessible data and resources. We hope our practice on scaling and improving CLIP can also inspire and transfer to the study of other large scale multi-modal foundation models.

Related Work
Masked image modeling (MIM) learns rich visual representations via predicting masked visual contents conditioned on visible context. ViT [33] and iGPT [18] report the first meaningful MIM pre-training results. The BEiT family [5,70,104] greatly improves MIM's performance via masked visual token prediction. Recent work [4,21,32,34,40,106,116,130] (re-)explore pixel / feature regression in MIM, but only in a relatively small model and data scales. In this work, we explore the limits of large scale MIM pre-training via masked image-text aligned feature prediction [44,109].
Vision foundation models. ConvNets [56] have long been the de-facto standard visual architecture ab initio. Since AlexNet [55], ConvNets have rapidly evolved and become deeper, wider and larger [41,45,65,91,93,96,114]. However, at sufficient model and data scales, ConvNets lag behind ViTs [33] due to a lack of scalable pre-training tasks and the built-in inductive biases. Entering the 2020s, large pretrained ViTs [33,123] such as SwinV2-G [64] with hierarchical architectures as well as BEiT-3 [104] with multi-modal representations started to demonstrate various vision benchmarks. In this work, we show by leveraging unlabeled images, vanilla ViT can be efficiently scaled up to billion-scale parameters, and stands out in various downstream tasks.
cleaned K-722 contains 0.63M training videos, covering 722 human action classes. Table 15 lists the detailed settings & hyper-parameters for fine-tuning on this dataset. In the second stage, we further fine-tune on each dataset using more input video frames of 16 with a resolution of 224 2 . For the frame sampling, we adopt the sparse sampling strategy [103]. During testing, we follow the common practice of multi-view inference [35,66,97,106] with 4 temporal clips and 3 spatial crops. The final prediction is the ensemble of all trials. Table 16 lists the detailed hyper-parameters for fine-tuning on K-400, K-600 and K-700.

A.3. Object Detection & Instance Segmentation
The detailed hyper-parameters are shown in Table 17 and  Table 18. For intermediate fine-tuning on Objects365 [87], the model is trained with a batch size of 128 for 380k iterations. To accelerate the training process, we use a smaller input resolution of 1024 2 for the first 320k iteration. Af-terward, the input resolution is lifted to 1280 2 for a better adaptation to the fine-tuning of COCO and LVIS.
For fine-tuning COCO and LVIS, the learning rate is initialized as 2.5e-5 and step by a factor of 10 for the last 5k iterations. As shown in Table 18, we use almost identical hyper-parameters for training COCO and LVIS. Except for the commonly used repeat factor sampling [39] and federated loss [131] that are specialized for long-tailed recognition, the only difference in training is that we train the model for 45k steps on COCO, while a longer 75k step on LVIS, since the tail classes generally take a longer schedule to converge [36].

A.4. Semantic Segmentation
Detailed configurations about semantic segmentation are available in Table 19. Our settings basically follow ViT-Adapter [23] with Mask2Former [24] as the segmentation head. For ADE20K, we use COCO-Stuff pre-trained weights as initialization.  Table 19. COCO-Stuff-164K and ADE20K semantic segmentation fine-tuning settings.


extfieldsofstudy:
Computer Science
Engineering

text:
A Simple Discretization Scheme for Gain Matrix Conditioning

In industrial model predictive controllers (MPCs), models generated from regression-based system identification methods typically contain small or even physically non-existent degrees of freedom. Control issues can arise when the steady-state optimizer uses these small degrees of freedom to calculate targets for plant operation due to matrix ill-conditioning. Mathematical techniques like Relative Gain Array (RGA) and Singular Value Decomposition (SVD) are helpful for analyzing controller gain interactions and identifying conditioning issues, which can be corrected relatively easily in small models. However, these techniques are difficult and tedious to apply for larger, more complex models. This paper describes a novel, non-iterative, RGA-based, binning technique for discretizing the gain matrix and quickly solving 2x2 conditioning issues for any model size, while guaranteeing gain adjustments below a certain threshold. Higher order interactions are also discussed.

I. INTRODUCTION
Industrial model predictive controllers (MPCs) are commonly implemented with an integrated steady-state, Linear Program (LP) optimizer to generate targets that drive the process to an economic optimum [1]. In industrial MPC projects, one of the most difficult and time-consuming steps is understanding the interaction of the steady-state model gains with the optimizer costs in the LP solution. Part of this consideration is respecting the material and energy balance of the process, and determining how much "detail" in terms of smaller gains should be built into the model.
In practice, designers must make structural decisions in the gain matrix to reconcile the model with the true process. This step relies on the designer's process knowledge and experience, along with the usage of process data to justify any curve or gain manipulation of the model. Adjusting the gain matrix is often a balancing act between maintaining model accuracy and usability, and arbitrarily deleting or scaling gains should be avoided. A case study from Mitsubishi demonstrated how the erroneous removal of a gain element, thought to be insignificant based on engineering judgment, resulted in controller instabilities [2].
Fundamentally, there are 3 types of MV-CV (Manipulated Variable-Controlled Variable) interactions: • Non-collinearities: MV-CV interactions that are easy to independently control • Collinearities: MV-CV interactions that cannot be independently controlled, because the gains are collinear • Near-collinearities: MV-CV interactions that are difficult to independently control The non-collinear MV-CV interactions are typically trivial to model and tune, and do not warrant in-depth discussions for the purposes of this paper.
The collinear MV-CV interactions can often be identified based on process data and a priori process knowledge. These interactions may not be exactly collinear in the raw model, but can be corrected by forcing the gain ratios of the collinear MV-CV pairs to be numerically equal. Doing so ensures that the LP will not try to use control handles that are nonexistent.
The near-collinear MV-CV pairs can result in problematic interactions due to numerically ill-conditioned matrices, which, if active, cause undesirable large MV movements to control the near-collinear CVs independently at their constraint limits. In industry, it is common for control practitioners to adjust the steady-state gains manually [2] to deal with ill-conditioning issues. When a gain is changed, its interaction with all of the other gains also changes. Consequently, fixing one gain conditioning issue may create other issues in other parts of the matrix. For further details, Waller and Waller (1995) [3] provides an excellent review of process directionality literature.
An ExxonMobil patent [4] describes a mathematical procedure for adjusting gain matrices. Some commercial applications try to automatically correct the ill-conditioning, such as Honeywell's RMPCT which uses Singular Value Thresholding (SVT) [5], but automatic techniques can fail because the raw gains can be far enough from the collinearity threshold that they are not fixed. On the other hand, making the SVT threshold large enough to fix these gains could incorrectly remove small but critical interactions/degrees of freedom. Semi-automated techniques, such as those described in a patent by Aspentech [6], may not always produce satisfactory results that are consistent with the process, and may warrant further manual adjustments.
In this paper, we present a novel, non-iterative, binning approach to gain conditioning that significantly reduces the difficulties associated with gain matrix adjustments faced by control practitioners.

II. REVIEW OF TECHNIQUES
Two primary tools for analyzing model conditioning are Singular Value Decomposition (SVD), and the Relative Gain Array (RGA).

A. Singular Value Decomposition
Consider a 2 × 2 steady-state gain matrix, G, SVD decomposes G into three matrices with G = U ΣV T , describing its most responsive and least responsive directions in terms of both MV moves and CV responses, where 22 v T A physical interpretation of SVD in the context of multivariable control is described in detail by Moore (1986) [7], as illustrated in Fig. 2. CVs will be most responsive in the direction of u when the MVs are moved in the direction of v T . The opposite is true for u and v T . The singular values (σ, σ) represent the magnitude of each CV direction's controllability.
Singular values are proportional to system sensitivity [7]; high σ values indicate that MV moves will need to be small to maintain accurate control, and vice versa. Singular values that are close to each other indicate that, generally, a controller will be able to control different CVs relatively independently. In contrast, systems with very different singular values will require larger MV moves for disproportionately smaller CV changes. This effect is encapsulated by the condition number, which is defined as the ratio of the largest to the smallest singular value: It is important to note that the SVD is scale-dependent. The singular values and condition numbers calculated are based on the engineering units of the MVs and CVs.
As an example, consider an unscaled gain matrix in Eq. (3) based on the debutanizer process in Figure 1. MV1 is the TC-REBOILER-SP, MV2 is the FC-REFLUX-SP, Fig. 2. Graphical interpretation of SVD. The left subfigure represents a unit circle of MV input moves, and the right subfigure represents the CV responses. SVD breaks down the CV responses in terms of a 'strong' (red) and 'weak' (blue) direction illustrated by the CV ellipse. As the weak direction becomes weaker, γ increases and the system converges towards collinearity. A collinear matrix would collapse the CV ellipse into a line in the direction of u.
CV1 is the AI-RVP-PV in the bottoms and CV2 is the AI-DIST-C5 in the overhead.
The SVD results based on Eq. (3) are summarized below. The condition number is calculated to be γ = 11.74. However, due to the scale-dependent properties of SVD, the results obtained with Eq. (3) may not be meaningful.
Researchers have proposed various scaling choices, including scaling by physical limits, relative importance of variables, or by other problem-specific scaling that minimizes the condition number to obtain γ min [3]. In practice, a common and useful way to scale the matrix is typical move scaling. This method multiplies each gain by the maximum MV move size made during identification, then divides it by the maximum gain of that CV across all other MVs. The scaled matrix, G using typical move scaling is given by where S a and S b are non-zero, diagonal scaling matrices such that S a = diag (∆MV 1 , ∆MV 2 ) and In the first scaling step, the diagonal elements in S a scale each MV column in G by its ∆MV move size. In the second scaling step, the diagonal elements in S b scale each CV row by dividing each row by that CV's maximum gain in the GS a matrix. Typical move scaling 'normalizes' the gains by considering both MV move sizes and CV responses to all related MVs. Consequently, small scaled gains may indicate that the gain is small or insignificant, or the MV move size was too small for that CV, or that large MV moves were made with a high-gain CV, making other CV scaled gains look small.
We now consider a scaled gain matrix based on Eq. (3). Let the TC-REBOILER-SP move size, ∆MV 1 = 2, and the FC-REFLUX-SP move size, ∆MV 2 = 10. Using typical move scaling, the gains in each CV now lie between -1 to +1. The scaled gain matrix, G is calculated using Eq. (4) as Scaling has changed our understanding of the problem. γ = 2.68 is now smaller for G , compared to γ = 11.7 for G prior to scaling. The higher the condition number, the harder it is to drive in the weak direction. From a distillation control perspective, we know that it is much easier to lower the RVP and increase the C5s simultaneously (shift the column composition profile u p), than it is to lower both the RVP and C5s (increase separation) [8].

B. Relative Gain Array
Inspecting the RGA value is another approach to identify gain ratio issues [9]. The advantage of RGA over SVD is that it is scale independent. In practice, for the gains in a multivariable controller matrix, the RGA is useful in setting an interaction threshold for significant and insignificant gain ratios. For the case of the 2 × 2 matrix, G in Eq. (1), the RGA computations can be simplified [10] as: As the RGA number increases, the gain ratios in the 2 × 2 matrix approach each other and the matrix approaches singularity. When the gains are collinear, the RGA number is infinite. For an APC model, it is useful to set an RGA threshold such that any gain ratios above a threshold RGA value need to be fixed. For G in Eq. (3), the RGA is calculated to be a desirable, small value of 1.11. In industrial practice, a reasonable 2x2 RGA threshold in a typical multivariable controller could be around 12. Because RGA is independent of scaling, the RGA describes process directionality with regard to inherent process stability, while SVD, because it is considering gain magnitude, does so with regard to performance [8]. From an MPC standpoint, we may or may not be concerned from the performance point of view. Economically we still may want to use the MV with a small directional effect because it costs nothing to do so. For further details and control applications of RGA, the expository text by McAvoy [11] is an excellent resource.

III. METHODOLOGY: BINNING THE GAINS
We now present a novel, non-iterative methodology for gain matrix conditioning. The traditional approach involves tedious manual gain adjustments or a complicated optimization analysis that may not converge, until a reasonable RGA value is reached for all submatrices. Our methodology takes a user-defined RGA threshold to generate a discretized grid of K gain values with desirable properties for matrix conditioning.
Given an N × M gain matrix, there are a total of M 2 × N 2 combinations of 2x2 submatrices. By expanding the factorials in the combination terms, it follows that the cost of calculating the RGA for all submatrices is O(N 2 M 2 ). Our method mathematically guarantees that after a single-pass adjustment of all N ×M gains, by matching each gain with the closest 'bin' in K grid values, all 2x2 submatrices will have an RGA value below the threshold, with a time complexity of O(N M · log K) using a binary search implementation, without the need to inspect individual submatrices.
Consider any 2 × 2 non-zero gain matrix, G. The gains can be scaled to obtain K, such that 3 of the 4 gains are 1, with a single non-unity gain, k, that is dependent on the matrix's RGA value: The k term defines the relationship between the gain ratios and the RGA, as shown by rearranging Eq. (6) to get where Eq. (8a) defines the relationship for the column ratios, and likewise, Eq. (8b) for the rows. For RGA = ∞, k = 1 and the matrix is singular, meaning that physically, either the MVs or CVs are collinear. For RGA = 12, k = 0.9167. The intuition behind our method is that, instead of just using the matrix gains to compute k and RGA, we could also flip the relationship around to enforce a certain RGA value for any matrix. This is done by defining our desired RGA, calculating k and then adjusting the gains to equal k.
In any typical-move scaled matrix obtained with Eq. (4), the absolute values of the scaled gains lie between 0 and 1. Starting with the largest scaled gain of 1, for RGA = 12, the next-largest value is k = 0.9167 from Eq. (7). To 'condition' the matrix, gains between 1 and 0.9167 are adjusted to the closest value, either to 1, enforcing collinearity, or to 0.9167, enforcing RGA = 12.
To cover the entire range of typical move-scaled gains, we now seek a series of values, k 1 , · · · , k n such that all scaled gains can be adjusted to those values, and the entire matrix can be conditioned to any user-defined RGA threshold. We now ask, what is the next largest value of k that will ensure that the matrix meets the desired RGA = 12 threshold? It is found by multiplying K by k: Similar to the first step, the absolute value of the scaled gains between 0.9167 and 0.8403 are adjusted to the closest value, either 0.9167 or 0.8403. We can repeat this procedure to get K (3) , · · · , K (n) until we generate enough values to cover the range of the scaled gain matrix.  Fig. 3. Illustration of the binning process. A gain, g that lies in the bin interval [B i+1 , B i ] is adjusted to the closest bin boundary to output a binned gain,ĝ. Gains that lie on the midpoint of an interval are at equidistant from both bin boundaries, and will experience equal gain adjustment in either direction, as shown by the magenta arrows.
Formally, the pattern in Eq. (9) can be used to construct a recurrence relation that yields a geometric sequence of gains that satisfy a desired RGA threshold. This sequence forms the basis of a set of discretized bins. We define bins as having strictly decreasing bin boundaries, B 0 , B 1 , · · · , B n such that B i+1 < B i for i = 0, 1, · · · , n. The bin boundaries define the gain values we can use. The bin width for each interval, . This definition constructs n bins with n + 1 bin boundaries. n is chosen such that the bins cover the range of all gains in the matrix.
Let the first bin boundary be B 0 = 1. The next term, B 1 , is found with a recurrence relation derived from Eq. (9) as Solving the recurrence relation yields a closed-form solution for any i th bin boundary as a function of B 0 and RGA: Consider a scaled gain element G ij in G , with an absolute value |G ij | that lies in the interval B k+1 ≤ |G ij | < B k . The binning procedure discretizes G ij to obtain a binned gain, G ij , by assigning G ij to the closest boundary with In other words, the procedure simply adjusts the typicalmove scaled gain to the closest bin boundary, based on the absolute value of the gain. Due to the exponential term in Eq. (10), the bin widths are also strictly decreasing, such that w k+1 < w k . It follows that the maximum gain change introduced by this procedure is bounded by the width of the first bin interval [B 1 , B 0 ], with the largest gain adjustment occurring at the midpoint of [B 1 , B 0 ], denoted by B m . As shown by the magenta marker in Figure 3, a gain at the midpoint of [B 6 , B 5 ] could be assigned to either boundaries and receive an equal amount of gain adjustment.
The maximum relative gain change (%), δ max is a function of only RGA and independent of the choice of B 0 . By definition, the relative gain change is the ratio of the gain adjustment to the initial gain value, B m By substituting Eq. (10) into Eq. (12), the B 0 terms cancel out and we get δ max = RGA −1 2−RGA −1 × 100%. The maximum relative gain change percentage as a function of RGA choice is shown in Figure 4. For an RGA choice of 12, the largest relative gain change percentage would have an upper bound of 4.34%, where the 0.9583 midpoint gain is adjusted to either B 1 = 0.9167 or to B 0 = 1.0.
The selected RGA threshold builds a binning structure for gains to be fitted into. Note that applying the binning algorithm in a single pass over all gains will honor the RGA threshold for all 2x2 submatrices. Unlike the traditional method, we do not need to inspect individual submatrices and iteratively adjust the entire matrix to maintain the RGA threshold. If new gains are added to the gain matrix at a later time (e.g. controller revamps), they are simply adjusted to the existing binned structure. However, if certain gain values must be preserved (e.g. mass balance constraints), users can choose to adjust only the gains in submatrices that exceed the RGA threshold. All other gains can be left as-is.
The binning procedure is applied to the scaled gain matrix, G in Eq. (5) to obtain binned gains G as shown in Eq. (13). CV1-MV2 was adjusted by 2.5% and CV2-MV2 by 1.4%.

IV. CASE STUDY: DEBUTANIZER PROBLEM
We apply these techniques on the debutanizer process described in Figure 1. Although the operation of a debutanizer seems relatively simple, the process illustrates many complexities commonly faced by APC engineers. The raw gain matrix for the process is shown in Table I with the accompanying debutanizer schematic in Fig. 1.
The raw gains are scaled according to their typical moves using Eq. (4), and analyzed using SVD and RGA to identify ill-conditioned 2×2 submatrices based on an RGA threshold of 12 and a corresponding condition number of 59, as shown in Table II. The markers indicate that the gain is part of an  ill-conditioned 2 × 2 submatrix based on SVD results (red stars, ), and also on RGA results (blue squares, ). As shown in Table III   Applying the binning strategy of Eq. (11) to the scaled gains yields the binned gains in Table IV, which shows each gain's percentage change in red. The resulting gain matrix has three 2 × 2 submatrices with an SVD condition number above 59. To eliminate the pairs identified by the condition number threshold, larger gain changes would be required. One of the SVD-identified pairs has an RGA of 8.4. It is therefore unreasonable to remove or adjust the gains associated with this pair because of a somewhat arbitrary SVD condition number. Waller and Waller [3] came to the same conclusion related to using the condition number as a measure of ill-conditioning decades ago.
Reviewing the matrix, the binned gain matrix has 10 collinear 2×2 pairs, where there were none in the raw matrix. Table V highlights the new collinear pairings.   TABLE III   ILL Notably, the number of 3 × 3 gain combinations in the binned gain matrix above a condition number of 100 has been reduced from 34 in the raw matrix to two. The condition numbers of the two 3 × 3 matrices are 135 and 105. Similarly, the number of 4 × 4 gain combinations with a higher condition number than 100 has been reduced from 36 to 1, where that condition number is 156. By conditioning the 2 × 2 gains, the conditioning of the entire gain matrix has improved significantly. It is important to note that a high condition number on a higher-order matrix does not necessarily mean that this combination of gains will be active in an LP solution. Still, reducing the high condition number interactions means that it is impossible for such combinations to play a part in the LP solution under any situation.

V. DISCUSSIONS
How do we know if a gain pair should be collinear or not collinear? Often the answer relies on process knowledge. Second to process understanding, if a gain pair is nearly collinear, removing the degree of freedom based on a high RGA or condition number is reasonable.
In the debutanizer example, the TOP-PCT is a pressure compensated temperature, and it should be collinear with the AI-DIST-C5 composition. In the binned matrix, it is collinear since the scaled gains are equal. However, if one or more of the gains had been slightly bigger or smaller, the binning process would put the gain in a bin above or below the correct bin. If this were to happen, a degree of freedom would exist in the model where none should be. The opposite situation is also possible, where a degree of freedom is removed in a set of gains where one should exist.
Under certain circumstances, ill-conditioned gains may not be a problem in the controller. Why would we even include the TOP-PCT in the model if we have AI-DIST-C5 which serves the same purpose of measuring C5 composition? One reason is the C5 analyzer may not be reliable. If the controller is intermittently hitting the AI-DIST-C5 or TOP-PCT upper limit, with an unconditioned gain matrix, the LP may believe that temporarily it can control both CVs. As long as the CV limits are spread widely, and the gains are nearly collinear, there likely will be no observed problems. The RGA number is high, but the LP is not trading off the degrees of freedom that exist between the two CVs.
Conditioning problems for this case become apparent when the analyzer starts to drift, and the controller tries to control one CV at a high limit, and the other at a low limit. A properly conditioned model will have no difference in the gain ratios of the two CVs, and when the LP is faced with conflicting limits, it will simply give up on the least important CV. An unconditioned model will find small degrees of freedom that will allow the LP to trade off MV moves that try to fix the problem. If the CV gain ratios are a little farther from collinearity, the moves to control CVs at different limits may not be large, but the LP will drive the process in unreasonable directions and CV errors will not be reduced. When differences in gain ratios of collinear variables exist, the controller LP may find a different solution based on which CV limit is active. This can lead to optimization cost tuning in order to get the controller to behave the same way when faced with high or low limits on either variable.
The gain issues described can spread to higher dimensions as well, and the issues are often magnified when the optimizer becomes infeasible. As long as the MV directions that try to solve the infeasibility are the same as the feasible economic direction, little changes may be noticed when the solution becomes infeasible. When the infeasible direction opposes the economic direction, the controller can make large moves in an uneconomic direction trying to reduce or eliminate CV error. From these behaviors, the practitioner may erroneously conclude that the model is inaccurate and needs re-evaluation. A "good" but unconditioned model can account for this behavior. Retesting to fix an ill-conditioned matrix is unlikely to solve the problem.
These simple examples show why binning the gains is not the final answer, rather the first step in the overall model gain analysis. The "correct" structure of the gain matrix must be determined based on plant data and process understanding. The binning procedure makes the next step of validating the model gain structure much easier. In a real-world MPC project, the practitioner must perform an interaction analysis of the model gains by simulating the LP solution.

VI. CONCLUSIONS
In this paper, we have demonstrated a binning procedure to condition the gain matrix used in industrial MPCs. By conditioning the matrix, we remove small degrees of freedom that are either insignificant or zero, without large perturbations to the process gains. This gain conditioning step prevents the optimizer from erroneously exploiting small degrees of freedom for control. The binning approach can also include small gains in the model to maintain gain consistency and respect material balances, without introducing further conditioning problems.
There are several areas of interesting future work here. The tables presented earlier for identifying gain interactions using markers are quite basic -as these tables grow in size, this colour-based strategy can result in cluttered and unintuitive displays, making near-collinearity analysis visually difficult. There is an opportunity for improving the visualization of degrees of freedom and gain ratios. We have made some recent progress in visualizing the LP solution for troubleshooting and understanding online MPCs [12].
Unlike SVD, higher-order RGA calculations rely on the matrix inverse, which does not exist for a rank-deficient matrix with collinearities. Therefore, to identify conditioning issues in larger submatrices, the RGA is mathematically undefined and cannot be used, and SVD may not be entirely reliable either due to scaling issues. In industry, higherdimensional analysis of the gain matrix beyond 3 × 3 submatrices are often not investigated by practitioners. It may be that a higher order RGA analysis could be useful once all 2 × 2 submatrices have been conditioned to the desired RGA threshold. The goal, in practice, has always been to certify that a model is conditioned to the dimensionality of the problem. Here the dimensionality is defined as a square case, corresponding to the lesser of the number of MVs or the number CVs. For a non-square SVD case, the extra variables generally guarantee that there are significant directions related to the degrees of freedom in the matrix. With SVD, it seems reasonable that for a specific problem, we should be able to identify physically meaningful orthogonal directions in the model, and then force the model gains to follow combinations of these directions.


Versatile Skill Control via Self-supervised Adversarial Imitation of Unlabeled Mixed Motions

Learning diverse skills is one of the main challenges in robotics. To this end, imitation learning approaches have achieved impressive results. These methods require explicitly labeled datasets or assume consistent skill execution to enable learning and active control of individual behaviors, which limits their applicability. In this work, we propose a cooperative adversarial method for obtaining single versatile policies with controllable skill sets from unlabeled datasets containing diverse state transition patterns by maximizing their discriminability. Moreover, we show that by utilizing unsupervised skill discovery in the generative adversarial imitation learning framework, novel and useful skills emerge with successful task fulfillment. Finally, the obtained versatile policies are tested on an agile quadruped robot called Solo 8 and present faithful replications of diverse skills encoded in the demonstrations.

Abstract-Learning diverse skills is one of the main challenges in robotics. To this end, imitation learning approaches have achieved impressive results. These methods require explicitly labeled datasets or assume consistent skill execution to enable learning and active control of individual behaviors, which limits their applicability. In this work, we propose a cooperative adversarial method for obtaining single versatile policies with controllable skill sets from unlabeled datasets containing diverse state transition patterns by maximizing their discriminability. Moreover, we show that by utilizing unsupervised skill discovery in the generative adversarial imitation learning framework, novel and useful skills emerge with successful task fulfillment. Finally, the obtained versatile policies are tested on an agile quadruped robot called Solo 8 and present faithful replications of diverse skills encoded in the demonstrations.
Index Terms-Reinforcement learning, generative adversarial imitation learning, unsupervised skill discovery

I. INTRODUCTION
Reinforcement Learning (RL) has demonstrated its capability of learning diverse and complex skills for robotic platforms. In the field of legged systems, RL has achieved success in learning-based quadrupedal locomotion control in challenging environments [1]- [4]. Typically, deep RL techniques learn desired behaviors motivated by optimizing reward functions that are specifically tailored for the training task. As a result, it can sometimes become very demanding to develop complex skills with reward engineering, where various terms of motivation and regularization need to be carefully designed and balanced.
Given the availability of some expert references, Imitation Learning (IL) allows an agent to learn to reproduce expert behavior. In particular, Generative Adversarial Imitation Learning (GAIL, [5]) employs Generative Adversarial Networks (GANs, [6]), which train a policy to deceive an imitation discriminator that constantly tries to distinguish state transitions generated between the policy and the reference data distribution. The output of the discriminator is used as a training signal that encourages the learning agent to generate similar behaviors to the demonstration [7], [8]. However, given a large dataset of unlabeled motion clips with diverse behaviors, extracting and learning individual sensible skills can be challenging. Such scenarios are commonly encountered in motion captures from creatures or underactuated experts, whose motion execution produces intrinsic noise and inconsistency and thus consists of miscellaneous behaviors. Due to the unknown skill types in the dataset, supervised learning Fig. 1: Method overview. Cooperative Adversarial Selfsupervised Skill Imitation (CASSI) enables skill extraction and learning from unlabeled diverse references by motivating diversity in a generative adversarial imitation learning setting using unsupervised learning techniques. The resulting versatile policy allows active control of individual skills. Supplementary videos and implementation details for this work are available at https://sites.google.com/view/icra2023-cassi/home. techniques fail to find correspondence between the policy and the individual skills it replicates.
In this case, unsupervised learning methods could help to address the problem. Unsupervised skill discovery has enabled learning agents to obtain intrinsic behaviors without explicit task rewards. One promising attempt for unsupervised skill discovery is based on maximizing the discriminability of skills represented by latent variables on which a policy is conditioned [9]- [13]. Policy training signals are often constructed from variational approximations of the mutual information between latent variables and traversed state histories using a learned skill discriminator. Such mutual information encodes both the diversity in state transitions and the predictability of them given known latent skills [12]. In this setting, strong discriminating performances are expected cooperatively by both the policy and the discriminator, where the latter is trained with supervised learning techniques. The resulting policy is capable of producing consistently distinguishable behavioral patterns, or skills.
In this work, we present Cooperative Adversarial Selfsupervised Skill Imitation (CASSI), where we show that by applying unsupervised skill discovery in a generative adversarial imitation learning framework, a quadrupedal system manages to learn versatile policies that extract sensible high-level skills from a reference dataset containing diverse unlabeled motions (Fig. 1). The learned policy enables active control of the individual skills embedded in the reference dataset, which manifests itself by its execution of consistently distinguishable behaviors. The policy is also deployed and tested on the real robot without further adaptation. To the best of our knowledge, this is the first time that individual skills are extracted and learned simultaneously from an unlabeled dataset motivated by unsupervised RL algorithms. In summary, our contributions include: (i) A cooperative adversarial self-supervised skill imitation method for extracting and learning diverse skills from unlabeled references. (ii) Fidelity and diversity evaluation of learned versatile policies using an oracle classifier. (iii) Experimental validation in simulation and on a quadruped robot.

II. RELATED WORK
Recent development in robotics has created many potential applications that require intelligent systems to not only make decisions but also perform expected physical movements. While learning a task is often formulated as an optimization problem, it has become widely recognized that having prior knowledge provided by an expert can result in more effective and efficient learning than attempting to solve the problem from scratch [14], [15].
Instead of relying on a handcrafted imitation objective, GAIL techniques train an adversarial discriminator to distinguish between behaviors generated by the agent and expert demonstrations [5], [16]. The effectiveness of imitation is thus measured by the performance of the discriminator, whose output encodes the similarity descriptions. To extend the applicability of GAIL to accommodate large datasets of unstructured motion clips, [7] employ an adversarial RL procedure that automatically selects the motion to perform in the reference, dynamically interpolating and generalizing from the dataset. While this method has achieved success in learning adaptive motions, active control over individual skill execution relies still on labeled motion datasets. To this end, one may either collect disparate reference motions and learn individual skills separately [8], or stack the training of multiple discriminators corresponding to each skill in parallel [17]. Both methods require supervision from prior knowledge of motion labels in the datasets. However, given a large dataset of diverse unlabeled motion clips, enabling autonomous acquisitions of individual skills requires unsupervised training techniques.
Unsupervised RL is motivated by intrinsic skill development of intelligent agents that are believed to learn in the absence of supervision in order to acquire repurposable task-agnostic knowledge. These skills can be quickly and efficiently utilized when specific tasks are later defined. Achieving this goal requires specifying a learning objective that ensures that each skill individually is distinct and that the skills collectively explore large parts of the state space [10]. In this setting, mutual information is commonly established as a notion of empowerment of an intrinsically motivated agent [18]. It is shown that a discriminability objective is equivalent to maximizing the mutual information between the skill and some aspect of the induced trajectory [19], [20]. [9], [21] try to maximize the mutual information between the skill and initial and final states. [10], [13] attempt to maximize the mutual information between the skill and states along the trajectory. [12], [22] propose to maximize the mutual information between the skill and the following state conditioned on the current state. With the skill diversity motivated by mutual information maximization in various settings, diverse skills are acquired with minimal supervision. Finally, unsupervised RL algorithms have been shown effective in conjunction with task rewards, which serve as sparse guidance for generating regularized and robust behaviors [23], [24].

III. APPROACH
In this section, we describe our method, which enables learning from unlabeled demonstrations in a generative adversarial setting using unsupervised skill discovery techniques. In the following context, we denote the discriminator distinguishing state transitions from the policy and reference in the generative adversarial imitation learning as the imitation discriminator (d ψ ), and the discriminator distinguishing different skills yielded by the policy in the unsupervised skill discovery as the skill discriminator (q φ ).

A. Generative Adversarial Imitation
Our generative adversarial imitation framework is built upon the Adversarial Motion Prior (AMP, [7]). In this framework, the imitation discriminator reflects the imitation performance and thus specifies the training signal for the policy. It is crucial to select an appropriate set of features that provide effective learning feedback. We consider the imitation observation space for the reference observations and s t = (s t−H I +1 , . . . , s t ) for the states induced by the policy. For clarity, we omit the time index in the following. To simplify notation, we write f I (s) to express that each state in s is mapped to O I . In our experiments, we select linear and angular velocities of the robot base in the robot frame, measurement of the gravity vector in the robot frame, the base height, and joint angular position and velocity as the observation space O I . As such, the discriminator's goal in this setup is to distinguish samples of the policy transition distribution d π from the reference motion distribution d M .
1) Imitation Discriminator Formulation: Similar to [7], we use the least-squares GAN (LSGAN) loss [25] for discriminator optimization. Using H I -step inputs and a gradient penalty, the discriminator objective is formulated as where the last term denotes the penalty for non-zero gradients on samples from the dataset with weight w GP to stabilize training [7].
Intuitively, the LSGAN loss forces the discriminator to output +1 for samples from the reference motion and −1 for those from the policy. In contrast, the policy is trained to deceive the discriminator by generating transition patterns similar to those present in the reference dataset. The reward function for training the policy is then given by The imitation reward r I provides a well-scaled output bounded between 0 and 1 which eases downstream policy learning.

B. Mutual Information Maximization
Our unsupervised skill discovery framework is built upon Diversity Is All You Need (DIAYN, [10]) with the Discriminator Disagreement Intrinsic reward (DISDAIN, [13]) for exploration motivation.
In the unsupervised RL setting, the agent seeks to develop a set of skills, indexed by a latent variable z ∈ Z and represented by a policy π θ (· | s, z) parameterized by θ. The latent skills z ∼ p z are sampled at the beginning of each trajectory and then fixed over the episode. Therefore, each skill z represents a temporally extended behavior within a state sequence of length H S . Similarly, we consider a function f S : S → O S that maps the full state space S to the skill observation space O S . Thus, the transition in the extracted features can be represented by where τ S denotes the state visitation distribution along the trajectory. Again, we omit the time index in the following for clarity. Clearly, trajectories sampled from the policy conditioned on a particular skill depend on the selection of z. We highlight this dependence by writing o S z . 1) Skill Discriminator Observation Space: Since skills refer to consistently distinguishable behavioral motifs in the states of interest, how a skill distinguishes itself from other skills, or discriminability between skills, depends crucially on the definition of the skill observation space O S . In unsupervised RL literature, skills are often identified over only simple state transition patterns (e.g. different moving directions, speeds, or reached positions) [9], [10], [12], [13], as it is typically hard to extract high-level skill descriptions without any supervision. In our work, as the policy space is constrained by an imitation target, we are able to define a skill observation space O S that specifies high-level distinction (e.g. trot, leap).
Note that the selection of states in O S depends on users' interest. If we are interested in obtaining different base motion patterns, including only the base information in O S suffices. If we also aim to recover transitions with different joint configurations (e.g. different gaits) in the reference dataset, including joint information will help with discrimination. In addition, we assume that the skill space Z is discrete and with cardinality N z , although much of the discussion may extend to continuous skill space.
2) Variational Approximation of Mutual Information: A large and growing class of objectives for unsupervised skill discovery are derived from maximizing the mutual information between the latent skill z and the resulting trajectories of extracted features o S z . The formulation is expressed as In practice, a variational approximation of the intractable conditional distribution p(z | o S z ) with a learned parametric model q φ (z | o S z ) is often applied to obtain a lower bound of F(θ) [26]. The model q φ (z | o S z ) is commonly referred to as a discriminator, as it is trained to discriminate between skills from state transitions. In our work, it is termed as the skill discriminator, to distinguish itself from the imitation discriminator (d ψ ) used in the generative adversarial imitation setting.
Optimizing the lower bound with respect to the policy parameters θ corresponds to training the policy with a skill reward To learn a full repertoire of skills, the skill prior is typically fixed to be uniform [10], [21], [27], in which case − log p z (z) = log N z . An arbitrary discriminator which simply ignores the trajectory will give a zero skill reward. In contrast, a perfect discriminator will yield a skill reward of log N z [13].
To tighten the lower bound, the discriminator should approximate the true distribution p(z | o S z ), which results in a supervised learning setting where the training data is collected from policy roll-outs. In our work, the skill discriminator is implemented as a classifier whose objective is to assign the correct skill to a given style observation by minimizing the cross-entropy classification loss.
As such, both the policy and the skill discriminator expect a strong discriminating performance by maximizing and tightening the lower bound of the mutual information cooperatively.
3) Policy Exploration Bonus: Ideally, the policy presents different state transitions conditioned on different skills. The discriminator receives these transitions and attempts to decode the original skill z. However, at the beginning of the training, before the policy relates its behavior to the skill, the agent may yield similar transitions under different latent variables z. As the skill is sampled randomly, this will generate mislabeled data which confuses the training of the discriminator. When the discriminator fails to distinguish the transition patterns, the skill reward will be trivial and thus does not generate any information for the policy to diversify its behaviors. This dead loop is termed as the pessimistic exploration problem in unsupervised skill discovery [13]. The reason behind this is that when the policy tries to maximize the skill reward r S , it tends to reduce the epistemic uncertainty by visiting the same states it has visited before. To encourage exploration, [13] propose the use of a discriminator ensemble of size N and a resulting DISDAIN reward that compensates for the increase of the epistemic uncertainty when the policy visits new states: Fig. 2: System overview. Given an unlabeled dataset, an imitation discriminator learns sampled state transition patterns from demonstrations. At the beginning of each episode, a latent skill variable z is sampled as input to the policy. A skill discriminator is trained to decode the original skill from these motions. The policy is rewarded for making the skills distinguishable.
where φ i corresponds to the parameters of the ensemble member i. For trajectories with rich training data for the skill discriminator, the ensemble members should agree and result in a zero DISDAIN reward. In contrast, for state transitions of high discriminator disagreement, this reward will be positive and thus encourage exploration.

C. Overview
Note that the imitation reward r I , the skill reward r S , and the DISDAIN reward r D are task-agnostic. This allows the training of primitive skills that facilitate downstream learning such as hierarchical RL when a task is later specified. Alternatively, we are able to learn high-level tasks alongside self-supervised skill extraction. In this setting, we define task reward r T , in addition to some regularization terms r R that enforce stable policy outputs on the real platform. Putting everything together, the total reward that the policy receives encompasses five parts where w denotes the weight of each term and w D = w R = 1.0 is kept constant throughout our work. The joint optimization of the policy and the imitation discriminator utilizes adversarial training. The imitation discriminator tries to distinguish state transitions between the agent and the reference motion, whereas the policy tries to make this difficult by generating similar behaviors. Meanwhile, the joint training of the policy and the skill discriminator forms a cooperative game. The agent samples a skill and tries to present a distinct behavior for the ease of discrimination by the skill discriminator, and the skill discriminator tries to decode the original skill and yield training signals to reward the policy for diversifying its skills. Figure 2 provides a schematic overview of our method, and an algorithm overview is detailed in Algorithm 1. sample transition mini-batches b π ∼ B with bagging 20: update q φ using b π with cross-entropy or loss 21: end for 22: end for

IV. EXPERIMENTS
We evaluate our method on the Solo 8 robot, an opensource research quadruped robot that performs a wide range of physical actions [28], in simulation and on the real system.
The unlabeled motion dataset is constructed using mixed trajectories induced by individual expert policies learned in a previous work [8], including crawl, walk, trot, leap, wave, and stilt. These colors are used consistently throughout the paper. For each motion, 1000 trajectories are recorded in simulation from parallel robot instances with randomized mechanical properties. Each trajectory contains 120 consecutive time steps. To break down the consistency in skill execution, these consecutive state transitions are sliced into small motion clips of horizon length 8. In all of our experiments, we use Proximal Policy Optimization (PPO, [29]) in Isaac Gym [30].

A. Skill Extraction and Imitation
In this section, we set task reward weight r T = 0 and compare CASSI with spectral clustering and AMP in extracting and learning skills from unlabeled datasets.
1) Spectral Clustering: A potential strategy to obtain individual skills from unlabeled references is to perform preclustering on the dataset and learn each separately. To test its effectiveness, we perform spectral clustering with k-Nearest Neighbors on the reference dataset, compute the error rate with respect to the true labels, and compare it with the skill discriminator learned during the training of our method. We report the best result in Table I.  Partition generated via spectral clustering, whose error reports 43.57% (right).
The result reveals that the skill discriminator learned using our method with a horizon of only 8 achieves better discriminating performance than spectral clustering on the whole trajectories. This indicates that for spectral clustering to work with decent accuracy, strong assumptions of consistent skill execution over trajectories with much longer horizons must be made. The spectral clustering label assignment is visualized using t-SNE in Fig. 4 next to the ground-truth labels.
2) Oracle Classifier: As we construct the dataset by mixing individual expert motion clips, we are able to evaluate the diversity and fidelity of the learned skills by training an oracle classifier C using ground-truth labels. We denote the true label by c ∈ C and the true number of skills present in the dataset by N c , where C denotes the true skill space. In our experiments, C = {crawl, walk, trot, leap, wave, stilt} and N c = 6. Note that N c is unknown to the design of the skill discriminator. Thus, the predefined number of skills N z is a tuning parameter.
We write p(c | o S z ) to represent the probability of the oracle classifier predicting the true skill c given a state trajectory of horizon length H C , which is induced by the policy conditioned on a sampled latent skill z. When the policy conditioned on a skill z is able to generate a state trajectory that aligns well with the transition patterns exerted by a certain skill c in the true skill space C, the probability p(c | o S z ) is large for this specific skill c ∈ C and small for all other skills c ∈ C\{c}. We thus can quantify the diversity and fidelity of the skills learned by the policy π θ , using the following entropy expressions: Intuitively, fidelity is maximized if the policy is very certain that every generated trajectory corresponds to a true skill present in the reference dataset when conditioned on a latent skill. In contrast, diversity is maximized if the policy is able to generate trajectories corresponding to different true skills present in the reference dataset when conditioned on different latent skills.
We visualize the training performance in terms of div(π θ ), fid(π θ ), and p(c | o S z ) in Figures 5 and 6 for w I = 1.0, w S = 0.5, and N z = 6. Note that under the assumption N z = N c , the learned policy establishes a bijective correspondence g : Z → C, where all reference motions are assigned to distinct latent variables. In addition, the policy encounters skill collapse with AMP only, due to the absence of the skill reward r S which promotes discriminability among learned behaviors. We further analyze cases with N z = N c in Sec. IV-B.

B. Task Execution
In this section, we evaluate the skills learned using CASSI with an additional velocity tracking task, which is achieved by maximizing the reward r T = exp{−σ −2 (u − v x ) 2 }, where u denotes the velocity command sampled within the range [0, 1]. We denote by v x the linear velocity of the robot base, expressed in the robot frame, and set the constant scaling factor σ 2 = 0.25. The task reward weight is set constant w T = 1.0.
1) Novel Skill Discovery: In practice, motion types presented in the reference dataset are often unknown. Without prior knowledge of the actual number of component motions N c , the choice of the latent skill cardinality N z may require additional effort. We evaluate skill extraction performance for different N z using the oracle classifier. Figure 7b reveals that when N z = N c , all skills in the reference dataset are extracted and assigned to a distinct latent variable z. In contrast, for N z = N c , it is observed CASSI AMP  In contrast, CASSI encourages each latent skill z to converge to a distinct reference motion c. As the mapping between Z and C may differ between runs, the plots show results after aligning such correspondence across seeds.
that new skills emerge in a form of motion interpolation, whose existence is reflected by mixed predictions by the oracle classifier given a latent skill z (e.g. z = 2, 3 in Fig. 7a and z = 6, 7 in Fig. 7c). These skills present "mixed" features of the references. In addition, the skill-conditioned task rewards r T (z) report consistently high value for all latent skills with different choices of N z . This indicates that the new skills also yield high task performance while presenting mixed motion patterns. Within the scope of the defined task, these discovered skills are considered to be both novel and useful. The policy training using CASSI can be understood as a matching procedure between the latent skills and the reference skills. In this setting, the true skills c ∈ C can be viewed as vertices that define a skill space. The imitation reward r I motivates the policy to learn behaviors that stay within this skill space, while the skill reward r S motivates the behavior associated with each z to stay far away from the averaged behavior [31]. For N z = N c , each z takes a vertex of the skill space and is assigned to a distinct c ∈ C. For N z > N c , more  latent variables need to be fit into the skill space. Once the vertices are taken (pure skills), the remaining z have to stay in the interior of the space (mixed skills). For N z < N c , the latent variables have larger freedom to choose their relative position in the skill space to stay distant from each other, presenting both "pure" and "mixed" behaviors.
2) Real-system Deployment: We deploy the learned policy on the real system and record 10 state transition trajectories of 120 time steps when conditioned on each skill. The recorded motions are evaluated again with the oracle classifier as depicted in Fig. 8. We also illustrate the continuous change of behaviors within a trajectory by switching latent skill variables in Fig. 9. Figure 3 provides a motion sequence on Solo 8 with skill execution by the versatile policy.

V. CONCLUSION
In this work, we propose an adversarial imitation method named CASSI that is capable of extracting and learning individual skills from unlabeled datasets with diverse behaviors by utilizing unsupervised skill discovery techniques. Our results highlight the importance of the skill reward, whose absence results in skill collapses. The experiments also indicate that CASSI allows extracting robust versatile policies capable of active skill control with both high fidelity and diversity with respect to the original reference motions. Furthermore, policies can be learned with task specifications and are able to transfer to the real system without further adaptation. Finally, our experiments confirm the emergence of novel skills that yield high task performance and present common features shared by original reference motions.