summaries: This paper focuses on semi-supervised learning with outlier unlabeled data. This problem has attracted much attention in the semi-supervised learning community. The authors propose a method that contains two parts: First, detect outliers based on the OVA classifier. Second, propose an open-set soft-consistency regularization loss that can help improve the outlier detection performance.

summaries: This manuscript introduces OpenMatch for open-set semi-supervised learning. The OpenMatch integrates a one-vs-all classifier (OVA-classifier, working as an outlier detector) and FixMatch. The main contribution should be the soft open-set consistency regularization (SOCR) in Figure 1 and the OpenMatch Framework in Alg 1. Specifically, the SOCR follows the same self-supervised framework in FixMatch but it uses OVA-classifier to output consistent anomaly score distribution. After using the model in Figure 1 to identify inliers from unlabeled data, the FixMatch is allowed to reach a higher accuracy on the OSSL tasks.

summaries: This paper tackles open-set semi-supervised learning (OSSL) where outliers exist in unlabeled data. The key idea for detecting outliers is to learn class-specific, one-vs-all (OvA) classifiers—if all classifiers produce negative outputs for an unlabeled data, the data is considered an outlier. In addition to the classification loss computed on labeled data, the OvA classifiers are also trained on unlabeled data with an entropy minimization loss and a consistency regularization loss—the latter is computed on two views of the same data. The results on CIFAR and ImageNet show that the proposed approach largely outperforms the baselines in terms of both recognition accuracy and outlier detection accuracy.

summaries: The authors propose an idea to improve model's performance under the open-set semi-supervised learning setting. This work proposed to leverage OOD detector, combined with FixMatch, and soft open-set consistency regularization (SOCR) to improve the model training during open-set semi-supervised learning. I think the usage of OOD on ignore outliers is not new and an unsupervised method on learning the consistency between augmentations for unlabeled data is also not new. Thus, I am not sure if I can find too much of novelty from this work itself, but I do appreciate the author's effort in putting them together to be a complete model, which in fact make solid progress on the open-set semi-supervised learning setting.

summaries: The paper proposes a model for open-set semi-supervised learning. As a semi-supervised learning model, it tackles classification where only a portion of the training data is labeled. "Open set" refers to the fact that the unlabeled data are noisy and can contain out of distribution examples, for which the class is not among the known classes. Out of distribution samples are present also during test, and the model has to recognize them and avoid classification on them. The proposed model builds on top of existing works, combining several loss terms and ideas such as one-versus-all classifiers and FixMatch as the main semi-supervised learning engine. The main technical contribution is a loss term enforcing the out-of-distribution score for two augmented versions of the same unlabeled example to be similar. Extensive experiments validate the proposal against recent state-of-the-art models.

abstractText: Semi-supervised learning (SSL) is an effective means to leverage unlabeled data to improve a model’s performance. Typical SSL methods like FixMatch assume that labeled and unlabeled data share the same label space. However, in practice, unlabeled data can contain categories unseen in the labeled set, i.e., outliers, which can significantly harm the performance of SSL algorithms. To address this problem, we propose a novel Open-set Semi-Supervised Learning (OSSL) approach called OpenMatch. Learning representations of inliers while rejecting outliers is essential for the success of OSSL. To this end, OpenMatch unifies FixMatch with novelty detection based on one-vs-all (OVA) classifiers. The OVA-classifier outputs the confidence score of a sample being an inlier, providing a threshold to detect outliers. Another key contribution is an open-set soft-consistency regularization loss, which enhances the smoothness of the OVA-classifier with respect to input transformations and greatly improves outlier detection. OpenMatch achieves stateof-the-art performance on three datasets, and even outperforms a fully supervised model in detecting outliers unseen in unlabeled data on CIFAR10. The code is available at https://github.com/VisionLearningGroup/OP_Match.

authors:
David Berthelot
Nicholas Carlini
Ekin D Cubuk
Alex Kurakin
Kihyuk Sohn
Han Zhang
Colin Raffel

title: Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring

venue: arXiv preprint arXiv:1911.09785,

authors:
David Berthelot
Nicholas Carlini
Ian Goodfellow
Nicolas Papernot
Avital Oliver
Colin Raffel

title: Mixmatch: A holistic approach to semi-supervised learning

authors:
Pau Panareda Busto
Juergen Gall

title: Open set domain adaptation

venue: In Int. Conf. Comput. Vis.,

authors:
Ting Chen
Simon Kornblith
Mohammad Norouzi
Geoffrey Hinton

title: A simple framework for contrastive learning of visual representations

venue: In Int. Conf. Mach Learn.,

authors:
Ting Chen
Simon Kornblith
Kevin Swersky
Mohammad Norouzi
Geoffrey Hinton

title: Big selfsupervised models are strong semi-supervised learners

venue: arXiv preprint arXiv:2006.10029,

authors:
Yanbei Chen
Xiatian Zhu
Wei Li
Shaogang Gong

title: Semi-supervised learning under class distribution mismatch

venue: In AAAI,

authors:
Mircea Cimpoi
Subhransu Maji
Iasonas Kokkinos
Sammy Mohamed
Andrea Vedaldi

title: Describing textures in the wild

venue: In IEEE Conf. Comput. Vis. Pattern Recog.,

authors:
Ekin D Cubuk
Barret Zoph
Jonathon Shlens
Quoc V Le

title: Randaugment: Practical automated data augmentation with a reduced search space

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: Imagenet: A large-scale hierarchical image database

authors:
Terrance DeVries
Graham W Taylor

title: Learning confidence for out-of-distribution detection in neural networks

venue: arXiv preprint arXiv:1802.04865,

authors:
Chuanxing Geng
Sheng-jun Huang
Songcan Chen

title: Recent advances in open set recognition: A survey

venue: IEEE transactions on pattern analysis and machine intelligence,

authors:
Atin Ghosh
Alexandre H Thiery

title: On data-augmentation and consistency-based semi-supervised learning

venue: arXiv preprint arXiv:2101.06967,

authors:
Gregory Griffin
Alex Holub
Pietro Perona

title: Caltech-256 object category dataset

venue: Tech Report,

authors:
Lan-Zhe Guo
Zhen-Yu Zhang
Yuan Jiang
Yu-Feng Li
Zhi-Hua Zhou

title: Safe deep semi-supervised learning for unseen-class unlabeled data

authors:
Kaiming He
Haoqi Fan
Yuxin Wu
Saining Xie
Ross Girshick

title: Momentum contrast for unsupervised visual representation learning

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Dan Hendrycks
Kevin Gimpel

title: A baseline for detecting misclassified and out-of-distribution examples in neural networks

venue: arXiv preprint arXiv:1610.02136,

authors:
Dan Hendrycks
Mantas Mazeika
Thomas Dietterich

title: Deep anomaly detection with outlier exposure

venue: arXiv preprint arXiv:1812.04606,

authors:
Dan Hendrycks
Mantas Mazeika
Saurav Kadavath
Dawn Song

title: Using self-supervised learning can improve model robustness and uncertainty

venue: arXiv preprint arXiv:1906.12340,

authors:
Aditya Khosla
Nityananda Jayadevaprakash
Bangpeng Yao
Fei-Fei Li

title: Novel dataset for fine-grained image categorization: Stanford dogs

venue: In Proc. CVPR Workshop on Fine-Grained Visual Categorization (FGVC),

authors:
Alex Krizhevsky
Geoffrey Hinton

title: Learning multiple layers of features from tiny images

venue: arXiv preprint,

authors:
Samuli Laine
Timo Aila

title: Temporal ensembling for semi-supervised learning

title: Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks

venue: In Workshop on challenges in representation learning,

authors:
Shiyu Liang
Yixuan Li
Rayadurgam Srikant

title: Enhancing the reliability of out-of-distribution image detection in neural networks

venue: arXiv preprint arXiv:1706.02690,

authors:
Dongju Liu
Jian Yu

title: Otsu method and k-means

venue: Ninth International Conference on Hybrid Intelligent Systems,

authors:
Laurens van der Maaten
Geoffrey Hinton

title: Visualizing data using t-sne

authors:
Takeru Miyato
Shin-ichi Maeda
Masanori Koyama
Ken Nakae
Shin Ishii

title: Distributional smoothing with virtual adversarial training

authors:
Yuval Netzer
Tao Wang
Adam Coates
Alessandro Bissacco
Bo Wu
Andrew Y Ng

title: Reading digits in natural images with unsupervised feature learning

venue: In NIPS workshop on deep learning and unsupervised feature learning,

authors:
M-E Nilsback
Andrew Zisserman

title: A visual vocabulary for flower classification

authors:
Shreyas Padhy
Zachary Nado
Jie Ren
Jeremiah Liu
Jasper Snoek
Balaji Lakshminarayanan

title: Revisiting one-vs-all classifiers for predictive uncertainty and out-of-distribution detection in neural networks

authors:
Salah Rifai
Yann N Dauphin
Pascal Vincent
Yoshua Bengio
Xavier Muller

title: The manifold tangent classifier

venue: Adv. Neural Inform. Process. Syst.,

authors:
Kuniaki Saito
Kate Saenko

title: Ovanet: One-vs-all network for universal domain adaptation

venue: arXiv preprint arXiv:2104.03344,

authors:
Kuniaki Saito
Shohei Yamamoto
Yoshitaka Ushiku
Tatsuya Harada

title: Open set domain adaptation by backpropagation

venue: In Eur. Conf. Comput. Vis.,

authors:
Mehdi Sajjadi
Mehran Javanmardi
Tolga Tasdizen

title: Regularization with stochastic transformations and perturbations for deep semi-supervised learning

venue: arXiv preprint arXiv:1606.04586,

authors:
Kihyuk Sohn
David Berthelot
Chun-Liang Li
Zizhao Zhang
Nicholas Carlini
Ekin D Cubuk
Alex Kurakin
Han Zhang
Colin Raffel

title: Fixmatch: Simplifying semi-supervised learning with consistency and confidence

authors:
Jihoon Tack
Sangwoo Mo
Jongheon Jeong
Jinwoo Shin

title: Csi: Novelty detection via contrastive learning on distributionally shifted instances

venue: arXiv preprint arXiv:2007.08176,

authors:
Antti Tarvainen
Harri Valpola

title: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

venue: arXiv preprint arXiv:1703.01780,

authors:
Vikas Verma
Kenji Kawaguchi
Alex Lamb
Juho Kannala
Yoshua Bengio
David Lopez-Paz

title: Interpolation consistency training for semi-supervised learning

authors:
Catherine Wah
Steve Branson
Peter Welinder
Pietro Perona
Serge Belongie

title: The caltech-ucsd birds-200-2011 dataset

authors:
Jim Winkens
Rudy Bunel
Abhijit Guha Roy
Robert Stanforth
Vivek Natarajan
Joseph R Ledsam
Patricia MacWilliams
Pushmeet Kohli
Alan Karthikesalingam
Simon Kohl

title: Contrastive training for improved out-of-distribution detection

venue: arXiv preprint arXiv:2007.05566,

authors:
Qizhe Xie
Zihang Dai
Eduard Hovy
Minh-Thang Luong
Quoc V Le

title: Unsupervised data augmentation for consistency training

authors:
Kaichao You
Mingsheng Long
Zhangjie Cao
Jianmin Wang
Michael I. Jordan

title: Universal domain adaptation

authors:
Fisher Yu
Ari Seff
Yinda Zhang
Shuran Song
Thomas Funkhouser
Jianxiong Xiao

title: Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop

venue: arXiv preprint arXiv:1506.03365,

authors:
Qing Yu
Daiki Ikami
Go Irie
Kiyoharu Aizawa

title: Multi-task curriculum framework for open-set semi-supervised learning

venue: In European Conference on Computer Vision,

authors:
Sergey Zagoruyko
Nikos Komodakis

title: Wide residual networks

venue: arXiv preprint arXiv:1605.07146,

authors:
Xiaohua Zhai
Avital Oliver
Alexander Kolesnikov
Lucas Beyer

title: S4l: Self-supervised semi-supervised learning

heading: 1 Introduction

text:
Semi-supervised learning (SSL) leverages unlabeled data to improve a model’s performance [27, 38, 2, 1, 35, 22, 37]. An SSL model can propagate the class information of a small set of labeled data to a large set of unlabeled data, which significantly improves the recognition accuracy without any additional annotation cost. A common assumption of SSL is that the label spaces of labeled and unlabeled data are identical, but, in practice, the assumption is easily violated. Depending on how it was collected, the unlabeled data may contain novel categories unseen in the labeled training data, i.e., outliers. Since these outliers can significantly harm the performance of SSL algorithms [14], detecting them is necessary to make SSL more practical. Ideally, a model should classify samples of known categories i.e., inliers, into correct classes while identifying samples of novel categories as outliers. This task is called Open-set Semi-supervised Learning (OSSL) [44]. While OSSL is a more realistic and practical scenario than standard SSL, it has not been as widely explored.
Existing strong SSL methods [35, 41] do not work well for OSSL. For example, FixMatch [35] generates pseudo-labels using the model’s predictions on weakly augmented unlabeled images and trains the model to match its predictions on strongly augmented images with the pseudo-labels. This method exploits the advantage of pseudo-labeling as well as regularizing a model with the consistency between differently augmented images. But, in OSSL, this risks assigning pseudo-labels of known categories to outliers, which degrades the recognition accuracy. A possible solution is to compute the SSL objective only for unlabeled samples considered to be inliers, where confidence thresholding is used to pick inliers. For instance, MTC [44] regards some proportion of samples as outliers by using Otsu thresholding [25]. However, this is not robust to varying proportions of outliers as discussed in
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
[44]. DS3L [14] proposes meta-optimization that attempts to pick unlabeled data useful to improve generalization performance. But, this method does not have an objective to separate inliers from outliers.
Given the limitations of the existing methods, we aim to learn representations that separate outliers from inliers in a feature space and a threshold effective for detecting outliers. The challenge is learning such representations with a small number of labeled inliers and no supervision to find outliers. Moreover, choosing an accurate threshold value is not a trivial problem.
We propose a new framework, OpenMatch, to address the above drawbacks of OSSL. First, we propose to utilize a One-Vs-All (OVA) network [32] that can learn a threshold to distinguish outliers from inliers. A separate OVA-classifier is trained for each class, and a sample is labeled an outlier if all of the classifiers determine it to be one. Thus, this technique allows us to identify outliers in an unsupervised way. We call this an outlier detector. Second, we propose a novel open-set soft-consistency loss to learn more effective representations for detecting outliers. We first transform an unlabeled input in two ways and obtain two logits from the outlier detector. Then, we minimize the distance between the two logits to encourage consistency (See Fig. 1). The main difference between SSL and OSSL is that unlabeled outliers do not have any neighboring labeled samples, which makes it risky to perform hard-labeling such as pseudo-labeling. The outlier detector outputs a distance from inliers given an input, and enhancing the smoothness of this function allows us to improve its ability to find outliers. Empirically, this objective provides significant improvements in detecting outliers. Finally, to correctly classify inliers, we propose to apply FixMatch [35] to unlabeled samples that are considered to be inliers by the outlier detector.
The resulting framework shows consistent gains over baselines on various datasets and settings of OSSL. For example, OpenMatch achieves a 10.4% error rate on CIFAR10 with 300 labeled examples compared to the previous state-of-the-art of 20.3%. Surprisingly, OpenMatch demonstrates good performance in detecting outliers unseen in unlabeled training data. For instance, in the experiments on CIFAR10 with 100 labels per class, OpenMatch achieves a 3.4% higher AUROC in detecting outliers than a supervised model trained with all training samples. To summarize, our contributions are as follows:
• A soft open-set consistency regularization (SOCR), to improve outlier detection in OSSL.
• A new framework, OpenMatch, which combines a OVA-classifier, SOCR, and FixMatch.
• A new state-of-the-art in both correctly classifying inliers and detecting outliers, even when the outliers are unseen in unlabeled training data.

heading: 2 Related Work

text:
Semi-supervised Learning. Pseudo-labeling (PL) is a strong baseline in semi-supervised learning [23], and methods like FixMatch [35] or UDA [41] combining data augmentation and PL show the highest performance on many benchmark datasets. However, even with these strong methods,
performance drops when a model is exposed to noisy unlabeled data that includes novel categories. On the other hand, methods based on soft consistency regularization enforce smoothness of the decision boundary with respect to stochastic transformations [34] or models [22, 37]. By "soft", we mean that no sharpening or pseudo-labeling is applied to logits to propagate training signals through soft targets. The soft consistency regularization can be understood as approximated Jacobian regularization method [12, 31]. We propose to apply soft consistency regularization to our outlier detector, which we call soft open-set consistency regularization. Training a model with self-supervised learning and SSL achieves label-efficient training [5, 46]. Self-supervised learning and SSL seem to be complementary to each other to improve the recognition accuracy. But, unlabeled data with outliers will hurt the performance of these approaches since they rely on existing SSL objectives. Since our work focuses on the aspect of SSL, we hope it is complementary to self-supervised learning.
Open-set SSL methods include MTC [44], D3SL [14], and UASD [6]. MTC updates the network parameters and the anomaly score of unlabeled data alternately. In addition, it minimizes SSL loss (MixMatch [2] for a part of unlabeled data that is considered to be inliers. D3SL selectively employs unlabeled data to optimize SSL loss as well as optimizing a function that selects unlabeled data. UASD generates soft targets of the classifier for inliers (closed-set classifier) by averaging over many temporally ensembled networks’ predictions. By contrast, OpenMatch does not have to store the temporal ensembles and applies soft consistency loss to the outlier detector. Our empirical results show that the consistency loss for the outlier detector outperforms that for the closed-set classifier by a lot (See Sec. 4.4).
Open-set Domain Adaptation (ODA). OSSL is similar to open-set domain adaptation [3, 33] in that the unlabeled data contains novel categories. A key difference is that the unlabeled and labeled data follow different data distributions. Another key difference is that we aim to train a model from scratch while the domain adaptation task assumes access to models pre-trained on ImageNet [9]. Since the ImageNet contains 1,000 diverse categories, the pre-trained models have discriminative representations useful to detect outliers. In this point, our task is more challenging than ODA. To detect outliers in an unsupervised way, we employ a OVA-classifier model [32] proposed for this task.
Novelty Detection. Novelty detection or open-set classification aims to identify outliers that are completely unseen during training [17, 24, 10, 11]. Self-supervised learning such as rotation prediction and contrastive learning is shown to be useful to separate outliers from inliers [19, 36, 40]. The task assumes plenty of labeled inliers in training data. Padhy et al. employ OVA-classifiers for this task [30]. Hendrycks et al. reveal that exposing a model to outlier data allows it to effectively detect anomalies [18] and train it on out-of-distribution training data in a supervised way. But, collecting such useful out-of-distribution data is not always possible. In OSSL, a model cannot be trained on many labeled samples and needs to detect which samples are inlier or outliers in unlabeled data. In this sense, OSSL is more challenging and more realistic.

heading: 3 OpenMatch

text:
Problem Setting. Our task is to learn a classifier using open-set semi-supervised learning. For a K-way classification problem, let X = {(xb, yb) : b ∈ (1, ...B)} be a batch of B labeled examples xb randomly sampled from the labeled training set, S, where yb ∈ (1, ...K) is the corresponding label. Let U = {(ub) : b ∈ (1, ...µB)} be a batch of µB data randomly sampled from the unlabeled training set, Su, where µ is a hyper-parameter that determines the relative sizes of X and U . Unlike in SSL, which assumes all unlabeled data come from the K known classes, in our OSSL setting they may come from unknown outlier classes. The goal of OSSL is to classify inliers into the correct classes while learning to detect the outliers.
Approach Overview. Our model has three components: (1) a shared feature extractor F (·), (2) an outlier detector consisting of K one-vs-all (OVA) sub-classifiers Dj(·), j ∈ {1, ...,K}, and (3) a closed-set classifier C(·) which outputs a probability vector ∈ RK for K-class classification. At test time, the closed-set classifier is first applied to predict the K-way label, ŷ. If Dŷ predicts an inlier, then the output class is ŷ, otherwise, the output is "outlier". The main technical novelty comes from the choice of OVA-classifiers for outlier detection as well as training them with soft open-set consistency regularization. We first describe the training of the OVA-classifiers [32] before describing the remaining details of our framework.

heading: 3.1 One-vs-All Outlier Detector

text:
For open-set [33] or universal domain adaptation [42], Saito et al. [32] propose to train OVA-classifiers to detect outlier samples of an unlabeled target domain. They aim to learn a boundary between inliers and outliers for each class. Each sub-classifier is trained to distinguish if the sample is an inlier for the corresponding class. For example, the sub-classifier for class j, Dj , outputs a 2-dimensional vector zjb = D
j(F (xb)) ∈ R2, where each dimension indicates the score of a sample being an inlier and outlier respectively. We denote pj(t = 0|xb) and pj(t = 1|xb) as the probability of xb being the inlier and outlier for the class j, computed by Softmax(zjb) (Note that p
j(t = 0|xb) + pj(t = 1|xb) = 1). Then, the sub-classifier Dj is trained such that the data from class j is treated as positives, and the data from all the other classes is treated as negatives. To achieve this, we minimize the following loss for the outlier detector for a given batch X := {xb, yb}Bb=1:
Lova(X ) := 1
B B∑ b=1 − log(pyb(t = 0|xb))−min i 6=yb log(pi(t = 1|xb)). (1)
Here we use the hard-negative sub-classifier sampling technique proposed by [32] to effectively learn the threshold. We refer the reader to [32] for more details. The key is that each sub-classifier outputs a distance representing how far the input is from the corresponding class. Therefore, the classifiers are effective to identify unlabeled outliers. The output of Dj is used to detect outliers as follows. If the probability of a sample being an outlier for the class highest-scored by C() is larger than that of a sample being an inlier, we consider the input as an outlier. To be specific, we identify the unlabeled sample ub as an outlier if pŷ(t = 0|ub) < 0.5, where ŷ = argmaxj C(F (ub)). This detection is employed in selecting pseudo-inliers from unlabeled data as explained in Sec. 3.3.
Open-set Entropy Minimization. For unlabeled samples, [32] apply entropy minimization with respect to the OVA-classifiers, denoted by Lem(U). The separation between inliers and outliers is enhanced through the minimization of this loss for a given batch U := {ub}µBb=1:
Lem(U) := − 1
µB µB∑ b=1 K∑ j=1 pj(t = 0|ub) log pj(t = 0|ub) + pj(t = 1|ub) log pj(t = 1|ub). (2)
The difference between our setup and [32] is that they employ a model pre-trained on ImageNet [9], which extracts highly discriminative features, thus, inliers and outliers are well-separated even before fine-tuning. However, we aim to train a model to learn representations separating them from scratch. Thus the inliers and outliers are confused before training, and we need an objective to ensure the separation in addition to Eq. 2.

heading: 3.2 Soft Open-set Consistency Regularization (SOCR)

text:
To motivate our approach, we describe the difference between SSL and OSSL in Fig. 2. In SSL, the hard consistency loss such as pseudo-labeling can be effective to propagate label information from
Algorithm 1 OpenMatch Algorithm. 1: Input: Set of labeled data S = ( (xb, yb); b ∈ (1, . . . , N) ) , set of unlabeled data Su = ( ub; b ∈
(1, . . . , Nu) ) , and set of pseudo-inlier data K = ∅. Data augmentation function T . Model parameters w, learning rate η, epoch Efix and Emax, iteration Imax, trade-off parameters, λem, λoc, λfm; for Epoch = 1 to Emax do
for Iteration = 1 to Imax do 2: Sample a batch of labeled data X ∈ S and unlabeled data U ∈ Su; 3: Compute Lall = Lsup(X ) + λemLem(U) + λocLoc(U , T ); // Eq.1, 2 and 3 if Epoch > Efix then
4: Sample a batch of pseudo-inliers I ∈ K; // Sample pseudo-inliers. 5: Compute Lall += λfmLfm(I) ; // FixMatch for pseudo-inliers. end 6: Update w = w − η∇wLall; // Update weights
end if Epoch ≥ Efix then
7: Update K = Select(w,Du); // Detect outliers and select pseudo-inliers. end
end 8: Output: Model parameters w.
labeled to unlabeled samples. Through the hard consistency loss, unlabeled samples get training signals from neighboring labeled samples (Fig. 2(a)). By contrast, in OSSL, since outliers are not assigned any labels, labeled samples are far from outliers, which makes the hard labeling unreliable, and unsupervised training signals can be incorrect (Fig. 2(b)). The open-set entropy minimization above can also be considered a variant of the hard consistency since it is minimized if the predicted output class distribution is one-hot. To propagate useful training signals to unlabeled samples, we propose to smooth the decision boundary of the outlier detector by minimizing the distance between its predictions on two augmentations of the same image (Fig. 2(c)). We use unsharpened logits, keeping the training signal soft to avoid introducing incorrect pseudo-labels. We call this loss soft open-set consistency regularization (SOCR). Given the smoothed outlier detector, the hope is that the training signals of the open-set entropy minimization will also become more correct.
Specifically, SOCR enhances the smoothness of the outlier detector over data augmentation T , which in our experiments is just standard random cropping. Note that the standard cropping is also used in computing Eq. 1 and 2, but we omit it for the simplicity of notation. We obtain two different views of ub, T1(ub) and T2(ub), where T1 and T2 are data augmentation functions stochastically sampled from T . Let pj(t|T1(ub)) be the output logits of T1(ub) from the jth OVA-classifier. We encourage the consistency of the output logits over T to enhance the smoothness by minimizing the following loss for a given batch U := {ub}µBb=1:
Loc(U , T ) := 1
µB µB∑ b=1 K∑ j=1 ∑ t∈(0,1) |pj(t|T1(ub))− pj(t|T2(ub))|2. (3)
This minimizes the distance between two probability outputs to ensure smoothness with respect to T . Note that the key part of the regularization is not applying any sharpening to the output logits, keeping them soft. The major difference from the previous soft consistency regularization [12, 37, 22] is that we use the outlier detector to compute the regularization loss.

heading: 3.3 Overall Framework

text:
The entire OpenMatch algorithm is described in Alg. 1. The unsupervised losses (Eqs. 2, 3) for the outlier detector are effective to detect outliers in unlabeled data. However, these losses are not sufficient to correctly classify unlabeled inliers. Therefore, we propose to introduce a semi-supervised learning loss for unlabeled samples classified as inliers. We adopt FixMatch [35] since it is a simple, yet very strong SSL method. FixMatch first generates pseudo-labels using the model’s predictions on weakly augmented unlabeled images. The model is then trained to predict these pseudo-labels when fed a strongly augmented version of the same image. "Weak" denotes augmentations such as a simple
random cropping while "strong" refers to extensive data augmentation such as RandAugment [8] and CTAugment [1].
For labeled samples, we also compute the standard cross-entropy loss to train C(·), Lcls(X ), for the closed-set output, C(F (xb)). We summarize Lsup(X ) as the sum of Lova(X ) and Lcls(X ). After training a model for Efix epochs with Lsup, Lem, Loc, we begin to select pseudo-inliers from unlabeled data at every following epoch. Specifically, in Alg. 1, Select(w,Du) denotes the process of classifying unlabeled samples given model parameters w of models (F (·), D(·), and C(·)), and unlabeled data. The outlier-detection process is summarized in Sec. 3.1.
Then, the FixMatch loss, Lfm(I), is computed only for the pseudo-inliers to optimize the model, where I denotes a batch of pseudo-inliers. The following is the resulting overall objective of OpenMatch:
Lall(X ,U , T , I) := Lsup(X ) + λemLem(X ) + λocLoc(U , T ) + λfmLfm(I), (4) where λem, λoc and λfm control the trade-off for each objective.

heading: 4 Experiments

text:
Setup. We evaluate the efficacy of OpenMatch on several SSL image classification benchmarks. Specifically, we perform experiments with varying amounts of labeled data and varying numbers of known/unknown classes on CIFAR10/100 [21] and ImageNet [9]. We attempt to cover various scenarios by using these three datasets.
Note that we use an identical set of hyper-parameters except for λoc, which is tuned on each dataset. λem is set 0.1 in all experiments. λfm is set to 0 before Efix epochs and then set to 1 for all experiments. Efix is set to 10 in all experiments. The hyper-parameters for FixMatch, e.g., data augmentation, confidence threshold, are fixed across all experiments for simplicity. The hyperparameters are set by tuning on a validation set that contains a small number of labeled samples. Note that the validation set does not contain any outliers. A complete list of hyper-parameters is reported in the appendix. Each experiment is done with a single 12-GB GPU, such as an NVIDIA TitanX.
Baselines. For the OSSL baseline, we employ MTC [44] using the author’s implementation. In addition, we show the result of a model trained only with labeled samples (Labeled Only). We exclude UASD [6] and D3SL [14] from our baselines since their reported results underperform a model we train only with labeled samples. For instance, in the experiments for CIFAR10 at 400 labels per class, their reported error rates are worse than 20% while the rate of the Labeled Only baseline is 20.0%. A comparison with FixMatch [35] reveals the effect of OVA-classifiers trained with the soft consistency loss. The hyper-parameters of both OpenMatch and baselines are tuned by maximizing the accuracy on the validation set. Since the validation set does not have any outliers, we choose the validation accuracy as a criterion.
Evaluation. We assume the test set contains both known (inlier) and unknown (outlier) classes. For known classes, classification accuracy is used to evaluate performance. To evaluate the separation into inliers and outliers, we use AUROC following the standard evaluation protocol of novelty detection [17]. To compute the anomaly score, the outlier detector’s score is employed for OpenMatch, Labeled Only, and FixMatch, where we add our outlier detector to the latter two models. We report the results averaged over three runs and their standard deviations.

heading: 4.1 CIFAR10 and CIFAR100

text:
We compare OpenMatch with several baselines on the standard benchmark datasets of SSL, CIFAR10, and CIFAR100. Following [44], we use a randomly initialized Wide ResNet-28-2 [45] with 1.5M parameters in these experiments. For CIFAR10, we split the classes into known and unknown classes by defining animal classes as known and others as unknown, which results in 6 known classes and 4 unknown classes. For CIFAR100, we first split the classes by their super-classes provided by the dataset so that known and unknown classes do not share super-classes. Then, we split the super-classes into known and unknown. To evaluate performance with different numbers of outliers, we run experiments on two settings: 80 known classes (20 unknown classes) and 55 known classes (45 unknown classes).
Tables 1 and 2 describe the error rate on inliers and AUROC values respectively. Note that the number of labeled samples per class is shown in each column. With respect to the error rate, OpenMatch achieves state-of-the-art performance in all cases, and the gain over the baselines is remarkable in
many cases. In particular, we see about a 10% improvement in CIFAR10 error at 50 labels. In addition, the improvements in AUROC are also clear. MTC [44] does not improve the performance very much compared to the labeled-only model on CIFAR100 because their method of identifying outliers, i.e., Otsu’s thresholding [25], is not very robust to the number of outliers as discussed in their paper. On the other hand, OpenMatch improves the error rate by more than 3% and AUROC by more than 6% in all cases. Simply applying FixMatch [35] also does not always improve the error compared to the labeled-only model. This is because FixMatch confuses inliers and outliers and assigns pseudo-labels to outliers, as the significant decrease in AUROC indicates.

heading: 4.2 ImageNet

text: We evaluate OpenMatch on ImageNet to see its behavior on more complex and challenging datasets. Since training on the entire ImageNet is computationally expensive, we choose ImageNet-30 [17], which is a subset of ImageNet containing 30 classes. The subset is useful for evaluation on unseen outliers as done in Sec. 4.5, because the classes do not include several super-classes such as a bird class. We pick the first 20 classes, in alphabetical order, as known classes, and the remaining 10 classes as unknown. We utilize the ResNet-18 [16] network architecture. Hyper-parameters such as a batch-size, learning rate, data augmentation are the same as ones used in CIFAR10 and CIFAR100. As shown in the rightmost columns of Tables 1 and 2, OpenMatch achieves the best performance in both accuracy and AUROC. Unlike the results on CIFAR, FixMatch improves the performance over the Labeled Only model. This is probably because outliers are easier to detect, as the AUROC score of the Labeled Only model already shows a higher score than other scenarios. Therefore, even without any outlier detection objectives, FixMatch can correctly select pseudo-inliers with confidence thresholding.

heading: 4.3 Ablation Study on Soft Open-set Consistency Regularization

text:
In this section, we validate our claim that the SOCR regularization is effective at separating inliers and outliers. We perform several ablation studies to see the benefit of SOCR and measure the model’s ability to detect outliers using AUROC. In summary, both numerical and qualitative evaluations confirm the effectiveness of SOCR.
Table 3 describes the numerical comparison of ablated models. In this evaluation, we do not apply FixMatch to pseudo-inliers to measure the pure gain from the consistency loss. Introducing SOCR increases AUROC by more than 20% in CIFAR10, 8% in CIFAR100, and 8% in ImageNet. These results support the effectiveness of smoothing the outlier detector’s output.
Histograms of Anomaly Scores. Fig. 3 illustrates the anomaly score of inlier and outlier test samples in the ablation experiment on CIFAR10. If SOCR is not employed (Fig. 3(a)), inliers and outliers are confused while SOCR makes the scores of outliers larger and enhances the separation (Fig. 3(b)). Further applying FixMatch to pseudo-inliers decreases the score of inliers (Fig. 3(c)), which separates inliers and outliers better. Considering the results of Table 2, just applying FixMatch to all unlabeled samples is often not useful to detect outliers, but applying it to the pseudo-inliers can improve the outlier detection.

heading: 4.4 Analysis

text:
Feature Visualization. Fig. 4 visualizes the feature distributions with T-SNE [26] in the experiment on CIFAR10 at 100 labels. OpenMatch (c) separates inliers and outliers well while a model trained only with 100 labeled samples (a) confuses them. The separation of OpenMatch appears to be better than a model trained with all labeled inliers (b).
Closed-set Consistency vs Open-set Consistency. The soft consistency loss is computed for the outlier detector in OpenMatch. An alternative is to compute it for the closed-set classifier as done in UASD [6]. We replace SOCR with the soft consistency loss with respect to the closed-set classifier and perform the same analysis as in Sec. 4.3. In CIFAR10 at 50 labeled samples, this model achieves 70.6 ± 1.3 % AUROC, which is 10.1 % better than a model without consistency loss and 10.7 % worse than a model with SOCR. In CIFAR10 at 400 labeled samples, this model achieves 88.5± 2.4 % AUROC, which is 12.7 % better than a model without consistency loss and 8.3 % worse than a model with SOCR. These results indicate that even using soft consistency for the closed-set classifier helps to detect outliers, as shown in UASD [6], but SOCR helps much more. Thus, our choice of combining both the outlier detector and soft consistency loss is validated in this analysis.
Comparison with OOD with self-supervised learning. We compare our method with outlier detection methods using self-supervised learning. Several works [19, 36] have shown that training a model with self-supervised learning objectives such as contrastive loss or rotation prediction loss improves OOD performance. We provide a brief analysis of these baselines in open-set SSL.
Does self-supervised pre-training help open-set SSL? A model trained with self-supervised learning objectives shows great performance in many downstream image recognition tasks. We investigate whether initializing a model trained with self-supervised learning (SimCLR [4]) improves perfor-
mance in open-set SSL (Table 4). We pre-train a model with SimCLR loss using all unlabeled data. The initialization does not make a clear difference from random initialization, i.e., default initialization.
Does self-supervised training help open-set SSL? Next, we investigate whether using selfsupervised learning loss in addition to the classification loss on labeled data helps open-set SSL. In Table 5, we show the results of adding self-supervised loss for labeled data or unlabeled data. SimCLR does not help in any case compared to a model trained only with classification loss for labeled data. Constrative learning basically attempts to uniformly distribute all samples in a unit sphere, which does not necessarily help in open-set SSL. The goal of open-set SSL is to keep outliers far from inliers whereas clustering inliers within each class. By contrast, rotation prediction improves performance if applied to labeled samples in CIFAR-10. If applied to unlabeled data, a model can assign the same rotation labels to both inliers and outliers, which can prevent from separating them.

heading: 4.5 Novelty Detection

text:
We have seen that OpenMatch can learn to detect outliers from unlabeled data. In this experiment, we evaluate how well our OSSL model can separate inliers from outliers unseen in the unlabeled data. This setting is similar to general outlier or out-of-distribution sample detection. Note that the models evaluated in this section are the same as the ones in Tables 1 and 2. We consider the following datasets as out-of-distribution: SVHN [28], resized LSUN [43], ImageNet, and CIFAR100 for CIFAR10 experiments (Table 6(a)), and LSUN, CUB-200 [39], Dogs [20], Caltech [13], Flowers [29], and DTD [7] for ImageNet-30 (Table 6(b)). See supplemental material for the details. We utilize AUROC to measure how well inliers and outliers are separated. To see the gap from a supervised model, we train a model using all labeled samples of inlier classes and show the comparison.
As seen in Table 6, OpenMatch outperforms OSSL baselines by 5.8% on average and even the supervised model by 3.4% on average on CIFAR10. On ImageNet-30, OpenMatch outperforms OSSL baselines by more than 4.1% on average. This result verifies that OpenMatch is robust to various out-of-distribution data by virtue of being exposed to unlabeled data containing outliers.

heading: 5 Conclusion

text:
In this paper, we introduce a method for open-set semi-supervised learning (OSSL), where samples of novel categories are present in unlabeled data. To approach the task, we propose a novel framework, OpenMatch, which unifies one-vs-all classifiers and FixMatch. Our proposed objective, open-set soft consistency loss, is shown to be effective to detect outliers from unlabeled data, which allows FixMatch to work well in the OSSL setting. In addition, OpenMatch sometimes detects outliers unseen in unlabeled data better than a supervised model. We believe that our framework for OSSL will make label-efficient techniques more practical.
Limitations. OpenMatch can have difficulty in detecting outliers similar to inliers, though any methods for outlier detection can have the same limitation. If the outliers have similar visual characteristics, separating them from inliers samples is difficult. A potential solution is to introduce self-supervised learning [15, 4] for unlabeled data.
Broader Impact. OpenMatch is an effective tool for semi-supervised learning with noisy unlabeled data containing outliers. It would benefit projects with small budgets or hard-to-label data. At the same time, it is possible for malicious actors to exploit the advantages of learning from limited data. Broadly speaking, any progress on semi-supervised learning can have these consequences.

heading: 6 Acknowledgement

text: This work was supported by Honda, DARPA LwLL and NSF Award No. 1535797.

title: OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers


summaries: This paper proposes a method to learn goal-conditioned policies by combining ideas from goal-conditioned RL and model-based exploration. It operates by training an exploration policy within the model to maximize model disagreement (as proposed in previous works), as well as a goal-conditioned policy to reach states in the replay buffer. These are then both deployed to grow the replay buffer, and the process is repeated. The approach is evaluated on a newly-defined set of benchmark tasks based on the Deepmind Control suite, as well as simulated robotic manipulation. They report better performance compared to previous methods such as Skew-Fit, DIAYN and two others. They also investigate two different choices of distance functions for the goal-conditioned RL component, and show that the optimal one depends on the setting.

summaries: This paper proposes a model based RL setting where an agent learns a task agnostic model, learns two policies to explore and reach input goal states (specified as images) respectively and is able to operate in a goal conditional mode during evaluation time.   The key thesis is to use the world model to incentivize the agent to explore the environment by maximizing expected information gain and reach input goal states by learning to reach states from its past experiences.   There have been other model free and model based approaches studying the same problem but the claim is that its the first to combine model based exploration and reachability on new benchmark environments.

summaries: This paper proposes to jointly train two agents in a differentiable world model: an *explorer* that is rewarded to explore uncertain states and an *achiever* whose task is to achieve a given goal state. After training, the *achiever* can be deployed as a goal-reaching policy without further tuning. Authors also spent time investigating how to design the loss function for the *achiever*. In the experiments, three new benchmarks are introduced and the proposed method is shown to outperform baselines.

summaries: Summary: This paper proposes a model-based method with an ensemble disagreement-based exploration bonus for learning universal goal-conditioned policies. The world model and policy learning methods are adapted from DREAMER. In addition to that, the paper first introduces an explorer to first explore the state with large uncertainties, and then train the achiever to reach the proposed goals. The use of the world model enables data-efficient learning and exploration. The paper makes solid experiments and evaluates their approach on various domains. The paper also discusses two different distance metrics for learning goal-conditioned policy on images.

summaries: This paper introduces a framework to train policies for visual goal oriented tasks. The framework comprises two policies, the first, an explorer, which is trained to seek out novel states (where novelty is defined as the level of agreement between an ensemble of future state predictors - an approximate measure of epistemic uncertainty), while the second, an achiever, is trained to reach these states. A model based approach to train the achiever, where the achiever is trained using trajectories generated by a learned dynamics model. The paper also introduces a new benchmark for testing goal oriented polices of this form, comprising a kitchen manipulation task, a block handling task, and a yoga task.

abstractText: How can artificial agents learn to solve many diverse tasks in complex visual environments without any supervision? We decompose this question into two challenges: discovering new goals and learning to reliably achieve them. Our proposed agent, Latent Explorer Achiever (LEXA), addresses both challenges by learning a world model from image inputs and using it to train an explorer and an achiever policy via imagined rollouts. Unlike prior methods that explore by reaching previously visited states, the explorer plans to discover unseen surprising states through foresight, which are then used as diverse targets for the achiever to practice. After the unsupervised phase, LEXA solves tasks specified as goal images zero-shot without any additional learning. LEXA substantially outperforms previous approaches to unsupervised goal reaching, both on prior benchmarks and on a new challenging benchmark with 40 test tasks spanning across four robotic manipulation and locomotion domains. LEXA further achieves goals that require interacting with multiple objects in sequence.

authors:
J. Achiam
H. Edwards
D. Amodei
P. Abbeel

title: Variational option discovery algorithms

venue: arXiv preprint arXiv:1807.10299

authors:
M. Andrychowicz
F. Wolski
A. Ray
J. Schneider
R. Fong
P. Welinder
B. McGrew
J. Tobin
P. Abbeel
W. Zaremba

title: Hindsight experience replay

venue: arXiv preprint arXiv:1707.01495

authors:
P. Ball
J. Parker-Holder
A. Pacchiano
K. Choromanski
S. Roberts

title: Ready policy one: World building through active learning

venue: International Conference on Machine Learning, pages 591–601. PMLR

authors:
M. Bellemare
S. Srinivasan
G. Ostrovski
T. Schaul
D. Saxton
R. Munos

title: Unifying count-based exploration and intrinsic motivation

venue: Advances in Neural Information Processing Systems, pages 1471–1479

authors:
L. Beyer
D. Vincent
O. Teboul
S. Gelly
M. Geist
O. Pietquin

title: Mulex: Disentangling exploitation from exploration in deep rl

venue: arXiv preprint arXiv:1907.00868

authors:
B. Bucher
K. Schmeckpeper
N. Matni
K. Daniilidis

title: Adversarial curiosity

venue: arXiv preprint arXiv:2003.06082

authors:
L. Buesing
T. Weber
S. Racaniere
S. Eslami
D. Rezende
D.P. Reichert
F. Viola
F. Besse
K. Gregor
D. Hassabis

title: et al

venue: Learning and querying fast generative models for reinforcement learning. arXiv preprint arXiv:1802.03006

authors:
Y. Burda
H. Edwards
A. Storkey
O. Klimov

title: Exploration by random network distillation

venue: arXiv preprint arXiv:1810.12894

authors:
V. Campos
A. Trott
C. Xiong
R. Socher
X. Giró-i Nieto
J. Torres

title: Explore

venue: discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pages 1317–1327. PMLR

authors:
Y. Chebotar
K. Hausman
Y. Lu
T. Xiao
D. Kalashnikov
J. Varley
A. Irpan
B. Eysenbach
R. Julian
C. Finn

title: et al

venue: Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.07749

authors:
K. Cho
B. Van Merriënboer
C. Gulcehre
D. Bahdanau
F. Bougares
H. Schwenk
Y. Bengio

title: Learning phrase representations using rnn encoder-decoder for statistical machine translation

venue: arXiv preprint arXiv:1406.1078

authors:
J. Choi
A. Sharma
S. Levine
H. Lee
S.S. Gu

title: Variational empowerment as representation learning for goal-based reinforcement learning

venue: Deep Reinforcement Learning workshop at the Conference on Neural Information Processing Systems (DRL)

authors:
F. Ebert
C. Finn
S. Dasari
A. Xie
A. Lee
S. Levine

title: Visual foresight: Model-based deep reinforcement learning for vision-based robotic control

venue: arXiv preprint arXiv:1812.00568

authors:
A. Ecoffet
J. Huizinga
J. Lehman
K.O. Stanley
J. Clune

title: Go-explore: a new approach for hard-exploration problems

venue: arXiv preprint arXiv:1901.10995

authors:
B. Eysenbach
A. Gupta
J. Ibarz
S. Levine

title: Diversity is all you need: learning skills without a reward function

venue: arXiv preprint arXiv:1802.06070

authors:
B. Eysenbach
R. Salakhutdinov
S. Levine

title: Search on the replay buffer: Bridging planning and reinforcement learning

venue: arXiv preprint arXiv:1906.05253

authors:
C. Finn
S. Levine

title: Deep visual foresight for planning robot motion

venue: Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 2786–2793. IEEE

authors:
C. Florensa
D. Held
X. Geng
P. Abbeel

title: Automatic goal generation for reinforcement learning agents

venue: International conference on machine learning, pages 1515–1528. PMLR

authors:
C. Florensa
J. Degrave
N. Heess
J.T. Springenberg
M. Riedmiller

title: Self-supervised learning of image embedding for continuous control

venue: arXiv preprint arXiv:1901.00943

authors:
D. Ghosh
A. Gupta
J. Fu
A. Reddy
C. Devin
B. Eysenbach
S. Levine

title: Learning to reach goals without reinforcement learning

venue: arXiv preprint arXiv:1912.06088

authors:
K. Gregor
D.J. Rezende
D. Wierstra

title: Variational intrinsic control

venue: arXiv preprint arXiv:1611.07507

authors:
A. Gupta
V. Kumar
C. Lynch
S. Levine
K. Hausman

title: Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning

venue: arXiv preprint arXiv:1910.11956

authors:
D. Hafner
T. Lillicrap
I. Fischer
R. Villegas
D. Ha
H. Lee
J. Davidson

title: Learning latent dynamics for planning from pixels

venue: arXiv preprint arXiv:1811.04551

authors:
D. Hafner
T. Lillicrap
J. Ba
M. Norouzi

title: Dream to control: Learning behaviors by latent imagination

venue: arXiv preprint arXiv:1912.01603

authors:
K. Hartikainen
X. Geng
T. Haarnoja
S. Levine

title: Dynamical distance learning for semisupervised and unsupervised skill discovery

venue: ICLR

title: Learning to achieve goals

venue: IJCAI, pages 1094–1099. Citeseer

authors:
D.P. Kingma
J. Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980

authors:
D.P. Kingma
M. Welling

title: Auto-encoding variational bayes

venue: arXiv preprint arXiv:1312.6114

authors:
B. Lakshminarayanan
A. Pritzel
C. Blundell

title: Simple and scalable predictive uncertainty estimation using deep ensembles

venue: arXiv preprint arXiv:1612.01474

title: On a measure of the information provided by an experiment

venue: The Annals of Mathematical Statistics,

authors:
V. Mnih
K. Kavukcuoglu
D. Silver
A.A. Rusu
J. Veness
M.G. Bellemare
A. Graves
M. Riedmiller
A.K. Fidjeland
G. Ostrovski

title: et al

venue: Human-level control through deep reinforcement learning. Nature, 518(7540):529

authors:
A.V. Nair
V. Pong
M. Dalal
S. Bahl
S. Lin
S. Levine

title: Visual reinforcement learning with imagined goals

venue: Advances in Neural Information Processing Systems, pages 9191–9200

authors:
O. OpenAI
M. Plappert
R. Sampedro
T. Xu
I. Akkaya
V. Kosaraju
P. Welinder

title: R

venue: D’Sa, A. Petron, H. P. d. O. Pinto, et al. Asymmetric self-play for automatic goal discovery in robotic manipulation. arXiv preprint arXiv:2101.04882

authors:
D. Pathak
P. Agrawal
A.A. Efros
T. Darrell

title: Curiosity-driven exploration by selfsupervised prediction

venue: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 16–17

authors:
D. Pathak
P. Mahmoudieh
G. Luo
P. Agrawal
D. Chen
Y. Shentu
E. Shelhamer
J. Malik
A.A. Efros
T. Darrell

title: Zero-shot visual imitation

venue: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 2050–2053

authors:
D. Pathak
D. Gandhi
A. Gupta

title: Self-supervised exploration via disagreement

venue: International Conference on Machine Learning, pages 5062–5071

authors:
V.H. Pong
M. Dalal
S. Lin
A. Nair
S. Bahl
S. Levine

title: Skew-fit: State-covering self-supervised reinforcement learning

venue: arXiv preprint arXiv:1903.03698

authors:
D.J. Rezende
S. Mohamed
D. Wierstra

title: Stochastic backpropagation and approximate inference in deep generative models

venue: arXiv preprint arXiv:1401.4082

authors:
V. Saxena
J. Ba
D. Hafner

title: Clockwork variational autoencoders

venue: arXiv preprint arXiv:2102.09532

authors:
T. Schaul
D. Horgan
K. Gregor
D. Silver

title: Universal value function approximators

venue: International conference on machine learning, pages 1312–1320. PMLR

title: Curious model-building control systems

venue: [Proceedings] 1991 IEEE International Joint Conference on Neural Networks, pages 1458–1463. IEEE

title: Finding new facts; thinking new thoughts

venue: Advances in child development and behavior, 43:269–294

authors:
R. Sekar
O. Rybkin
K. Daniilidis
P. Abbeel
D. Hafner
D. Pathak

title: Planning to explore via self-supervised world models

venue: ICML

authors:
P. Shyam
W. Jaśkowski
F. Gomez

title: Model-based active exploration

venue: arXiv preprint arXiv:1810.12162

authors:
S. Sukhbaatar
Z. Lin
I. Kostrikov
G. Synnaeve
A. Szlam
R. Fergus

title: Intrinsic motivation and automatic curricula via asymmetric self-play

authors:
Y. Sun
F. Gomez
J. Schmidhuber

title: Planning to be surprised: Optimal bayesian exploration in dynamic environments

venue: International Conference on Artificial General Intelligence, pages 41–51. Springer

authors:
R.S. Sutton
A.G. Barto

title: Reinforcement learning: An introduction

venue: MIT press

authors:
Y. Tassa
Y. Doron
A. Muldal
T. Erez
Y. Li

title: D

venue: d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690

authors:
S. Tian
S. Nair
F. Ebert
S. Dasari
B. Eysenbach
C. Finn
S. Levine

title: Model-based visual planning with self-supervised functional distances

venue: arXiv preprint arXiv:2012.15373

title: T

venue: Van de Wiele, T. Kulkarni, C. Ionescu, S. Hansen, and V. Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint arXiv:1811.11359

authors:
M. Watter
J.T. Springenberg
J. Boedecker
M. Riedmiller

title: Embed to control: A locally linear latent dynamics model for control from raw images

authors:
D. Yarats
R. Fergus
A. Lazaric
L. Pinto

title: Mastering visual continuous control: Improved data-augmented reinforcement learning

venue: arXiv preprint arXiv:2107.09645

authors:
T. Yu
D. Quillen
Z. He
R. Julian
K. Hausman
C. Finn
S. Levine

title: Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning

venue: Conference on Robot Learning, pages 1094–1100. PMLR

authors:
R. Zhang
P. Isola
A.A. Efros
E. Shechtman
O. Wang

title: The unreasonable effectiveness of deep features as a perceptual metric

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595

authors:
Y. Zhang
P. Abbeel
L. Pinto

title: Automatic curriculum learning through value disagreement

venue: Advances in Neural Information Processing Systems, 33

text:
1 Introduction
How can we build an agent that learns to solve hundreds of tasks in complex visual environments, such as rearranging objects with a robot arm or completing chores in a kitchen? While traditional reinforcement learning (RL) has been successful for individual tasks, it requires a substantial amount of human effort for every new task. Specifying task rewards requires domain knowledge, access to object positions, is timeconsuming, and prone to human errors. Moreover, traditional RL would require environment interaction to explore and practice in the environment for every new task. Instead, we approach learning hundreds of tasks through the paradigm of unsupervised goal-conditioned RL, where the agent learns many diverse skills in the environment in the complete absence of supervision, to later solve tasks via user-specified goal images immediately without further training [2, 26, 40]. Challenges Exploring the environment and learning to solve many different tasks is substantially more challenging than traditional RL with a dense reward function or learning from
∗ Equal contribution. Ordering determined at random. Project page: https://orybkin.github.io/lexa/
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
expert demonstrations. Existing methods are limited to simple tasks, such as picking or pushing a puck [13, 32, 37] or controlling simple 2D robots [50]. The key challenge in improving the performance of unsupervised RL is exploration. In particular, previous approaches explore by either revisiting previously seen rare goals [14, 18, 55] or sampling goals from a generative model [32, 37]. However, in both these approaches, the policy as well as the generative model are trained on previously visited states from the replay buffer, and hence the sampled goals are either within or near the frontier of agent’s experience. Ideally, we would like the agent to discover goals much beyond its frontier for efficient exploration, but how does an agent generate goals that it is yet to encounter? This is an open question not just for AI but for cognitive science too [42]. Approach To rectify this issue, we leverage a learned world model to train a separate explorer and achiever policy in imagination. Instead of randomly sampling or generating goals, our explorer policy discovers distant goals by first planning a sequence of actions optimized in imagination of the world model to find novel states with high expected information gain [30, 43, 44]. It then executes those imagined actions in the environment to discover interesting states without the need to generate them. Note these actions are likely to lead the agent to states which are several steps outside the frontier because otherwise the model wouldn’t have had high uncertainty or information gain. Finally, these discovered states are used as diverse targets for the achiever to practice. We train the achiever from on-policy imagination rollouts within the world model and without relying on experience relabeling, therefore leveraging foresight over hindsight. After this unsupervised training phase, the achiever solves tasks specified as goal images zero-shot without any additional learning at deployment. Unlike in the conventional RL paradigm [31, 47], our method is trained once and then used to achieve several tasks at test time without any supervision during training or testing. Contributions We introduce Latent Explorer Achiever (LEXA), an unsupervised goal reaching agent that trains an explorer and an achiever within a shared world model. At training, LEXA unlocks diverse data for goal reaching in environments where exploration is nontrivial. At test time, the achiever solves challenging locomotion and manipulation tasks provided as user-specified goal images. Our contributions are summarized as follows: • We propose to learn separate explorer and achiever policies as an approach to overcome the
exploration problem of unsupervised goal-conditioned RL. • We show that forward-looking exploration by planning with a learned world model substantially
outperforms previous strategies for goal exploration. • To evaluate on challenging tasks, we introduce a new goal reaching benchmark with a total of 40
diverse goal images across 4 different robot locomotion and manipulation environments. • LEXA outperforms prior methods, being the first to show success in the Kitchen robotic manipu-
lation environment, and achieves goal images where multiple objects need to be moved.

heading: 2 Latent Explorer Achiever (LEXA)

text: Our aim is to build an agent that can achieve arbitrary user-specified goals after learning in the environment without any supervision. This presents two challenges - collecting trajectories that contain diverse goals and learning to achieve these goals when specified as a goal image. We introduce a simple solution based on a world model and imagination training that addresses both challenges. The world model represents the agent’s current knowledge about the environment and is used for training two policies, the explorer and the achiever. To explore novel situations, we construct an estimate of which states the world model is still uncertain about. To achieve goals, we train the goal-conditioned achiever in imagination, using the images found so far as unsupervised goals. At test time, the achiever is deployed to reach user-specified goals. The training procedure is in Algorithm 1.

heading: 2.1 World Model

text:
To efficiently predict potential outcomes of future actions in environments with high-dimensional image inputs, we leverage a Recurrent State Space Model (RSSM) [23] that learns to predict forward using compact model states that facilitate planning [7, 51]. In contrast to predicting forward in image space, the model states enable efficient parallel planning with a large batch size and can reduce accumulating errors [39]. The world model consists of the following components:
Encoder: et = encφ(xt) Posterior: qφ(st | st−1, at−1, et) Dynamics: pφ(st | st−1, at−1) Image decoder: pφ(xt | st)
(1)
The model states st contain a deterministic component ht and a stochastic component zt with diagonalcovariance Gaussian distribution. ht is the recurrent state of a Gated Recurrent Unit (GRU) [11]. The encoder and decoder are convolutional neural networks (CNNs) and the remaining components are multi-layer perceptrons (MLPs). The world model is trained end-to-end by optimizing the evidence lower bound (ELBO) via stochastic backpropagation [28, 38] with the Adam optimizer [27].

heading: 2.2 Explorer

text:
To efficiently explore, we seek out surprising states imagined by the world model [6, 41, 43, 44, 46], as opposed to retrospectively exploring by revisiting previously novel states [4, 5, 8, 34]. As the world model can predict model states that correspond to unseen situations in the environment, the imagined trajectories contain more novel goals, compared to model-free exploration that is limited to the replay buffer. To collect informative novel trajectories in the environment, we train an exploration
Algorithm 1: Latent Explorer Achiever (LEXA) 1: initialize: World modelM, Replay buffer D, Explorer πe(at | zt), Achiever πg(at | zt, g) 2: while exploring do 3: TrainM on D 4: Train πe in imagination ofM to maximize exploration rewards ∑ t r e t .
5: Train πg in imagination ofM to maximize ∑ t r g t (zt, g) for images g ∼ D. 6: (Optional) Train d(zi, zj) to predict distances j − i on the imagination data from last step. 7: Deploy πe in the environment to explore and grow D. 8: Deploy πg in the environment to achieve a goal image g ∼ D to grow D. 9: end while
10: while evaluating do 11: given: Evaluation goal g 12: Deploy πg in the world to reach g. 13: end while
policy πe from the model states st in imagination of the world model to maximize an exploration reward:
Explorer: πe(at | st) Explorer Value: ve(st) (2) To explore the most informative model states, we estimate the epistemic uncertainty as a disagreement of an ensemble of transition functions. We train an ensemble of 1-step models to predict the next model state from the current model state. The ensemble model is trained alongside the world model on model states produced by the encoder qφ. Because the ensemble models are initialized at random, they will differ, especially for inputs that they have not been trained on [29, 36]:
Ensemble: f(st, θk) = ẑkt+1 for k = 1..K (3)
Leveraging the ensemble, we estimate the epistemic uncertainty as the ensemble disagreement. The exploration reward is the variance of the ensemble predictions averaged across dimension of the model state, which approximates the expected information gain [3, 43]:
ret (st) . =
1
N ∑ n Var{k} [ f(st, θk) ] n (4)
The explorer πe maximizes the sum of future exploration rewards ret using the Dreamer algorithm [24], which considers long-term rewards into the future by maximizing λ-returns under a learned value function. As a result, the explorer is trained to seek out situations are as informative as possible from imagined latent trajectories of the world model, and is periodically deployed in the environment to add novel trajectories to the replay buffer, so the world model and goal achiever policy can improve.

heading: 2.3 Achiever

text:
To leverage the knowledge obtained by exploration for learning to reach goals, we train a goal achiever policy πg that receives a model state and a goal as input. Our aim is to train a general policy that is capable of reaching many diverse goals. To achieve this in a data-efficient way, it is crucial that environment trajectories that were collected with one goal in mind are reused to also learn how to reach other goals. While prior work addressed this by goal relabeling which makes off-policy policy optimization a necessity [2], we instead leverage past trajectories via the world model trained on them that lets us generate an unlimited amount of new imagined trajectories for training the goal achiever on-policy in imagination. This simplifies policy optimization and can improve stability, while still sharing all collected experience across many goals.
Achiever: πg(at | st, eg) Achiever Value: vg(st, eg) (5) To train the goal achiever, we sample a goal image xg from the replay buffer and compute its embedding eg = encφ(xg). The achiever aims to maximize an unsupervised goal-reaching reward rg(st, eg). We discuss different choices for this reward in Section 2.4. We again use the Dreamer algorithm [24] for training, where now the value function also receives the goal embedding as input. In addition to imagination training, it can also be important to perform practice trials with the goal achiever in the true environment, so that any model inaccuracies along the goal reaching trajectories
may be corrected. To perform practice trials, we sample a goal from the replay buffer and execute the goal achiever policy for that goal in the environment. These trials are interleaved with exploration episodes collected by the exploration policy in equal proportion. We note that the goal achiever learning is entirely unsupervised because the practice goals are simply images the agent encountered through exploration or during previous practice trails.

heading: 2.4 Latent Distances

text:
Training the achiever policy requires us to define a goal achievement reward rg(st, eg) that measures how close the latent state st should be considered to the goal eg. One simple measure is the cosine distance in the latent space obtained by inputting image observations into the world-model. However, such a distance function brings visually similar states together even if they could be farther apart in temporal manner as measured by actions needed to reach from one to other. This bias makes this suitable only to scenarios where most of pixels in the observations are directly controllable, e.g., trying to arrange robot’s body in certain shape, such as RoboYoga poses in Figure 2. However, many environments contain agent as well as the world, such as manipulation involves interacting with objects that are not directly controllable. The cosine distance would try matching the entire goal image, and thus places a large weight on both matching the robot and object positions with the desired goal. Since the robot position is directly controllable it is much easier to match, but this metric overly focuses on it, yielding poor policies that ignore objects. We address this is by using the number of timesteps it takes to move from one image to another as a distance measure [25, 26]. This ignores large changes in robot position, since these can be completed in very few steps, and will instead focus more on the objects. This temporal cost function can be learned purely in imagination rollouts from our world model allowing as much data as needed without taking any steps in the real world.
Cosine Distance To use cosine distance with LEXA, for a latent state st, and a goal embedding eg , we use the latent inference network q to infer sg , and define the reward as the cosine similarity [54]:
rgt (st, eg) . = ∑ i stisgi, where st = st/‖st‖2, sg = sg/‖sg‖2, (6)
i.e. the cosine of the angle between the two vectors st, sg in the N−dimensional latent space. Temporal Distance To use temporal distances with LEXA, we train a neural network d to predict the number of time steps between two embeddings. We train it by sampling pairs of states st, st+k from an imagined rollout of the achiever and predicting the distance k. We implement the temporal distance in terms of predicted image embeddings êt+k in order to remove extra recurrent information:
Predicted embedding: emb(st) = êt ≈ et Temporal distance: dω(êt, êt+k) ≈ k/H, (7) where H is the maximum distance equal to the imagination horizon. Training distance function only on imagination data from the same trajectory would cause it to predict poor distance to far away states coming from other trajectories, such as images that are impossible to reach during one episode. In order to incorporate learning signal from such far-away goals, we include them by sampling images
from a different trajectory. We annotate these negative samples with the maximum possible distance, so that the agent always prefers images that were seen in the same trajectory.
rgt (st, eg) = −dω(êt, eg), where êt = emb(st), eg = encφ(xg) (8) The learned distance function depends on the training data policy. However, as the policy becomes more competent, the distance estimates will be closer to the optimal number of time steps to reach a particular goal, and the policy converges to the optimal solution [25]. LEXA always uses the latest data to train the distance function using imagination, ensuring that the convergence is fast.

heading: 3 Experiments

text: Our evaluation focuses on the following scientific questions: 1. Does LEXA outperform prior work on previous benchmarks and a new challenging benchmark? 2. How does forward-looking exploration of goals compare to previous goal exploration strategies? 3. How does the distance function affect the ability to reach goals in different types of environments? 4. Can we train one general LEXA to control different robots across visually distinct environments? 5. What components of LEXA are important for performance? We evaluate LEXA on prior benchmarks used by SkewFit [37], DISCERN [50], and Plan2Explore [43] in Section 3.3. Since these benchmarks are largely saturated, we also introduce a new challenging benchmark shown in Figure 2. We evaluate LEXA on this benchmark is Section 3.2.

heading: 3.1 Experimental setup

text:
As not many prior methods have shown success on reaching diverse goals from image inputs, we perform an apples-to-apples comparison by implementing the baselines using the same world model and policy optimization as our method: • SkewFit SkewFit [37] uses model-free hindsight experience replay and explores by sampling
goals from the latent space of a variational autoencoder [28, 38]. Being one of the state-of-the-art agents, we use the original implementation that does not use a world model or explorer policy.
• DDL Dynamic Distance Learning [25] trains a temporal distance function similar to our method. Following the original algorithm, DDL uses greedy exploration and trains the distance function on the replay buffer instead of in imagination.
• DIAYN Diversity is All You Need [15] learns a latent skill space and uses mutual information between skills and reached states as the objective. We augment DIAYN with our explorer policy and train a learned skill predictor to obtain a skill for a given test image [12].
• GCSL Goal-Conditioned Supervised Learning [20] trains the goal policy on replay buffer goals and mimics the actions that previously led to the goal. We also augment GCSL with our explorer policy, as we found no learning success without it.
Our new benchmark defines goal images for a diverse set of four existing environments as follows:
• RoboYoga We use the walker and quadruped domains of the DeepMind Control Suite [48] to define the RoboYoga benchmark, consisting of 12 goal images that correspond to different body poses for each of the two environments, such as lying down, standing up, and balancing.
• RoboBins Based on MetaWorld [53], we create a scene with a Sawyer robotic arm, two bins, and two blocks of different colors. The goal images specify tasks that include reaching, manipulating only one block, and manipulating both blocks.
• RoboKitchen The last benchmark involves the challenging kitchen environment from [22], where a franka robot can interact with various objects including a burner, light switch, sliding cabinet, hinge cabinet, microwave, or kettle. The goal images we include describe tasks that require interacting with only one object, as well as interacting with two objects.

heading: 3.2 Performance on New Benchmark

text:
We show the results on our main benchmark in Figure 6 and include heatmaps that show per-task success on each of the evaluation tasks from the benchmarks in the Appendix. Further, we report success averaged across tasks for each domain at the end of training in Table 1. We visualize example successful trajectory executions for tasks that require manipulating multiple objects in Fig. 4. RoboYoga The environments in this benchmark are directly controllable since they contain no other objects except the robot. We recall that for such settings we expect the cosine distance to be effective, as perceptual distance is quite accurate. Training is thus faster compared to using learned temporal distances, where the metric is learned from scratch. From Table 1 and Figure 6 we see that this is indeed the case for these environments (Walker and Quadruped), as LEXA with the cosine metric outperforms all prior approaches. Furthermore with temporal distances LEXA makes better progress compared to prior work on a much larger number of goals as can be seen from the per-task performance (Figures ??, ??), even though average success over goals looks similar to that of DDL. RoboBins This environment involves interaction with block objects, and thus is not directly controllable, and so we expect LEXA to perform better with the temporal distance metric. From Table 1 and
Figure 6, we see that LEXA gets higher average success than all prior approaches. Further from the per-task performance in 7, LEXA with the temporal distance metric is the only approach that makes progress on all goals in the benchmark. The main difference in performance between using temporal and cosine distance can be seen in the tasks involving two blocks, which are the most complex tasks in this environment (the last 3 columns of the per-task plot). The best performing prior method is DDL which solves reaching, and can perform simple pushing tasks. This method performs poorly due to poor exploration, as shown in Figure 5. We see that while other prior methods make some progress on reaching, they fail on harder tasks.
RoboKitchen This benchmark involves diverse objects that require different manipulation behavior. From Table 1 and Figure 6 and ?? we find that LEXA with temporal distance is able to learn multiple RoboKitchen tasks, some of which require sequentially completing 2 tasks in the environment. All prior methods barely make progress due to the challenging nature of this benchmark, and furthermore using the cosine distance function makes very limited progress. The gap in performance between using the two distance functions is much larger in this environment compared to RoboBins since there are many more objects and they are not as clearly visible as the blocks.
Single Agent Across All Environments In the previous sections we have shown that our approach can achieve diverse goals in different environments. However, we trained a new agent for every new environment, which doesn’t scale well to large numbers of environments. Thus we investigate if we can train a train a single agent across four environments in the benchmark. From Figure ?? we see that our approach with learned temporal distance is able to make progress on tasks from RoboKitchen, RoboBins Reaching, RoboBins Pick & Place and Walker, while the best prior method on the single-environment tasks (DDL) mainly solves walker tasks and reaching from RoboBin.

heading: 3.3 Performance on Prior benchmarks

text:
To further verify the results obtained on our benchmark, we evaluate LEXA on previously used benchmarks. We observe that LEXA significantly outperforms prior work on these benchmarks, and is often close to the optimal policy. Additional details are provided in ??????. SkewFit Benchmark SkewFit [37] introduces a robotic manipulation benchmark for unsupervised methods with simple tasks like planar pushing or picking. We evaluate on this benchmark in Table 2.
Baseline results are taken from [37]. LEXA significantly outperforms prior work on these tasks. Pushing and picking up blocks from images is largely solved and future work can focus on harder benchmarks such as those introduced in our paper.
DISCERN Benchmark We attempted to replicate the tasks described in [50] that are based on simple two-dimensional robots [48]. While the original tasks are not released, we followed the procedure for generating the goals described in the paper. Despite following the exact procedure, we were not able to obtain similar goals to the ones used in the original paper. Nevertheless, we show the goal completion percentage results obtained with our reproduced evaluation compared to DISCERN results from the original paper. LEXA results were obtained with early stopping. In Table 3 we see that our agent solves many of the tasks in this benchmark.
Plan2Explore Benchmark We provide a comparison on the standard reward-based DM control tasks [48] in Table 4. To compare on this benchmark, we create goal images that correspond to the reward functions. This setup is arguably harder for our agent, but is much more practical. Note our agent never observes the reward function and only observes the goal at test time. Plan2Explore adapts to new tasks but it needs the reward function to be known at test time, while DrQV2 is an oracle agent that observes the reward at training time. Baseline results are taken from [43, 52]. LEXA results were obtained with early stopping. LEXA outperforms Plan2Explore on most tasks and even performs comparably to state of the
art oracle agents (DrQ, DrQv2, Dreamer) that use true task rewards during training.
3.4 Analysis
Prior work Most work we compared against struggles with exploration, such as SkewFit and DLL methods. DIAYN is augmented with our explorer, but still fails to leverage the exploration data to learn a diverse set of skills. GCSL struggles to fit the exploration data and produces behavior that does not solve the task, perhaps because the exploration data is too diverse. We observed that all baselines make progress on the simple reaching, but struggle with other tasks. We have experimented with several versions and improvements to the baselines and report the best obtained performance. Ablation of different components We ablated components of LEXA on the RoboBins environment in Figure 8. Using a separate explorer policy crucial as without it the agent does not discover the more interesting tasks. Without negative sampling the agent learns slower, perhaps because the distance function doesn’t produce reasonable outputs when queried on images that are more than horizon length apart. Training the distance function with real data converges to slightly lower success than using imagination data, since real data is sampled in an off-policy manner due to its limited quantity.
Exploration performance Due to importance of exploration, we further examine the diversity of the data collected during training. We log the instances where the agent coincidentally solves an evaluation task during exploration, for the RoboKitchen and RoboBins environments. In Figure 5, we see that our method encounters harder tasks involving multiple objects much more often.

heading: 4 Related Work

text: Learning to Achieve Goals The problem of learning to reach many different goals has been commonly addressed with model-free methods that learn a single goal-conditioned policy [2, 26, 40]. Recent work has combined these approaches with various ways to generate training goals, such as asymmetric self-play [33, 45] or by sampling goals of intermediate difficulty [14, 18]. These approaches can achieve remarkable performance in simulated robotic domains, however, they focus on the settings where the agent can directly perceive the low-dimensional environment state. A few works have attempted to scale these model-free methods to visual goals by using contrastive [50] or reconstructive [32, 37] representation learning. However, these approaches struggle to perform meaningful exploration as no clear reward signal is available to guide the agent toward solving interesting tasks. Some works [10, 49] avoid this challenge by using a large dataset of interesting behaviors. Other works [37, 55] attempt to explore by generating goals similar to those that have already been seen, but do not try to explore truly novel states. A particularly relevant set of approaches used model-based methods to achieve goals via planning [13, 17] or learning model-regularized policies [35]. However, these approaches are limited by short planning horizons. In contrast, we learn long-horizon goal-conditioned value functions which allows us to solve more challenging tasks. More generally, most of the above approaches are limited by simplistic exploration, while our method leverages model imagination to search for novel states, which significantly improves exploration and in turn the downstream capabilities of the agent. Learning Distance Functions A crucial challenge for visual goal reaching is the choice of the reward or the cost function for the goal achieving policy. Several approaches use representation learning to create a distance in the feature space [9, 32, 50, 51]. However, this naive distance may not be most reflective of how hard a particular goal is to reach. One line of research has proposed using the mutual information between the current state and the goal as the distance metric [1, 12, 15, 21], however, it remains to be seen whether this approach can scale to more complex tasks. Other works proposed temporal distances that measure the amount of time it takes to reach the goal. One approach is to learn the distance with approximate dynamic programming using Q-learning methods [16, 19, 26]. Our distance function is most similar to Hartikainen et al. [25], who learn a temporal distance with supervised learning on recent policy experience. In contrast to [25], we always train the distance on-policy in imagination, and we further integrate this achiever policy into our latent explorer achiever framework to discover novel goals for the achiever to practice on.

heading: 5 Conclusion

text:
We presented Latent Explorer Achiever (LEXA), an agent for unsupervised RL that explores its environment, learns to achieve the discovered goals, and solves image-based tasks in a zero-shot way. By planning for novelty in imagination, LEXA prospectively explores to discover meaningful behaviors in substantially more diverse environments than considered by prior work. Further, LEXA is able to solve challenging downstream tasks specified as images without any supervision such as rewards or demonstrations. By proposing a challenging benchmark and the first agent to achieve meaningful performance on these tasks, we hope to stimulate future research on unsupervised agents, which we believe are fundamentally more scalable than traditional agents that require a human to design the tasks and rewards for learning. Many challenges remain for building unsupervised agents. Many tasks in our benchmark are still unsolved and there remains room for progress on the algorithmic side both for the world model and policy optimization. Further, it is important to demonstrate the benefits of unsupervised agents on real-world systems to verify their scalability. Finally, for widespread adoption, it is crucial to consider the problem of goal specification and design methods that act on goals that are easy to specify, such as via natural language. We believe LEXA will enable future work to tackle these goals effectively.
Acknowledgements We thank Ben Eysenbach, Stephen Tian, Sergey Levine, Dinesh Jayaraman, Karl Pertsch, Ed Hu and the members of GRASP lab and Pathak lab for insightful discussions. We also thank Murtaza Dalal and Chuning Zhu for help with MuJoCo environments. Finally, DP would like to thank Laura Schulz, Josh Tenenbaum and Alison Gopnik for seeding the idea of goal-setting in children, and Pulkit Agrawal for several crucial discussions (over gelato) since then. OR and KD were supported by ARL DCIST CRA W911NF-17-2-0181, ONR N00014-17-1-2093, and by Honda Research Institute. This work was partially supported by GoodAI Research Award and DARPA Machine Common Sense grant.

title: Discovering and Achieving Goals via World Models