summaries: This paper focuses on semi-supervised learning with outlier unlabeled data. This problem has attracted much attention in the semi-supervised learning community. The authors propose a method that contains two parts: First, detect outliers based on the OVA classifier. Second, propose an open-set soft-consistency regularization loss that can help improve the outlier detection performance. This manuscript introduces OpenMatch for open-set semi-supervised learning. The OpenMatch integrates a one-vs-all classifier (OVA-classifier, working as an outlier detector) and FixMatch. The main contribution should be the soft open-set consistency regularization (SOCR) in Figure 1 and the OpenMatch Framework in Alg 1. Specifically, the SOCR follows the same self-supervised framework in FixMatch but it uses OVA-classifier to output consistent anomaly score distribution. After using the model in Figure 1 to identify inliers from unlabeled data, the FixMatch is allowed to reach a higher accuracy on the OSSL tasks. This paper tackles open-set semi-supervised learning (OSSL) where outliers exist in unlabeled data. The key idea for detecting outliers is to learn class-specific, one-vs-all (OvA) classifiers—if all classifiers produce negative outputs for an unlabeled data, the data is considered an outlier. In addition to the classification loss computed on labeled data, the OvA classifiers are also trained on unlabeled data with an entropy minimization loss and a consistency regularization loss—the latter is computed on two views of the same data. The results on CIFAR and ImageNet show that the proposed approach largely outperforms the baselines in terms of both recognition accuracy and outlier detection accuracy. The authors propose an idea to improve model's performance under the open-set semi-supervised learning setting. This work proposed to leverage OOD detector, combined with FixMatch, and soft open-set consistency regularization (SOCR) to improve the model training during open-set semi-supervised learning. I think the usage of OOD on ignore outliers is not new and an unsupervised method on learning the consistency between augmentations for unlabeled data is also not new. Thus, I am not sure if I can find too much of novelty from this work itself, but I do appreciate the author's effort in putting them together to be a complete model, which in fact make solid progress on the open-set semi-supervised learning setting.  The paper proposes a model for open-set semi-supervised learning. As a semi-supervised learning model, it tackles classification where only a portion of the training data is labeled. "Open set" refers to the fact that the unlabeled data are noisy and can contain out of distribution examples, for which the class is not among the known classes. Out of distribution samples are present also during test, and the model has to recognize them and avoid classification on them. The proposed model builds on top of existing works, combining several loss terms and ideas such as one-versus-all classifiers and FixMatch as the main semi-supervised learning engine. The main technical contribution is a loss term enforcing the out-of-distribution score for two augmented versions of the same unlabeled example to be similar. Extensive experiments validate the proposal against recent state-of-the-art models.
abstractText: Semi-supervised learning (SSL) is an effective means to leverage unlabeled data to improve a model’s performance. Typical SSL methods like FixMatch assume that labeled and unlabeled data share the same label space. However, in practice, unlabeled data can contain categories unseen in the labeled set, i.e., outliers, which can significantly harm the performance of SSL algorithms. To address this problem, we propose a novel Open-set Semi-Supervised Learning (OSSL) approach called OpenMatch. Learning representations of inliers while rejecting outliers is essential for the success of OSSL. To this end, OpenMatch unifies FixMatch with novelty detection based on one-vs-all (OVA) classifiers. The OVA-classifier outputs the confidence score of a sample being an inlier, providing a threshold to detect outliers. Another key contribution is an open-set soft-consistency regularization loss, which enhances the smoothness of the OVA-classifier with respect to input transformations and greatly improves outlier detection. OpenMatch achieves stateof-the-art performance on three datasets, and even outperforms a fully supervised model in detecting outliers unseen in unlabeled data on CIFAR10. The code is available at https://github.com/VisionLearningGroup/OP_Match.
authors.name: Kuniaki Saito
authors.name: Donghyun Kim
authors.name: Kate Saenko
references.authors: David Berthelot Nicholas Carlini Ekin D Cubuk Alex Kurakin Kihyuk Sohn Han Zhang Colin Raffel
references.title: Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring
references.venue: arXiv preprint arXiv:1911.09785,
references.authors: David Berthelot Nicholas Carlini Ian Goodfellow Nicolas Papernot Avital Oliver Colin Raffel
references.title: Mixmatch: A holistic approach to semi-supervised learning
references.authors: Pau Panareda Busto Juergen Gall
references.title: Open set domain adaptation
references.venue: In Int. Conf. Comput. Vis.,
references.authors: Ting Chen Simon Kornblith Mohammad Norouzi Geoffrey Hinton
references.title: A simple framework for contrastive learning of visual representations
references.venue: In Int. Conf. Mach Learn.,
references.authors: Ting Chen Simon Kornblith Kevin Swersky Mohammad Norouzi Geoffrey Hinton
references.title: Big selfsupervised models are strong semi-supervised learners
references.venue: arXiv preprint arXiv:2006.10029,
references.authors: Yanbei Chen Xiatian Zhu Wei Li Shaogang Gong
references.title: Semi-supervised learning under class distribution mismatch
references.venue: In AAAI,
references.authors: Mircea Cimpoi Subhransu Maji Iasonas Kokkinos Sammy Mohamed Andrea Vedaldi
references.title: Describing textures in the wild
references.venue: In IEEE Conf. Comput. Vis. Pattern Recog.,
references.authors: Ekin D Cubuk Barret Zoph Jonathon Shlens Quoc V Le
references.title: Randaugment: Practical automated data augmentation with a reduced search space
references.venue: In IEEE Conf. Comput. Vis. Pattern Recog.,
references.authors: Jia Deng Wei Dong Richard Socher Li-Jia Li Kai Li Li Fei-Fei
references.title: Imagenet: A large-scale hierarchical image database
references.venue: In IEEE Conf. Comput. Vis. Pattern Recog.,
references.authors: Terrance DeVries Graham W Taylor
references.title: Learning confidence for out-of-distribution detection in neural networks
references.venue: arXiv preprint arXiv:1802.04865,
references.authors: Chuanxing Geng Sheng-jun Huang Songcan Chen
references.title: Recent advances in open set recognition: A survey
references.venue: IEEE transactions on pattern analysis and machine intelligence,
references.authors: Atin Ghosh Alexandre H Thiery
references.title: On data-augmentation and consistency-based semi-supervised learning
references.venue: arXiv preprint arXiv:2101.06967,
references.authors: Gregory Griffin Alex Holub Pietro Perona
references.title: Caltech-256 object category dataset
references.venue: Tech Report,
references.authors: Lan-Zhe Guo Zhen-Yu Zhang Yuan Jiang Yu-Feng Li Zhi-Hua Zhou
references.title: Safe deep semi-supervised learning for unseen-class unlabeled data
references.venue: In Int. Conf. Mach Learn.,
references.authors: Kaiming He Haoqi Fan Yuxin Wu Saining Xie Ross Girshick
references.title: Momentum contrast for unsupervised visual representation learning
references.venue: In IEEE Conf. Comput. Vis. Pattern Recog.,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In IEEE Conf. Comput. Vis. Pattern Recog.,
references.authors: Dan Hendrycks Kevin Gimpel
references.title: A baseline for detecting misclassified and out-of-distribution examples in neural networks
references.venue: arXiv preprint arXiv:1610.02136,
references.authors: Dan Hendrycks Mantas Mazeika Thomas Dietterich
references.title: Deep anomaly detection with outlier exposure
references.venue: arXiv preprint arXiv:1812.04606,
references.authors: Dan Hendrycks Mantas Mazeika Saurav Kadavath Dawn Song
references.title: Using self-supervised learning can improve model robustness and uncertainty
references.venue: arXiv preprint arXiv:1906.12340,
references.authors: Aditya Khosla Nityananda Jayadevaprakash Bangpeng Yao Fei-Fei Li
references.title: Novel dataset for fine-grained image categorization: Stanford dogs
references.venue: In Proc. CVPR Workshop on Fine-Grained Visual Categorization (FGVC),
references.authors: Alex Krizhevsky Geoffrey Hinton
references.title: Learning multiple layers of features from tiny images
references.venue: arXiv preprint,
references.authors: Samuli Laine Timo Aila
references.title: Temporal ensembling for semi-supervised learning
references.authors: Dong-Hyun Lee
references.title: Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks
references.venue: In Workshop on challenges in representation learning,
references.authors: Shiyu Liang Yixuan Li Rayadurgam Srikant
references.title: Enhancing the reliability of out-of-distribution image detection in neural networks
references.venue: arXiv preprint arXiv:1706.02690,
references.authors: Dongju Liu Jian Yu
references.title: Otsu method and k-means
references.venue: Ninth International Conference on Hybrid Intelligent Systems,
references.authors: Laurens van der Maaten Geoffrey Hinton
references.title: Visualizing data using t-sne
references.venue: JMLR, 9(11):2579–2605,
references.authors: Takeru Miyato Shin-ichi Maeda Masanori Koyama Ken Nakae Shin Ishii
references.title: Distributional smoothing with virtual adversarial training
references.authors: Yuval Netzer Tao Wang Adam Coates Alessandro Bissacco Bo Wu Andrew Y Ng
references.title: Reading digits in natural images with unsupervised feature learning
references.venue: In NIPS workshop on deep learning and unsupervised feature learning,
references.authors: M-E Nilsback Andrew Zisserman
references.title: A visual vocabulary for flower classification
references.venue: In IEEE Conf. Comput. Vis. Pattern Recog.,
references.authors: Shreyas Padhy Zachary Nado Jie Ren Jeremiah Liu Jasper Snoek Balaji Lakshminarayanan
references.title: Revisiting one-vs-all classifiers for predictive uncertainty and out-of-distribution detection in neural networks
references.authors: Salah Rifai Yann N Dauphin Pascal Vincent Yoshua Bengio Xavier Muller
references.title: The manifold tangent classifier
references.venue: Adv. Neural Inform. Process. Syst.,
references.authors: Kuniaki Saito Kate Saenko
references.title: Ovanet: One-vs-all network for universal domain adaptation
references.venue: arXiv preprint arXiv:2104.03344,
references.authors: Kuniaki Saito Shohei Yamamoto Yoshitaka Ushiku Tatsuya Harada
references.title: Open set domain adaptation by backpropagation
references.venue: In Eur. Conf. Comput. Vis.,
references.authors: Mehdi Sajjadi Mehran Javanmardi Tolga Tasdizen
references.title: Regularization with stochastic transformations and perturbations for deep semi-supervised learning
references.venue: arXiv preprint arXiv:1606.04586,
references.authors: Kihyuk Sohn David Berthelot Chun-Liang Li Zizhao Zhang Nicholas Carlini Ekin D Cubuk Alex Kurakin Han Zhang Colin Raffel
references.title: Fixmatch: Simplifying semi-supervised learning with consistency and confidence
references.authors: Jihoon Tack Sangwoo Mo Jongheon Jeong Jinwoo Shin
references.title: Csi: Novelty detection via contrastive learning on distributionally shifted instances
references.venue: arXiv preprint arXiv:2007.08176,
references.authors: Antti Tarvainen Harri Valpola
references.title: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
references.venue: arXiv preprint arXiv:1703.01780,
references.authors: Vikas Verma Kenji Kawaguchi Alex Lamb Juho Kannala Yoshua Bengio David Lopez-Paz
references.title: Interpolation consistency training for semi-supervised learning
references.authors: Catherine Wah Steve Branson Peter Welinder Pietro Perona Serge Belongie
references.title: The caltech-ucsd birds-200-2011 dataset
references.venue: Tech Report,
references.authors: Jim Winkens Rudy Bunel Abhijit Guha Roy Robert Stanforth Vivek Natarajan Joseph R Ledsam Patricia MacWilliams Pushmeet Kohli Alan Karthikesalingam Simon Kohl
references.title: Contrastive training for improved out-of-distribution detection
references.venue: arXiv preprint arXiv:2007.05566,
references.authors: Qizhe Xie Zihang Dai Eduard Hovy Minh-Thang Luong Quoc V Le
references.title: Unsupervised data augmentation for consistency training
references.authors: Kaichao You Mingsheng Long Zhangjie Cao Jianmin Wang Michael I. Jordan
references.title: Universal domain adaptation
references.venue: In IEEE Conf. Comput. Vis. Pattern Recog.,
references.authors: Fisher Yu Ari Seff Yinda Zhang Shuran Song Thomas Funkhouser Jianxiong Xiao
references.title: Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop
references.venue: arXiv preprint arXiv:1506.03365,
references.authors: Qing Yu Daiki Ikami Go Irie Kiyoharu Aizawa
references.title: Multi-task curriculum framework for open-set semi-supervised learning
references.venue: In European Conference on Computer Vision,
references.authors: Sergey Zagoruyko Nikos Komodakis
references.title: Wide residual networks
references.venue: arXiv preprint arXiv:1605.07146,
references.authors: Xiaohua Zhai Avital Oliver Alexander Kolesnikov Lucas Beyer
references.title: S4l: Self-supervised semi-supervised learning
references.venue: In Int. Conf. Comput. Vis.,
sections.heading: 1 Introduction
sections.text: Semi-supervised learning (SSL) leverages unlabeled data to improve a model’s performance [27, 38, 2, 1, 35, 22, 37]. An SSL model can propagate the class information of a small set of labeled data to a large set of unlabeled data, which significantly improves the recognition accuracy without any additional annotation cost. A common assumption of SSL is that the label spaces of labeled and unlabeled data are identical, but, in practice, the assumption is easily violated. Depending on how it was collected, the unlabeled data may contain novel categories unseen in the labeled training data, i.e., outliers. Since these outliers can significantly harm the performance of SSL algorithms [14], detecting them is necessary to make SSL more practical. Ideally, a model should classify samples of known categories i.e., inliers, into correct classes while identifying samples of novel categories as outliers. This task is called Open-set Semi-supervised Learning (OSSL) [44]. While OSSL is a more realistic and practical scenario than standard SSL, it has not been as widely explored.
  Existing strong SSL methods [35, 41] do not work well for OSSL. For example, FixMatch [35] generates pseudo-labels using the model’s predictions on weakly augmented unlabeled images and trains the model to match its predictions on strongly augmented images with the pseudo-labels. This method exploits the advantage of pseudo-labeling as well as regularizing a model with the consistency between differently augmented images. But, in OSSL, this risks assigning pseudo-labels of known categories to outliers, which degrades the recognition accuracy. A possible solution is to compute the SSL objective only for unlabeled samples considered to be inliers, where confidence thresholding is used to pick inliers. For instance, MTC [44] regards some proportion of samples as outliers by using Otsu thresholding [25]. However, this is not robust to varying proportions of outliers as discussed in
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  [44]. DS3L [14] proposes meta-optimization that attempts to pick unlabeled data useful to improve generalization performance. But, this method does not have an objective to separate inliers from outliers.
  Given the limitations of the existing methods, we aim to learn representations that separate outliers from inliers in a feature space and a threshold effective for detecting outliers. The challenge is learning such representations with a small number of labeled inliers and no supervision to find outliers. Moreover, choosing an accurate threshold value is not a trivial problem.
  We propose a new framework, OpenMatch, to address the above drawbacks of OSSL. First, we propose to utilize a One-Vs-All (OVA) network [32] that can learn a threshold to distinguish outliers from inliers. A separate OVA-classifier is trained for each class, and a sample is labeled an outlier if all of the classifiers determine it to be one. Thus, this technique allows us to identify outliers in an unsupervised way. We call this an outlier detector. Second, we propose a novel open-set soft-consistency loss to learn more effective representations for detecting outliers. We first transform an unlabeled input in two ways and obtain two logits from the outlier detector. Then, we minimize the distance between the two logits to encourage consistency (See Fig. 1). The main difference between SSL and OSSL is that unlabeled outliers do not have any neighboring labeled samples, which makes it risky to perform hard-labeling such as pseudo-labeling. The outlier detector outputs a distance from inliers given an input, and enhancing the smoothness of this function allows us to improve its ability to find outliers. Empirically, this objective provides significant improvements in detecting outliers. Finally, to correctly classify inliers, we propose to apply FixMatch [35] to unlabeled samples that are considered to be inliers by the outlier detector.
  The resulting framework shows consistent gains over baselines on various datasets and settings of OSSL. For example, OpenMatch achieves a 10.4% error rate on CIFAR10 with 300 labeled examples compared to the previous state-of-the-art of 20.3%. Surprisingly, OpenMatch demonstrates good performance in detecting outliers unseen in unlabeled training data. For instance, in the experiments on CIFAR10 with 100 labels per class, OpenMatch achieves a 3.4% higher AUROC in detecting outliers than a supervised model trained with all training samples. To summarize, our contributions are as follows:
  • A soft open-set consistency regularization (SOCR), to improve outlier detection in OSSL.
  • A new framework, OpenMatch, which combines a OVA-classifier, SOCR, and FixMatch.
  • A new state-of-the-art in both correctly classifying inliers and detecting outliers, even when the outliers are unseen in unlabeled training data.
sections.heading: 2 Related Work
sections.text: Semi-supervised Learning. Pseudo-labeling (PL) is a strong baseline in semi-supervised learning [23], and methods like FixMatch [35] or UDA [41] combining data augmentation and PL show the highest performance on many benchmark datasets. However, even with these strong methods,
  performance drops when a model is exposed to noisy unlabeled data that includes novel categories. On the other hand, methods based on soft consistency regularization enforce smoothness of the decision boundary with respect to stochastic transformations [34] or models [22, 37]. By "soft", we mean that no sharpening or pseudo-labeling is applied to logits to propagate training signals through soft targets. The soft consistency regularization can be understood as approximated Jacobian regularization method [12, 31]. We propose to apply soft consistency regularization to our outlier detector, which we call soft open-set consistency regularization. Training a model with self-supervised learning and SSL achieves label-efficient training [5, 46]. Self-supervised learning and SSL seem to be complementary to each other to improve the recognition accuracy. But, unlabeled data with outliers will hurt the performance of these approaches since they rely on existing SSL objectives. Since our work focuses on the aspect of SSL, we hope it is complementary to self-supervised learning.
  Open-set SSL methods include MTC [44], D3SL [14], and UASD [6]. MTC updates the network parameters and the anomaly score of unlabeled data alternately. In addition, it minimizes SSL loss (MixMatch [2] for a part of unlabeled data that is considered to be inliers. D3SL selectively employs unlabeled data to optimize SSL loss as well as optimizing a function that selects unlabeled data. UASD generates soft targets of the classifier for inliers (closed-set classifier) by averaging over many temporally ensembled networks’ predictions. By contrast, OpenMatch does not have to store the temporal ensembles and applies soft consistency loss to the outlier detector. Our empirical results show that the consistency loss for the outlier detector outperforms that for the closed-set classifier by a lot (See Sec. 4.4).
  Open-set Domain Adaptation (ODA). OSSL is similar to open-set domain adaptation [3, 33] in that the unlabeled data contains novel categories. A key difference is that the unlabeled and labeled data follow different data distributions. Another key difference is that we aim to train a model from scratch while the domain adaptation task assumes access to models pre-trained on ImageNet [9]. Since the ImageNet contains 1,000 diverse categories, the pre-trained models have discriminative representations useful to detect outliers. In this point, our task is more challenging than ODA. To detect outliers in an unsupervised way, we employ a OVA-classifier model [32] proposed for this task.
  Novelty Detection. Novelty detection or open-set classification aims to identify outliers that are completely unseen during training [17, 24, 10, 11]. Self-supervised learning such as rotation prediction and contrastive learning is shown to be useful to separate outliers from inliers [19, 36, 40]. The task assumes plenty of labeled inliers in training data. Padhy et al. employ OVA-classifiers for this task [30]. Hendrycks et al. reveal that exposing a model to outlier data allows it to effectively detect anomalies [18] and train it on out-of-distribution training data in a supervised way. But, collecting such useful out-of-distribution data is not always possible. In OSSL, a model cannot be trained on many labeled samples and needs to detect which samples are inlier or outliers in unlabeled data. In this sense, OSSL is more challenging and more realistic.
sections.heading: 3 OpenMatch
sections.text: Problem Setting. Our task is to learn a classifier using open-set semi-supervised learning. For a K-way classification problem, let X = {(xb, yb) : b ∈ (1, ...B)} be a batch of B labeled examples xb randomly sampled from the labeled training set, S, where yb ∈ (1, ...K) is the corresponding label. Let U = {(ub) : b ∈ (1, ...µB)} be a batch of µB data randomly sampled from the unlabeled training set, Su, where µ is a hyper-parameter that determines the relative sizes of X and U . Unlike in SSL, which assumes all unlabeled data come from the K known classes, in our OSSL setting they may come from unknown outlier classes. The goal of OSSL is to classify inliers into the correct classes while learning to detect the outliers.
  Approach Overview. Our model has three components: (1) a shared feature extractor F (·), (2) an outlier detector consisting of K one-vs-all (OVA) sub-classifiers Dj(·), j ∈ {1, ...,K}, and (3) a closed-set classifier C(·) which outputs a probability vector ∈ RK for K-class classification. At test time, the closed-set classifier is first applied to predict the K-way label, ŷ. If Dŷ predicts an inlier, then the output class is ŷ, otherwise, the output is "outlier". The main technical novelty comes from the choice of OVA-classifiers for outlier detection as well as training them with soft open-set consistency regularization. We first describe the training of the OVA-classifiers [32] before describing the remaining details of our framework.
sections.heading: 3.1 One-vs-All Outlier Detector
sections.text: For open-set [33] or universal domain adaptation [42], Saito et al. [32] propose to train OVA-classifiers to detect outlier samples of an unlabeled target domain. They aim to learn a boundary between inliers and outliers for each class. Each sub-classifier is trained to distinguish if the sample is an inlier for the corresponding class. For example, the sub-classifier for class j, Dj , outputs a 2-dimensional vector zjb = D
  j(F (xb)) ∈ R2, where each dimension indicates the score of a sample being an inlier and outlier respectively. We denote pj(t = 0|xb) and pj(t = 1|xb) as the probability of xb being the inlier and outlier for the class j, computed by Softmax(zjb) (Note that p
  j(t = 0|xb) + pj(t = 1|xb) = 1). Then, the sub-classifier Dj is trained such that the data from class j is treated as positives, and the data from all the other classes is treated as negatives. To achieve this, we minimize the following loss for the outlier detector for a given batch X := {xb, yb}Bb=1:
  Lova(X ) := 1
  B B∑ b=1 − log(pyb(t = 0|xb))−min i 6=yb log(pi(t = 1|xb)). (1)
  Here we use the hard-negative sub-classifier sampling technique proposed by [32] to effectively learn the threshold. We refer the reader to [32] for more details. The key is that each sub-classifier outputs a distance representing how far the input is from the corresponding class. Therefore, the classifiers are effective to identify unlabeled outliers. The output of Dj is used to detect outliers as follows. If the probability of a sample being an outlier for the class highest-scored by C() is larger than that of a sample being an inlier, we consider the input as an outlier. To be specific, we identify the unlabeled sample ub as an outlier if pŷ(t = 0|ub) < 0.5, where ŷ = argmaxj C(F (ub)). This detection is employed in selecting pseudo-inliers from unlabeled data as explained in Sec. 3.3.
  Open-set Entropy Minimization. For unlabeled samples, [32] apply entropy minimization with respect to the OVA-classifiers, denoted by Lem(U). The separation between inliers and outliers is enhanced through the minimization of this loss for a given batch U := {ub}µBb=1:
  Lem(U) := − 1
  µB µB∑ b=1 K∑ j=1 pj(t = 0|ub) log pj(t = 0|ub) + pj(t = 1|ub) log pj(t = 1|ub). (2)
  The difference between our setup and [32] is that they employ a model pre-trained on ImageNet [9], which extracts highly discriminative features, thus, inliers and outliers are well-separated even before fine-tuning. However, we aim to train a model to learn representations separating them from scratch. Thus the inliers and outliers are confused before training, and we need an objective to ensure the separation in addition to Eq. 2.
sections.heading: 3.2 Soft Open-set Consistency Regularization (SOCR)
sections.text: To motivate our approach, we describe the difference between SSL and OSSL in Fig. 2. In SSL, the hard consistency loss such as pseudo-labeling can be effective to propagate label information from
  Algorithm 1 OpenMatch Algorithm. 1: Input: Set of labeled data S = ( (xb, yb); b ∈ (1, . . . , N) ) , set of unlabeled data Su = ( ub; b ∈
  (1, . . . , Nu) ) , and set of pseudo-inlier data K = ∅. Data augmentation function T . Model parameters w, learning rate η, epoch Efix and Emax, iteration Imax, trade-off parameters, λem, λoc, λfm; for Epoch = 1 to Emax do
  for Iteration = 1 to Imax do 2: Sample a batch of labeled data X ∈ S and unlabeled data U ∈ Su; 3: Compute Lall = Lsup(X ) + λemLem(U) + λocLoc(U , T ); // Eq.1, 2 and 3 if Epoch > Efix then
  4: Sample a batch of pseudo-inliers I ∈ K; // Sample pseudo-inliers. 5: Compute Lall += λfmLfm(I) ; // FixMatch for pseudo-inliers. end 6: Update w = w − η∇wLall; // Update weights
  end if Epoch ≥ Efix then
  7: Update K = Select(w,Du); // Detect outliers and select pseudo-inliers. end
  end 8: Output: Model parameters w.
  labeled to unlabeled samples. Through the hard consistency loss, unlabeled samples get training signals from neighboring labeled samples (Fig. 2(a)). By contrast, in OSSL, since outliers are not assigned any labels, labeled samples are far from outliers, which makes the hard labeling unreliable, and unsupervised training signals can be incorrect (Fig. 2(b)). The open-set entropy minimization above can also be considered a variant of the hard consistency since it is minimized if the predicted output class distribution is one-hot. To propagate useful training signals to unlabeled samples, we propose to smooth the decision boundary of the outlier detector by minimizing the distance between its predictions on two augmentations of the same image (Fig. 2(c)). We use unsharpened logits, keeping the training signal soft to avoid introducing incorrect pseudo-labels. We call this loss soft open-set consistency regularization (SOCR). Given the smoothed outlier detector, the hope is that the training signals of the open-set entropy minimization will also become more correct.
  Specifically, SOCR enhances the smoothness of the outlier detector over data augmentation T , which in our experiments is just standard random cropping. Note that the standard cropping is also used in computing Eq. 1 and 2, but we omit it for the simplicity of notation. We obtain two different views of ub, T1(ub) and T2(ub), where T1 and T2 are data augmentation functions stochastically sampled from T . Let pj(t|T1(ub)) be the output logits of T1(ub) from the jth OVA-classifier. We encourage the consistency of the output logits over T to enhance the smoothness by minimizing the following loss for a given batch U := {ub}µBb=1:
  Loc(U , T ) := 1
  µB µB∑ b=1 K∑ j=1 ∑ t∈(0,1) |pj(t|T1(ub))− pj(t|T2(ub))|2. (3)
  This minimizes the distance between two probability outputs to ensure smoothness with respect to T . Note that the key part of the regularization is not applying any sharpening to the output logits, keeping them soft. The major difference from the previous soft consistency regularization [12, 37, 22] is that we use the outlier detector to compute the regularization loss.
sections.heading: 3.3 Overall Framework
sections.text: The entire OpenMatch algorithm is described in Alg. 1. The unsupervised losses (Eqs. 2, 3) for the outlier detector are effective to detect outliers in unlabeled data. However, these losses are not sufficient to correctly classify unlabeled inliers. Therefore, we propose to introduce a semi-supervised learning loss for unlabeled samples classified as inliers. We adopt FixMatch [35] since it is a simple, yet very strong SSL method. FixMatch first generates pseudo-labels using the model’s predictions on weakly augmented unlabeled images. The model is then trained to predict these pseudo-labels when fed a strongly augmented version of the same image. "Weak" denotes augmentations such as a simple
  random cropping while "strong" refers to extensive data augmentation such as RandAugment [8] and CTAugment [1].
  For labeled samples, we also compute the standard cross-entropy loss to train C(·), Lcls(X ), for the closed-set output, C(F (xb)). We summarize Lsup(X ) as the sum of Lova(X ) and Lcls(X ). After training a model for Efix epochs with Lsup, Lem, Loc, we begin to select pseudo-inliers from unlabeled data at every following epoch. Specifically, in Alg. 1, Select(w,Du) denotes the process of classifying unlabeled samples given model parameters w of models (F (·), D(·), and C(·)), and unlabeled data. The outlier-detection process is summarized in Sec. 3.1.
  Then, the FixMatch loss, Lfm(I), is computed only for the pseudo-inliers to optimize the model, where I denotes a batch of pseudo-inliers. The following is the resulting overall objective of OpenMatch:
  Lall(X ,U , T , I) := Lsup(X ) + λemLem(X ) + λocLoc(U , T ) + λfmLfm(I), (4) where λem, λoc and λfm control the trade-off for each objective.
sections.heading: 4 Experiments
sections.text: Setup. We evaluate the efficacy of OpenMatch on several SSL image classification benchmarks. Specifically, we perform experiments with varying amounts of labeled data and varying numbers of known/unknown classes on CIFAR10/100 [21] and ImageNet [9]. We attempt to cover various scenarios by using these three datasets.
  Note that we use an identical set of hyper-parameters except for λoc, which is tuned on each dataset. λem is set 0.1 in all experiments. λfm is set to 0 before Efix epochs and then set to 1 for all experiments. Efix is set to 10 in all experiments. The hyper-parameters for FixMatch, e.g., data augmentation, confidence threshold, are fixed across all experiments for simplicity. The hyperparameters are set by tuning on a validation set that contains a small number of labeled samples. Note that the validation set does not contain any outliers. A complete list of hyper-parameters is reported in the appendix. Each experiment is done with a single 12-GB GPU, such as an NVIDIA TitanX.
  Baselines. For the OSSL baseline, we employ MTC [44] using the author’s implementation. In addition, we show the result of a model trained only with labeled samples (Labeled Only). We exclude UASD [6] and D3SL [14] from our baselines since their reported results underperform a model we train only with labeled samples. For instance, in the experiments for CIFAR10 at 400 labels per class, their reported error rates are worse than 20% while the rate of the Labeled Only baseline is 20.0%. A comparison with FixMatch [35] reveals the effect of OVA-classifiers trained with the soft consistency loss. The hyper-parameters of both OpenMatch and baselines are tuned by maximizing the accuracy on the validation set. Since the validation set does not have any outliers, we choose the validation accuracy as a criterion.
  Evaluation. We assume the test set contains both known (inlier) and unknown (outlier) classes. For known classes, classification accuracy is used to evaluate performance. To evaluate the separation into inliers and outliers, we use AUROC following the standard evaluation protocol of novelty detection [17]. To compute the anomaly score, the outlier detector’s score is employed for OpenMatch, Labeled Only, and FixMatch, where we add our outlier detector to the latter two models. We report the results averaged over three runs and their standard deviations.
sections.heading: 4.1 CIFAR10 and CIFAR100
sections.text: We compare OpenMatch with several baselines on the standard benchmark datasets of SSL, CIFAR10, and CIFAR100. Following [44], we use a randomly initialized Wide ResNet-28-2 [45] with 1.5M parameters in these experiments. For CIFAR10, we split the classes into known and unknown classes by defining animal classes as known and others as unknown, which results in 6 known classes and 4 unknown classes. For CIFAR100, we first split the classes by their super-classes provided by the dataset so that known and unknown classes do not share super-classes. Then, we split the super-classes into known and unknown. To evaluate performance with different numbers of outliers, we run experiments on two settings: 80 known classes (20 unknown classes) and 55 known classes (45 unknown classes).
  Tables 1 and 2 describe the error rate on inliers and AUROC values respectively. Note that the number of labeled samples per class is shown in each column. With respect to the error rate, OpenMatch achieves state-of-the-art performance in all cases, and the gain over the baselines is remarkable in
  many cases. In particular, we see about a 10% improvement in CIFAR10 error at 50 labels. In addition, the improvements in AUROC are also clear. MTC [44] does not improve the performance very much compared to the labeled-only model on CIFAR100 because their method of identifying outliers, i.e., Otsu’s thresholding [25], is not very robust to the number of outliers as discussed in their paper. On the other hand, OpenMatch improves the error rate by more than 3% and AUROC by more than 6% in all cases. Simply applying FixMatch [35] also does not always improve the error compared to the labeled-only model. This is because FixMatch confuses inliers and outliers and assigns pseudo-labels to outliers, as the significant decrease in AUROC indicates.
sections.heading: 4.2 ImageNet
sections.text: We evaluate OpenMatch on ImageNet to see its behavior on more complex and challenging datasets. Since training on the entire ImageNet is computationally expensive, we choose ImageNet-30 [17], which is a subset of ImageNet containing 30 classes. The subset is useful for evaluation on unseen outliers as done in Sec. 4.5, because the classes do not include several super-classes such as a bird class. We pick the first 20 classes, in alphabetical order, as known classes, and the remaining 10 classes as unknown. We utilize the ResNet-18 [16] network architecture. Hyper-parameters such as a batch-size, learning rate, data augmentation are the same as ones used in CIFAR10 and CIFAR100. As shown in the rightmost columns of Tables 1 and 2, OpenMatch achieves the best performance in both accuracy and AUROC. Unlike the results on CIFAR, FixMatch improves the performance over the Labeled Only model. This is probably because outliers are easier to detect, as the AUROC score of the Labeled Only model already shows a higher score than other scenarios. Therefore, even without any outlier detection objectives, FixMatch can correctly select pseudo-inliers with confidence thresholding.
sections.heading: 4.3 Ablation Study on Soft Open-set Consistency Regularization
sections.text: In this section, we validate our claim that the SOCR regularization is effective at separating inliers and outliers. We perform several ablation studies to see the benefit of SOCR and measure the model’s ability to detect outliers using AUROC. In summary, both numerical and qualitative evaluations confirm the effectiveness of SOCR.
  Table 3 describes the numerical comparison of ablated models. In this evaluation, we do not apply FixMatch to pseudo-inliers to measure the pure gain from the consistency loss. Introducing SOCR increases AUROC by more than 20% in CIFAR10, 8% in CIFAR100, and 8% in ImageNet. These results support the effectiveness of smoothing the outlier detector’s output.
  Histograms of Anomaly Scores. Fig. 3 illustrates the anomaly score of inlier and outlier test samples in the ablation experiment on CIFAR10. If SOCR is not employed (Fig. 3(a)), inliers and outliers are confused while SOCR makes the scores of outliers larger and enhances the separation (Fig. 3(b)). Further applying FixMatch to pseudo-inliers decreases the score of inliers (Fig. 3(c)), which separates inliers and outliers better. Considering the results of Table 2, just applying FixMatch to all unlabeled samples is often not useful to detect outliers, but applying it to the pseudo-inliers can improve the outlier detection.
sections.heading: 4.4 Analysis
sections.text: Feature Visualization. Fig. 4 visualizes the feature distributions with T-SNE [26] in the experiment on CIFAR10 at 100 labels. OpenMatch (c) separates inliers and outliers well while a model trained only with 100 labeled samples (a) confuses them. The separation of OpenMatch appears to be better than a model trained with all labeled inliers (b).
  Closed-set Consistency vs Open-set Consistency. The soft consistency loss is computed for the outlier detector in OpenMatch. An alternative is to compute it for the closed-set classifier as done in UASD [6]. We replace SOCR with the soft consistency loss with respect to the closed-set classifier and perform the same analysis as in Sec. 4.3. In CIFAR10 at 50 labeled samples, this model achieves 70.6 ± 1.3 % AUROC, which is 10.1 % better than a model without consistency loss and 10.7 % worse than a model with SOCR. In CIFAR10 at 400 labeled samples, this model achieves 88.5± 2.4 % AUROC, which is 12.7 % better than a model without consistency loss and 8.3 % worse than a model with SOCR. These results indicate that even using soft consistency for the closed-set classifier helps to detect outliers, as shown in UASD [6], but SOCR helps much more. Thus, our choice of combining both the outlier detector and soft consistency loss is validated in this analysis.
  Comparison with OOD with self-supervised learning. We compare our method with outlier detection methods using self-supervised learning. Several works [19, 36] have shown that training a model with self-supervised learning objectives such as contrastive loss or rotation prediction loss improves OOD performance. We provide a brief analysis of these baselines in open-set SSL.
  Does self-supervised pre-training help open-set SSL? A model trained with self-supervised learning objectives shows great performance in many downstream image recognition tasks. We investigate whether initializing a model trained with self-supervised learning (SimCLR [4]) improves perfor-
  mance in open-set SSL (Table 4). We pre-train a model with SimCLR loss using all unlabeled data. The initialization does not make a clear difference from random initialization, i.e., default initialization.
  Does self-supervised training help open-set SSL? Next, we investigate whether using selfsupervised learning loss in addition to the classification loss on labeled data helps open-set SSL. In Table 5, we show the results of adding self-supervised loss for labeled data or unlabeled data. SimCLR does not help in any case compared to a model trained only with classification loss for labeled data. Constrative learning basically attempts to uniformly distribute all samples in a unit sphere, which does not necessarily help in open-set SSL. The goal of open-set SSL is to keep outliers far from inliers whereas clustering inliers within each class. By contrast, rotation prediction improves performance if applied to labeled samples in CIFAR-10. If applied to unlabeled data, a model can assign the same rotation labels to both inliers and outliers, which can prevent from separating them.
sections.heading: 4.5 Novelty Detection
sections.text: We have seen that OpenMatch can learn to detect outliers from unlabeled data. In this experiment, we evaluate how well our OSSL model can separate inliers from outliers unseen in the unlabeled data. This setting is similar to general outlier or out-of-distribution sample detection. Note that the models evaluated in this section are the same as the ones in Tables 1 and 2. We consider the following datasets as out-of-distribution: SVHN [28], resized LSUN [43], ImageNet, and CIFAR100 for CIFAR10 experiments (Table 6(a)), and LSUN, CUB-200 [39], Dogs [20], Caltech [13], Flowers [29], and DTD [7] for ImageNet-30 (Table 6(b)). See supplemental material for the details. We utilize AUROC to measure how well inliers and outliers are separated. To see the gap from a supervised model, we train a model using all labeled samples of inlier classes and show the comparison.
  As seen in Table 6, OpenMatch outperforms OSSL baselines by 5.8% on average and even the supervised model by 3.4% on average on CIFAR10. On ImageNet-30, OpenMatch outperforms OSSL baselines by more than 4.1% on average. This result verifies that OpenMatch is robust to various out-of-distribution data by virtue of being exposed to unlabeled data containing outliers.
sections.heading: 5 Conclusion
sections.text: In this paper, we introduce a method for open-set semi-supervised learning (OSSL), where samples of novel categories are present in unlabeled data. To approach the task, we propose a novel framework, OpenMatch, which unifies one-vs-all classifiers and FixMatch. Our proposed objective, open-set soft consistency loss, is shown to be effective to detect outliers from unlabeled data, which allows FixMatch to work well in the OSSL setting. In addition, OpenMatch sometimes detects outliers unseen in unlabeled data better than a supervised model. We believe that our framework for OSSL will make label-efficient techniques more practical.
  Limitations. OpenMatch can have difficulty in detecting outliers similar to inliers, though any methods for outlier detection can have the same limitation. If the outliers have similar visual characteristics, separating them from inliers samples is difficult. A potential solution is to introduce self-supervised learning [15, 4] for unlabeled data.
  Broader Impact. OpenMatch is an effective tool for semi-supervised learning with noisy unlabeled data containing outliers. It would benefit projects with small budgets or hard-to-label data. At the same time, it is possible for malicious actors to exploit the advantages of learning from limited data. Broadly speaking, any progress on semi-supervised learning can have these consequences.
sections.heading: 6 Acknowledgement
sections.text: This work was supported by Honda, DARPA LwLL and NSF Award No. 1535797.
title: OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers