The Out-of-Domain (OOD) intent classification is a basic and challenging task for dialogue systems. Previous methods commonly restrict the region (in feature space) of In-domain (IND) intent features to be compact or simply-connected implicitly, which assumes no OOD intents reside, to learn discriminative semantic features. Then the distribution of the IND intent features is often assumed to obey a hypothetical distribution (Gaussian mostly) and samples outside this distribution are regarded as OOD samples. In this paper, we start from the nature of OOD intent classification and explore its optimization objective. We further propose a simple yet effective method, named KNN-contrastive learning. Our approach utilizes K-Nearest Neighbors (KNN) of IND intents to learn discriminative semantic features that are more conducive to OOD detection. Notably, the density-based novelty detection algorithm is so well-grounded in the essence of our method that it is reasonable to use it as the OOD detection algorithm without making any requirements for the feature distribution. Extensive experiments on four public datasets show that our approach can not only enhance the OOD detection performance substantially but also improve the IND intent classification while requiring no restrictions on feature distribution. Code is available. 1 1 Introduction People are getting accustomed to talking with or sending some instructions to task-oriented dialog system in natural language to assist them in fulfilling work. As the environment facing the dialogue systems becomes more open, there are more utterances with unknown or Out-of-Domain (OOD) intents that the dialog system does not know how to handle. As shown in Figure 1, the chatbot encounters an unsupported intent (OOD intent) utterance  Corresponding author. in the third interaction. It is significant to distinguish these utterances for the dialogue system because identifying the intents of the user determines whether subsequent actions can be carried out correctly. To solve this problem, the existing methods roughly can be summarized into two categories according to whether extensive labeled OOD intent samples are used during training. The first kind of method (use OOD samples during training) is represented by (Zheng et al., 2020; Zhan et al., 2021; Choi et al., 2021) which regards OOD intent classification as a (n+1)-class classification task that the extra ( n + 1) th class represents labeled OOD intent. These methods may need additional large and time-consuming labeled Out-of-Domain samples. Moreover, manually constructed OOD samples endowed with artificial inductive bias cannot cover all open classes in the actual environment so this kind of method have their limitations. In this paper, we focus on another kind of method which involves two stages, to learn dis-5129 How to learn semantic features benefit for OOD detection? Minimizing intra-class variance and maximizing inter-class variance, whose motivation are to facilitate detection by widening margin between IND and OOD intents, have always been regarded as the essence of solving this problem (Lin and Xu, 2019; Zeng et al., 2021a). Then, Shu et al. (2017); Zhang et al. (2021); Xu et al. (2020); Zeng et al. (2021a) impose (or implicitly) Gaussian distribution into the distribution of the learned intent features to OOD detection. In general previous methods implicitly assume the region of semantic features as compact or simply-connected region in feature space, which means OOD intents only exist between different IND classes and not within IND distribution so that tight IND semantic features can be helpful for OOD detection. However, as shown in Figure 2 (a), the actual location of OOD intents in semantic space is not limited, they can appear between IND classes or within IND distribution. We name those OOD intents that are distributed between different IND classes as inter OOD intents, and those within IND distribution or can be surrounded by a convex hull composed of local IND intent samples as intra OOD intents. As shown in Figure 2 (b), for inter OOD intents, minimizing intra-class variance and maximizing inter-class variance can reduce the risk of being identified IND while this risk may be increased for intra OOD intents due to being closer to IND intents. At the same time, we conduct Gaussian Hypothesis Testing on the IND semantic features distribution in the CLINC-FULL train set, which is learned by (Zeng et al., 2021a). we find only 57% IND classes conform to Gaussian distribution, which illustrates the Gaussian assumption for OOD detection in the previous methods may not be reasonable, see Appendix A.1 for results on other datasets and details. To solve these problems, we explicitly define the optimization objective of OOD intents classification using open space risk (Scheirer et al., 2012). Compared with the previous methods only considering inter OOD intents, we propose a simple yet effective method to consider both intra and inter OOD intents. We utilize k-nearest neighbors of IND intent sample as positive samples as shown in Figure 2 (c), and obtain more negative samples with the help of queue in MoCo (He et al., 2020) to learn discriminative semantic features. We further analyze why our method can better reduce open space risk. Intuitively, our method leaves more margin around OOD intents, which can ensure we employ the basic density-based method for OOD detection without any assumptions about the distribution. We summarize our contributions as follows. Firstly, following open space risk, we explicit the optimization objective of OOD intent classification to provide a paradigm for solving OOD intent classification. Secondly, we analyze the limitation of existing methods and propose a novel method to better reduce both empirical risk and open space risk. Thirdly, extensive experiments conducted on four challenging datasets show our approach achieves consistent improvements without restrictions on feature distribution. Schlkopf et al. (2001); Tax and Duin (2004) regard Out-of-Domain detection as One-class classification problem so that to find a hyperplane or hypersphere by kernels in high-dimensional space to distinguish OOD, Scheirer et al. (2012) firstly proposes the definition of open space risk and formalize open set recognition as a constrained optimization problem. Then to obtain better semantic representation, deep neural network has been brought into this field in recent years. Bendale and Boult (2016) propose OpenMax model, using the scores from the penultimate layer of deep network, to distinguish OOD. MSP (Hendrycks and Gimpel, 2017) presents a baseline based on maximum softmax probabilities to exhibit the ability of the network to distinguish between OOD and IND. To enlarge the difference between IND and OOD, Liang et al. (2018) add temperature scaling based on MSP and adds perturbations to the inputs. The above methods mainly focus on computer vision and assume (or implicitly) the feature region is compact (simply-connected region), many research works are carried out in natural language processing. DOC (Shu et al., 2017) builds a multi-class classifier and selects a threshold to reject. Further, they reduces the open space risk for rejection by tightening the decision boundaries of sigmoid functions with Gaussian fitting. LMCL (Lin and Xu, 2019) learns discriminative deep features with margin loss. Yan et al. (2020); Wan et al. (2018) model embeddings with a Gaussian mixture distribution to facilitate downstream outlier detection. Xu et al. (2020); Zeng et al. (2021b); Podolskiy et al. (2021) assume the IND semantic feature distribution as Gaussian discriminant analysis (GDA) and identify Out-of-domain samples by Mahalanobis distances. (Fei and Liu, 2016) reduce open space risk by decision boundaries and ABD (Zhang et al., 2021) propose to learn adaptive circular decision boundaries. Very recently, Zeng et al. (2021a) propose a supervised contrastive learning objective to maximize inter-class variance and to minimize intra-class variance. These methods also restrict the feature distribution in the feature learning stage or downstream detection stage and fail to solve Out-of-domain classification completely. Contrastive learning, which can be traced back to (Hadsell et al., 2006), is widely used in unsupervised or self-supervised learning (He et al., 2020; Wang and Isola, 2020; Khosla et al., 2020). With similarity by dot product, Gutmann and Hyvrinen (2010) propose InfoNCE loss to measure the similarities of sample pairs in semantic space. To obtain more number of negative samples for contrastive learning, He et al. (2020) introduce Momentum Contrastive (MoCo) that builds a large and consistent dictionary facilitating contrastive unsupervised learning. With the prevalence of pre-trained models (PTMs) (Qiu et al., 2020; Lin et al., 2021) in different fields, Dwibedi et al. (2021); Li et al. (2021) combine PTMs with contrastive learning paradigm, which adopts neighbors and uses MoCo or Memory Banks to obtain enough negative samples. Open space risk We define open space O and open space risk as follow (Scheirer et al., 2012; Bendale and Boult, 2015). Using IND training samples, open space can be defined as 2 : O = S  (cid:91) x X  ( x ) , (1) where  is a local (and small) semantic space spanned by IND training sample x , X is the set of all IND training samples and S including both open space O and remaining space. Consider a measurable recognizer (or discriminator) f that f ( x ) = 1( > 0) for the IND intents, otherwise f ( x ) = 0( < = 0) , probabilistic open space risk RO can be formed in terms of Lebesgue measure: RO ( f ) = (cid:82) O f ( x,  ) dx (cid:82) S f ( x,  ) dx . Objective of OOD Intent Classification To identify OOD intents, we need to learn intent representations at first, which also ensures the classification quality of IND in addition to adapting to downstream detection. Therefore we introduce an additional optimization objective as also suggested in (Scheirer et al., 2012; Bendale and Boult, 2015), 2 We also define a specific open space risk for limited OOD samples. where  is a hyper-parameter to balance empirical and open space risk and H is the function space. To optimize above objective. We first utilize the BERT (Devlin et al., 2019) to extract intent representation. Given the i-th in-domain utterance, we get its contextual embeddings [[ CLS ] , T 1 , T 2 , ..., TN ] . As suggested in (Zhang et al., 2021), we operate mean-pooling on these contextual token embeddings to obtain sentence semantic representation Z i : Z i = Mean-Pooling ([[ CLS ] , T 1 , ...T N ]) , (4) where Z i  RH , N is the sequence length and H is the hidden dimension. It is worth noting that due to the lack of OOD intent samples, we can not directly optimize the open space risk. Previous methods indirectly reduce the open space risk by pulling together IND samples belonging to the same class and pushing apart samples from different classes. However, such approaches may increase the risk of identifying intra OOD intents as IND based on the above analysis. Intuitively, to reduce the risk that identifying intra OOD intents as IND, we do not need to pull together all IND intent samples belonging to the same class and just pull together k-nearest neighbors while pushing apart them from different class intent samples as shown in Figure 2 (c). In order to achieve this goal, we get the KNN-contrastive loss L knn  cl by rewriting the contrastive loss: L knn-cl = N (cid:88) i =1 1 |X k | (cid:88) z j X k  log exp( z i  z j  ) (cid:80) z q  I exp( z q  z i  ) , (6) where X k denotes the set of k-nearest neighbors of sample z i , I  A (cid:83) { z j } , A is the set of samples whose classes are different from that of z j . is the temperature hyper-parameter. We further analyze how KNN-contrastive loss is benefit to inter OOD intents and intra OOD intents simultaneously, see Appendix A.5 and Appendix A.6 for more detailed and deeper analysis. When conducting KNN-contrastive learning, we need to solve two problems: a) large batch size, the more samples we can select, the more likely we to find k-nearest neighbors. Meanwhile, we also need enough negatives to distinguish. b) The k-nearest neighbors should keep consistent as they evolve during training, otherwise, KNN-contrastive train learning may be unstable. Interestingly, these problems mentioned above are also those Momentum Contrast (MoCo) (He et al., 2020) wants to solve. Following MoCo, we also maintain a queue containing IND samples and update it with features of the current batch while dequeuing the oldest features. The queue decouples the size of samples from the batch size, allowing us to obtain more negative samples (benefit to reduce open space risk). To maintain consistency, the features coming from the previous several batches are encoded by a slowly updating network (encoder), whose parameters are momentum-based average of the parameters from the query encoder (another net-work), see (He et al., 2020) for details. Combing softmax cross-entropy loss and KNN-contrastive learning loss, the final finetune obejective to learn discriminative features as: L obj =   L knn-cl + (1   )  L ce , (7) where  is a hyper-parameter to balance empirical and open space risk. To be closer to the realistic scenario, we prefer the detection algorithm downstream without assuming a potential distribution of the IND intents. Therefore, we adopt a simple and universal detection algorithm LOF algorithm (Breunig et al., 2000) and compute LOF score following (Lin and Xu, 2019), see Appendix A.3 for specific calculation steps. Our model architecture is shown in Figure 3. In order to verify the effectiveness and generality of our method, we conduct experiments on four different and challenging real-world datasets. The detailed statistics are shown in Appendix A.4. CLINC-FULL (Larson et al., 2019) is a dataset specially designed for OOD detection, which consists of 150 classes from 10 domains. This dataset includes 22500 IND utterances and 1200 OOD utterances. CLINC-SMALL (Larson et al., 2019) is the CLINC-FULL variant, in which there are only 50 training utterances per each IND class. This dataset includes 15000 IND utterances and 1200 OOD utterances. BANKING (Casanueva et al., 2020) is a dataset about banking. The dataset, covering 77 classes, consists of 9003, 1000 and 3080 utterances in training, validation and test sets respectively. StackOverflow (Xu et al., 2015) is a dataset published in Kaggle.com. This dataset has 20 classes and consists of 12000, 2000 and 6000 in training, validation and test sets respectively. We extensively compare our method with the following OOD classification methods: MSP (Hendrycks and Gimpel, 2017), DOC (Shu et al., 2017), SEG (Yan et al., 2020), LMCL (Lin and Xu, 2019), Softmax (Zhan et al., 2021), OpenMax (Ben-dale and Boult, 2016), ADB (Zhang et al., 2021), SCL (Zeng et al., 2021a). the backbone network. We report the current best results of various methods on the corresponding datasets. Softmax/LMCL learns discriminative features by softmax/large margin cosine loss and use additional detector such as LOF or GDA for detecting out-of-domain. ADB (Zhang et al., 2021)/SCL (Zeng et al., 2021a) are also related to our method, however, the original paper does not report results in on some datasets. We supplement results by running their released code. For all datasets, we follow previous work (Zhang et al., 2021; Zeng et al., 2021a; Zhan et al., 2021) and group all OOD classes as one rejected class. We calculate accuracy and F1-score in the same way as (Zeng et al., 2021a). To better evaluate the ability of our method to distinguish IND and OOD intents, we calculate macro F1-score over IND classes and OOD classes, represented by F1-IND and F1-OOD respectively. To comprehensively evaluate the performance of our model, we also compare accuracy score ( ACC-ALL ) and macro F1-score ( F1-ALL ) over all classes (IND and OOD classes). Due to no OOD classes in the BANKING and StackOverflow, we follow the setting in (Zhang et al., 2021; Zhan et al., 2021). After datasets are split into train, validation, and test respectively, we randomly sample 25%, 50%, and 75% of the intent classes and discard the remaining classes in the 5133 training and validation sets 3 . The disposed classes are kept in the test set as OOD classes. CLINC-FULL and CLINC-SMALL are constructed for OOD detection especially and the datasets themselves provide OOD classes. We follow (Zeng et al., 2021a,b) and take the OOD class provided by datasets as the objective of detecting without dividing datasets additionally. As a reminder, we do not use OOD classes during training in any cases. To reduce the deviation, we use two basic distances, Euclidean and Cosine (more discussed in Appendix A.2), to calculate the LOF score. For each distance, we take different random seeds to run 3 rounds, and we report the total average results. We employ the BERT model (bert-uncased, with 12-layer transformer) implemented by Hugging-face Transformers 4 and adopt most of its suggested hyperparameters for finetuning. We tried learning rate in {1e-5, 5e-5}, and training batch size is set 16 or 32. Concerning contrastive learning, we tried the size of the queue in {6500, 7500, 8000} and the momentum update parameter m = 0.999. We use the AdamW optimizer (Loshchilov and Hutter, 2019). We select the LOF threshold by calculating the best macro F1-score and accuracy over IND classes on the validation set. ALL experiments were conducted in the Nvidia Ge-Force RTX-2080 Graphical Card with 11G graphical memory. The results in BANKING and StackOverflow are presented in Table 1, where the best results are highlighted in bold. Compared with other baselines, our method consistently improves in all settings. F1-OOD represents the F1-score of OOD class, and F1-IND is the macro F1-score of IND classes. Our method achieves favorable performance simultaneously, which shows that our method improves the capability of detecting OOD intents without sacrificing the accuracy of IND classes classification. The results in CLINC-FULL and CLINC-SMALL datasets are presented in Table 2. Our method is also better than all other kinds of methods. It is worth noting that F1-IND and F1-OOD have improved compared with SCL significantly. We suppose this is due to the use of Momentum Contrast framework in our method, which obtains more negative samples by maintaining a queue (the capacity is much larger than batch size) so that fur-3 See more discusses in Appendix A.7. ther pushes apart samples from different classes, as shown in Section 5.1. To compare our method with the previous methods intuitively, we use t-SNE (Van der Maaten and Hinton, 2008) to visualize deep features of some intent samples sampled from CLINC-FULL learned by BERT, SCL, and our method. Firstly, compared 5134 Methods BANKING StackOverflow ACC-ALL F1-ALL F1-OOD F1-IND ACC-ALL F1-ALL F1-OOD F1-IND 25% MSP  43.67 50.09 41.43 50.55 28.67 37.85 13.03 42.82 DOC  56.99 58.03 61.42 57.85 42.74 47.73 41.25 49.02 OpenMax  49.94 54.14 51.32 54.28 40.28 45.98 36.41 47.89 Softmax  57.88 58.32 62.52 58.10 46.17 50.78 42.52 51.83 LMCL  64.21 61.36 70.44 60.88 47.84 52.05 49.29 52.60 SEG  51.11 55.68 53.22 55.81 47.00 52.83 46.17 54.16 ADB  78.85 71.62 84.56 70.94 86.72 80.83 90.88 78.82 SCL+GDA  83.87 67.94 89.44 66.81 82.29 70.92 88.99 67.44 SCL+LOF  84.05 74.86 89.01 74.12 80.10 78.51 84.45 77.32 Ours 85.62 77.13 90.19 76.44 89.04 81.61 92.7 79.39 50% MSP  59.73 71.18 41.19 71.97 52.42 63.01 23.99 66.91 DOC  64.81 73.12 55.14 73.59 52.53 62.84 25.44 66.58 OpenMax  65.31 74.24 54.33 74.76 60.35 68.18 45.00 70.49 Softmax  67.44 74.19 60.28 74.56 65.96 71.94 56.80 73.45 LMCL  72.73 77.53 69.53 77.74 58.98 68.01 43.01 70.51 SEG  68.44 76.48 60.42 76.90 68.50 74.18 60.89 75.51 ADB  78.86 80.90 78.44 80.96 86.40 85.83 87.34 85.68 SCL+GDA  79.38 79.84 79.97 79.83 82.31 79.54 84.42 79.04 SCL+LOF  80.54 82.4 80.42 82.6 84.47 84.57 85.01 84.53 Ours 83.14 83.87 83.58 83.88 87.62 87.18 88.36 87.06 75% MSP  75.89 83.60 39.23 84.36 72.17 77.95 33.96 80.88 DOC  76.77 83.34 50.60 83.91 68.91 75.06 16.76 78.95 OpenMax  77.45 84.07 50.85 84.64 74.42 79.78 44.87 82.11 Softmax  78.20 84.31 56.90 84.78 77.41 82.28 54.07 84.11 LMCL  78.52 84.31 58.54 84.75 72.33 78.28 37.59 81.00 SEG  78.87 85.66 54.43 86.20 80.83 84.78 62.30 86.28 ADB  81.08 85.96 66.47 86.29 82.78 85.99 73.86 86.80 SCL+GDA  79.86 85.14 64.49 85.5 80.88 84.79 68.83 85.86 SCL+LOF  81.56 86.97 65.05 87.35 80.92 83.98 71.71 84.79 Ours 81.77 87.07 67.66 87.41 83.85 87.06 74.20 87.92 Table 1: Results of OOD classificaion with different IND classes rate (25%, 50% and 75%) on BANKING and StackOverflow. with SCL Figure 4(b), our method further push apart samples from different classes, especially for the classes of whisper-mode(green) and can-cel(blue) . This shows our method can ensure the classification quality of IND intents and better optimize empirical risk. Can our method optimize open space risk better? As shown in Figure 4(c) and Figure 4(d), we notice that the features of vaccines class (brown) can be clustered into three clusters (top-k=5) or two clusters (top-k=15), which means leave more space for OOD intent samples as expected. To further verify the effectiveness of our method to characterize the difference between IND and OOD intents, we draw the LOF score distribution of samples in CLNIC-FULL test set. We show results in Figure 5. Compared with SCL (left) Figure 5(b), our LOF scores of OOD intent samples are spread in a larger range, which indicates that there is a larger margin around OOD intent samples according to the definition of LOF and further shows our method optimize the open space risk better. And this larger margin makes it is easier for us to find a baseline (brown dotted line) in Figure 5(b) (our method) to separate IND and OOD samples, which indicates they are distinguished better. The above results also ensure that we can detect OOD intents without making assumptions about the feature distribution. In Section 3.3, we propose a novel method, which utilizes the k-nearest neighbors of IND intents to learn discriminative features, to reduce open space risk. To investigate the effect of k, we compare the performance of the model in detecting OOD intents with different k values (in a certain range) during contrastive learning (fixing other hyper-parameters). As shown in Figure 6, we have observed that the performance (cosine-based) of the model first increase and then decrease on four datasets as the value increases. This phenomenon is as expected. At the beginning of k growth, due to the reduction of open space risk, the risk of inter and intra OOD samples being identified as IND decreases (corresponding F1-OOD increases). Later, due to the compression of IND semantic space, more and more intra OOD samples are identified as IND (corresponding F1-OOD decreases) and finally tend to be stable. The phenomenon also shows that our method can better reduce the open space risk than the previous methods (which pull together all IND intents belonging to the same class). While we want to minimize open space risk (help-ful to detect OOD intents), we also need to balance it against the empirical risk (ensure the classification quality of IND classification) over the training data. In the objective (in Eq.(7)) of OOD classification, we use the  hyper-parameter to balance empirical and open space risk. To analyze the effect of our introduced  , we experiment our method (cosine-based) with different  s in different datasets. As shown in Figure 7, we find with the increment of  , the model gradually reaches the best empirical-open trade-off, which means the model can ensure the classification quality of IND and OOD detection effectively. The only empirical risk or open space risk can not make the model achieve better results. In this paper, we explicit the optimization objective of OOD intent classification. We analyze the limitation of existing methods and propose a simple yet effective method to learn discriminative semantic features. Our approach pulls together k-nearest neighbors of IND intents and pushes apart them from different class samples to better reduce both empirical risk and open space risk. Extensive experiments conducted on four challenging datasets show our approach achieves consistent improvements without restrictions on feature distribution. We thank Demin Song for helpful discussion about experiment. We would also like to thank Hang Yan and TianXiang Sun for helpful suggestion for this paper. This work was supported by the National Key Research and Development Program of China (No. 2020AAA0108702) and National Natural Science Foundation of China (No. 62022027). The datasets used in all experiments are derived from previously published scientific papers, and to our knowledge, there are no privacy or ethical issues.
abstain abstain abstain objective objective method method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain other other other other other other other other other other other other other other other other other other method abstain other method method abstain abstain method method method method abstain abstain method abstain method abstain method method method method abstain abstain method method abstain abstain method method method result abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain result abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain result abstain method other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain objective objective method result other other other method