Albert Y.S. Lam Department of Computing, The Hong Kong Polytechnic University, Hong Kong S.A.R. 1 University of California, San Diego 2 Abstract New intent discovery aims to uncover novel intent categories from user utterances to expand the set of supported intent classes. It is a critical task for the development and ser-vice expansion of a practical dialogue system. Despite its importance, this problem remains under-explored in the literature. Existing approaches typically rely on a large amount of labeled utterances and employ pseudo-labeling methods for representation learning and clustering, which are label-intensive, inefficient, and inaccurate. In this paper, we provide new solutions to two important research questions for new intent discovery: (1) how to learn semantic utterance representations and (2) how to better cluster utterances. Particularly, we first propose a multi-task pre-training strategy to leverage rich unlabeled data along with external labeled data for representation learning. Then, we design a new contrastive loss to exploit self-supervisory signals in unlabeled data for clustering. Extensive experiments on three intent recognition benchmarks demonstrate the high effectiveness of our proposed method, which outperforms state-of-the-art methods by a large margin in both unsupervised and semi-supervised scenarios. The source code will be available at https://github. com/zhang-yu-wei/MTP-CLNN . Why Study New Intent Discovery (NID)? Recent years have witnessed the rapid growth of conversational AI applications. To design a natural language understanding system, a set of expected customer intentions are collected beforehand to train an intent recognition model. However, the pre-defined intents cannot fully meet customer needs. This implies the necessity of expanding the intent recognition model by repeatedly integrating new intents discovered from unlabeled user utterances (Fig. 1). To reduce the effort in manually identifying unknown intents from a mass of utterances, previous works commonly employ clustering algorithms to group utterances of similar intents (Cheung and Li, 2012; Hakkani-Tr et al., 2015; Padmasundari, 2018). The cluster assignments thereafter can either be directly used as new intent labels or as heuristics for faster annotations. Research Questions (RQ) and Challenges. Current study of NID centers around two basic research questions: 1) How to learn semantic utterance representations to provide proper cues for clustering? 2) How to better cluster the utterances? The study of the two questions are often interwoven in existing research. Utterances can be represented according to different aspects such as the style of language, the related topics, or even the length of sentences. It is important to learn semantic utterance representations to provide proper cues for clustering. Simply applying a vanilla pre-trained language model (PLM) to generate utterance representations is not a viable solution, which leads to poor performance on NID as shown by the experimental results in Section 4.2. Some recent works proposed to use labeled utterances of known intents 256 for representation learning (Forman et al., 2015; Haponchyk et al., 2018; Lin et al., 2020; Zhang et al., 2021c; Haponchyk and Moschitti, 2021), but they require a substantial amount of known intents and sufficient labeled utterances of each intent, which are not always available especially at the early development stage of a dialogue system. Further, pseudo-labeling approaches are often exploited to generate supervision signals for representation learning and clustering. For example, Lin et al. (2020) finetune a PLM with an utterance similarity prediction task on labeled utterances to guide the training of unlabeled data with pseudo-labels. Zhang et al. (2021c) adopt a deep clustering method (Caron et al., 2018) that uses k -means clustering to produce pseudo-labels. However, pseudo-labels are often noisy and can lead to error propagation. Our Solutions. In this work, we propose a simple yet effective solution for each research question. Solution to RQ 1: multi-task pre-training. We propose a multi-task pre-training strategy that takes advantage of both external data and internal data for representation learning. Specifically, we leverage publicly available, high-quality intent detection datasets, following Zhang et al. (2021d), as well as the provided labeled and unlabeled utterances in the current domain, to fine-tune a pre-trained PLM to learn task-specific utterance representations for NID. The multi-task learning strategy enables knowledge transfer from general intent detection tasks and adaptation to a specific application domain. Solution to RQ 2: contrastive learning with nearest neighbors. We propose to use a contrastive loss to produce compact clusters, which is motivated by the recent success of contrastive learning in both computer vision (Bachman et al., 2019; He et al., 2019; Chen et al., 2020; Khosla et al., 2020) and natural language processing (Gunel et al., 2021; Gao et al., 2021; Yan et al., 2021). Contrastive learning usually maximizes the agreement between different views of the same example and minimize that between different examples. However, the commonly used instance discrimination task may push away false negatives and hurts the clustering performance. Inspired by a recent work in computer vision (Van Gansbeke et al., 2020), we introduce neighborhood relationship to customize the contrastive loss for clustering in both unsupervised (i.e., without any labeled utterances of known intents) and semi-supervised scenarios. Intuitively, in a semantic feature space, neighboring utterances should have a similar intent, and pulling together neighboring samples makes clusters more compact. Our main contributions are three-fold. We show that our proposed multi-task pretraining method already leads to large performance gains over state-of-the-art models for both unsupervised and semi-supervised NID. We propose a self-supervised clustering method for NID by incorporating neighborhood relationship into the contrastive learning objective, which further boosts performance. We conduct extensive experiments and ablation studies on three benchmark datasets to verify the effectiveness of our methods. New Intent Discovery. The study of NID is still in an early stage. Pioneering works focus on unsupervised clustering methods. Shi et al. (2018) leveraged auto-encoder to extract features. Perkins and Yang (2019) considered the context of an utterance in a conversation. Chatterjee and Sengupta (2020) proposed to improve density-based models. Some recent works (Haponchyk et al., 2018; Haponchyk and Moschitti, 2021) studied supervised clustering algorithms for intent labeling, yet it can not handle new intents. Another line of works (Forman et al., 2015; Lin et al., 2020; Zhang et al., 2021c) investigated a more practical case where some known intents are provided to support the discovery of unknown intents, which is often referred to as semi-supervised NID. To tackle semi-supervised NID, Lin et al. (2020) proposed to first perform supervised training on known intents with a sentence similarity task and then use pseudo labeling on unlabeled utterances to learn a better embedding space. Zhang et al. (2021c) proposed to first pre-train on known intents and then perform k -means clustering to assign pseudo labels on unlabeled data for representation learning following Deep Clustering (Caron et al., 2018). They also proposed to align clusters to accelerate the learning of top layers. Another approach is to first classify the utterances as known and unknown and then uncover new intents with the unknown utterances (Vedula et al., 2020; Zhang et al., 2021b). Hence, it relies on accurate classification in the first stage. learning and a contrastive learning method for clustering. In contrast to previous methods that rely on ample annotated data in the current domain for pre-training, our method can be used in an unsupervised setting and work well in data-scarce scenarios (Section 4.3). Pre-training for Intent Recognition. Despite the effectiveness of large-scale pre-trained language models (Radford and Narasimhan, 2018; Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020), the inherent mismatch in linguistic behavior between the pre-training datasets and dialogues encourages the research of continual pre-training on dialogue corpus. Most previous works proposed to pre-train on open domain dialogues in a self-supervised manner (Mehri et al., 2020; Wu et al., 2020; Henderson et al., 2020; Hosseini-Asl et al., 2020). Recently, several works pointed out that pre-training with relavant tasks can be effective for intent recognition. For example, Zhang et al. (2020) formulated intent recognition as a sentence similarity task and pre-trained on natural language inference (NLI) datasets. Vulic et al. (2021); Zhang et al. (2021e) pre-trained with a contrastive loss on intent detection tasks. Our multi-task pre-training method is inspired from Zhang et al. (2021d) which leverages publicly available intent datasets and unlabeled data in the current domain for pre-training to improve the performance of few-shot intent detection. However, we argue that the method is more suitable to be applied for NID due to the natural existence of unlabeled utterances. Contrastive Representation Learning. Contrastive learning has shown promising results in computer vision (Bachman et al., 2019; Chen et al., 2020; He et al., 2019; Khosla et al., 2020) and gained popularity in natural language processing. Some recent works used unsupervised contrastive learning to learn sentence embeddings (Gao et al., 2021; Yan et al., 2021; Kim et al., 2021; Giorgi et al., 2021). Specifically, Gao et al. (2021); Yan et al. (2021) showed that contrastive loss can avoid an anisotropic embedding space. Kim et al. (2021) proposed a self-guided contrastive training to improve the quality of BERT representations. Giorgi et al. (2021) proposed to pre-train a universal sentence encoder by contrasting a randomly sampled text segment from nearby sentences. Zhang et al. (2021e) demonstrated that self-supervised contrastive pre-training and supervised contrastive fine-tuning can benefit few-shot intent recognition. Zhang et al. (2021a) showed that combining a contrastive loss with a clustering objective can improve short text clustering. Our proposed contrastive loss is tailored for clustering, which encourages utterances with similar semantics to group together and avoids pushing away false negatives as in the conventional contrastive loss. Problem Statement. To develop an intent recognition model, we usually prepare a set of expected intents C k along with a few annotated utterances D labeledknown = { ( x i , y i ) | y i  C k } for each intent. After deployed, the system will encounter utterances D unlabeled = { x i | y i  {C k , C u }} from both predefined (known) intents C k and unknown intents C u . The aim of new intent discovery (NID) is to identify the emerging intents C u in D unlabeled . NID can be viewed as a direct extension of out-of-distribution (OOD) detection, where we not only need to identify OOD examples but also discover the underlying clusters. NID is also different from zero-shot learning in that we do not presume access to any kind of class information during training. In this work, we consider both unsupervised and semi-supervised NID, which are distinguished by the existence of D labeledknown , following Zhang et al. (2021c). Overview of Our Approach. As shown in Fig. 2, we propose a two-stage framework that addresses the research questions mentioned in Sec. 1. In the first stage, we perform multi-task pre-training (MTP) that jointly optimizes a cross-entropy loss on external labeled data and a self-supervised loss on target unlabeled data (Sec. 3.1). In the second stage, we first mine topK nearest neighbors of each training instance in the embedding space and then perform contrastive learning with nearest neighbors (CLNN) (Sec. 3.2). After training, we employ a simple non-parametric clustering algorithm to obtain clustering results. We propose a multi-task pre-training objective that combines a classification task on external data from publicly available intent detection datasets and a self-supervised learning task on internal data from the current domain. Different from previous works (Lin et al., 2020; Zhang et al., 2021c), our pre-training method does not rely on annotated data ( D labeledknown ) from the current domain and hence can 258 Figure 2: The left part shows the overall workflow of our method where the training order is indicated by the red arrow. Specifically, we first initialize the model with a pre-trained BERT encoder (Devlin et al., 2019). Then, we employ a joint pre-training loss as in Zhang et al. (2021d). The loss consists of a cross-entropy loss on external labeled data and a masked language modelling (MLM) loss on all available data from the current domain: L stg1 = L ce ( D labeled external ;  ) (cid:124) (cid:123)(cid:122) (cid:125) supervised + L mlm ( D all internal ;  ) (cid:124) (cid:123)(cid:122) (cid:125) self-supervised , (1) where  are model parameters. For the supervised classification task, we leverage an external public intent dataset with diverse domains (e.g., CLINC150 (Larson et al., 2019)), denoted as D labeled external , following Zhang et al. (2021d). For the self-supervised MLM task, we use all available data (labeled or unlabeled) from the current domain, denoted as D all internal . Intuitively, the classification task aims to learn general knowledge of intent recognition with annotated utterances in external intent datasets, while the self-supervised task learns domain-specific semantics with utterances collected in the current domain. Together, they enable learning semantic utterance representations to provide proper cues for the subsequent clustering task. As will be shown in Sec. 4.3, both tasks are essential for NID. For semi-supervised NID, we can further utilize the annotated data in the current domain to conduct continual pre-training, by replacing D labeledexternal in Eq. 1 to D labeledknown . This step is not included in unsupervised NID. In the second stage, we propose a contrastive learning objective that pulls together neighboring instances and pushes away distant ones in the embedding space to learn compact representations for clustering. Concretely, we first encode the utterances with the pre-trained model from stage 1. Then, for each utterance x i , we search for its top-K nearest neighbors in the embedding space using inner product as distance metric to form a neighborhood N i . The utterances in N i are supposed to share a similar intent as x i . During training, we sample a minibatch of utterances B = { x i } Mi =1 . For each utterance x i  B , we uniformly sample one neighbor x  i from its neighborhood N i . We then use data augmentation to generate  x i and  x  i for x i and x  i respectively. Here, we treat  x i and  x  i as two views of x i , which form a positive pair. We then obtain an augmented batch B  = {  x i ,  x  i } Mi =1 with all the generated samples. To compute contrastive loss, we construct an adjacency matrix A  for B  , which is a 2 M  2 M binary matrix where 1 indicates positive relation (either being neighbors or having the same intent label in semi-supervised NID) and 0 indicates negative relation. Hence, we can write the contrastive loss as: l i =  1 |C i | (cid:88) j C i log exp( sim ( h i ,  h j ) / ) (cid:80) 2 Mk  = i exp( sim ( h i ,  h k ) / ) , (2) 259 L stg2 = 1 2 M 2 M (cid:88) i =1 l i , (3) where C i  { A  i,j = 1 | j  { 1 , ..., 2 M }} denotes the set of instances having positive relation with  x i and |C i | is the cardinality. h i is the embedding for utterance  x i . is the temperature parameter. sim (  ,  ) is a similarity function (e.g., dot product) on a pair of normalized feature vectors. During training, the neighborhood will be updated every few epochs. We implement the contrastive loss following Khosla et al. (2020). Notice that the main difference between Eq. 2 and conventional contrastive loss is how we construct the set of positive instances C i . Conventional contrastive loss can be regarded as a special case of Eq. 2 with neighborhood size K = 0 and the same instance is augmented twice to form a positive pair (Chen et al., 2020). After contrastive learning, a non-parametric clustering algorithm such as k means can be applied to obtain cluster assignments. Data Augmentation. Strong data augmentation has been shown to be beneficial in contrastive learning (Chen et al., 2020). We find that it is inefficient to directly apply existing data augmentation methods such as EDA (Wei and Zou, 2019), which are designed for general sentence embedding. We observe that the intent of an utterance can be expressed by only a small subset of words such as suggest restaurant or book a flight. While it is hard to identify the keywords for an unlabeled utterance, randomly replacing a small amount of tokens in it with some random tokens from the library will not affect intent semantics much. This approach works well in our experiments (See Table 5 RTR). Advantages of CLNN. By introducing the notion of neighborhood relationship in contrastive learning, CLNN can 1) pull together similar instances and push away dissimilar ones to obtain more compact clusters; 2) utilize proximity in the embedding space rather than assigning noisy pseudo-labels (Van Gansbeke et al., 2020); 3) directly optimize in the feature space rather than clustering logits as in Van Gansbeke et al. (2020), which has been proven to be more effective by Rebuffi et al. (2020); and 4) naturally incorporate known intents with the adjacency matrix. Datasets. We evaluate our proposed method on three popular intent recognition benchmarks. BANKING (Casanueva et al., 2020) is a fine-grained dataset with 77 intents collected from banking dialogues, StackOverflow (Xu et al., 2015) is a large scale dataset collected from online queries, M-CID (Arora et al., 2020) is a smaller dataset collected for Covid-19 services. We choose CLINC150 (Larson et al., 2019) as our external public intent dataset in stage 1 due to its high-quality annotations and coverage of diverse domains. The dataset statistics are summarized in Table 1. We use the same splits of BANKING and StackOverflow as in Zhang et al. (2021b). Details about dataset splitting are provided in the Appendix. Experimental Setup. We evaluate our proposed method on both unsupervised and semi-supervised NID. Notice that in unsupervised NID, no labeled utterances from the current domain are provided. For clarity, we define two variables. The proportion of known intents is defined as |C k | / ( |C k | + |C u | ) and referred to as  known class ratio (KCR) , and the proportion of labeled examples for each known intent is denoted as  labeled ratio (LAR) . The labeled data are randomly sampled from the original training set. Notice that, KCR = 0 means unsupervised NID, and KCR > 0 means semi-supervised NID. In the following sections, we provide experimental results for both unsupervised NID and semi-supervised NID with KCR = { 25% , 50% , 75% } and LAR = { 10% , 50% } . Evaluation Metrics. We adopt three popular evaluation metrics for clustering: normalized mutual information (NMI), adjusted rand index (ARI), and accuracy (ACC). Baselines and Model Variants. We summarize the baselines compared in our experiments for both unsupervised and semi-supervised NID. Our 260 BANKING StackOverflow M-CID Methods NMI ARI ACC NMI ARI ACC NMI ARI ACC unsupervised GloVe-KM 48.75 12.74 27.92 21.79 4.54 24.26 46.40 35.57 46.99 GloVe-AG 52.76 14.41 31.18 23.45 4.85 24.48 51.23 32.57 42.35 SAE-KM 60.12 24.00 37.38 48.72 23.36 37.16 51.03 43.51 52.95 SAE-DEC 62.92 25.68 39.35 61.32 21.17 57.09 50.69 44.52 53.07 SAE-DCN 62.94 25.69 39.36 61.34 34.98 57.09 50.69 44.52 53.07 BERT-KM 36.38 5.38 16.27 11.60 1.60 13.85 37.37 14.02 33.81 MTP (Ours) 77.32 47.33 57.99 63.85 48.71 66.18 72.40 53.04 68.94 MTP-CLNN (Ours) 81.80 55.75 65.90 78.71 67.63 81.43 79.95 66.71 79.14 Table 2: Performance on unsupervised NID. implementation is based on Zhang et al. (2021b). Unsupervised baselines. (1) GloVe-KM and (2) GloVe-AG are based on GloVe (Pen-nington et al., 2014) embeddings and then evaluated with k -means (MacQueen et al., 1967) or agglomerative clustering (Gowda, 1984) respectively. (3) BERT-KM applies k means on BERT embeddings. (4) SAE-KM 1 For fair comparison, the baselines are re-run with TEX-TOIR: https://github.com/thuiar/TEXTOIR , and hence some results are different from those reported in Lin et al. (2020); Zhang et al. (2021c). adopts k -means on embeddings of stacked auto-encoder. (5) Deep Embedding Clustering (SAE-DEC) (Xie et al., 2016) and (6) Deep Clustering Network (SAE-DCN) (Yang et al., 2017) are unsupervised clustering methods based on stacked auto-encoder. Semi-supervised baselines. (1) BERT-KCL (Hsu et al., 2018) and (2) BERT-MCL (Hsu et al., 2019) employs pairwise similarity task for semi-supervised clustering. (3) BERT-DTC (Han et al., 2019) extends DEC into semi-supervised scenario. (4) CDAC+ (Lin 261 et al., 2020) employs a pseudo-labeling process. (5) Deep Aligned Clustering (DAC) (Zhang et al., 2021c) improves Deep Clustering (Caron et al., 2018) by aligning clusters between iterations. Our model variants include MTP and MTP-CLNN, which correspond to applying k means on utterance representations learned in stage 1 and stage 2 respectively. Further, we continue to train a DAC model on top of MTP to form a stronger baseline MTP-DAC for semi-supervised NID. Implementation. We take pre-trained bert-base-uncased model from Wolf et al. (2019) 2 as our base model and we use the [CLS] token as the BERT representation. For MTP, we first train until convergence on the external dataset, and then when training on D labeledknown , we use a development set to validate early-stopping with a patience of 20 epochs following Zhang et al. (2021c). For contrastive learning, we project a 768-d BERT embedding to an 128-d vector with a two-layer MLP and set the temperature as 0 . 07 . For mining nearest neighbors, we use the inner product method 2 https://github.com/huggingface/ transformers provided by Johnson et al. (2017) 3 . We set neighborhood size K = 50 for BANKING and M-CID, and K = 500 for StackOverflow, since we empirically find that the optimal K should be roughly half of the average size of the training set for each class (see Section 4.4). The neighborhood is updated every 5 epochs. For data augmentation, the random token replacement probability is set to 0 . 25 . For model optimization, we use the AdamW provided by Wolf et al. (2019). In stage 1, the learning rate is set to 5 e  5 . In stage 2, the learning rate is set to 1 e  5 for BANKING and M-CID, and 1 e  6 for StackOverflow. The batch sizes are chosen based on available GPU memory. All the experiments are conducted on a single RTX-3090 and averaged over 10 different seeds. More details are provided in the Appendix. Unsupervised NID. We show the results for unsupervised NID in Table 2. First, comparing the performance of BERT-KM with GloVe-KM and SAE-KM, we observe that BERT embedding performs worse on NID even though it achieves better performance on NLP benchmarks such as GLUE, which manifests learning task-specific knowledge is important for NID. Second, our proposed pre-3 https://github.com/facebookresearch/ faiss 262 training method MTP improves upon baselines by a large margin. Take the NMI score of BANKING for example, MTP outperforms the strongest baseline SAE-DCN by 14 . 38% , which demonstrates the effectiveness of exploiting both external public datasets and unlabeled internal utterances. Furthermore, MTP-CLNN improves upon MTP by around 5% in NMI, 10% in ARI, and 10% in ACC across different datasets. Semi-supervised NID. The results for semi-supervised NID are shown in Table 3. First, MTP significantly outperforms the strongest baseline DAC in all settings. For instance, on M-CID, MTP achieves 22 . 57% improvement over DAC in NMI. Moreover, MTP is less sensitive to the proportion of labeled classes. From KCR = 75% to KCR = 25% on M-CID, MTP only drops 8 . 55% in NMI, as opposed to about 21 . 58% for DAC. The less performance decrease indicates that our pretraining method is much more label-efficient. Furthermore, with our proposed contrastive learning, MTP-CLNN consistently outperforms MTP and the combined baseline MTP-DAC. Take BANKING with KCR = 25% for example, MTP-CLNN improves upon MTP by 4 . 11% in NMI while surpassing MTP-DAC by 2 . 63% . A similar trend can be observed when LAR = 50% , and we provide the results in the Appendix. Visualization. In Fig. 3, we show the t-SNE visualization of clusters with embeddings learned by two strongest baselines and our methods. It clearly shows the advantage of our methods, which can produce more compact clusters. Results on other datasets can be found in the Appendix. To further illustrate the effectiveness of MTP, we conduct two ablation studies in this section. First, we compare MTP with the pre-training method employed in Zhang et al. (2021c), where only internal labeled data are utilized for supervised pretraining (denoted as SUP). 4 In Fig. 4, we show the results of both pre-training methods combined with CLNN with different proportions of known classes. Notice that when KCR = 0 there is no pretraining at all for SUP-CLNN. It can be seen that MTP-CLNN consistently outperforms SUP-CLNN. Furthermore, the performance gap increases while KCR decreases, and the largest gap is achieved 4 Notice that we make a simple modification to their pretraining to optimize the entire model rather than the last few layers for fair comparison. when KCR = 0 . This shows the high effectiveness of our method in data-scarce scenarios. Second, we decompose MTP into two parts: supervised pre-training on external public data (PUB) and self-supervised pre-training on internal unlabeled data (MLM). We report the results of the two pre-training methods combined with CLNN as well as MTP in Table 4. We can easily conclude that either PUB or MLM is indispensable and multi-task pre-training is beneficial. Number of Nearest Neighbors. We conduct an ablation study on neighborhood size K in Fig. 5. We can make two main observations. First, although the performance of MTP-CLNN varies with different K , it still significantly outperforms MTP (dashed horizontal line) for a wide range of K . For example, MTP-CLNN is still better than MTP when K = 50 on StackOverflow or K = 200 on BANKING. Second, despite the difficulty to search for K with only unlabeled data, we empirically find an effective estimation method, i.e. to choose K as half of the average size of the training set for each class 5 . It can be seen that the estimated K  60 on BANKING and K  40 on M-CID (vertical dashed lines) lie in the optimal regions, which shows the effectiveness of our empirical estimation method. Exploration of Data Augmentation. We compare Random Token Replacement (RTR) used in our experiments with other methods. For instance, dropout is applied on embeddings to provide data augmentation in Gao et al. (2021), randomly shuffling the order of input tokens is proven to be effective in Yan et al. (2021), and EDA (Wei and Zou, 2019) is often applied in text classification. Furthermore, we compare with a Stop-words Replacement (SWR) variant that only replaces the stop-words with other random stop-words so it minimally af-5 We presume prior knowledge of the number of clusters. There are some off-the-shelf methods that can be directly applied in the embedding space to determine the optimal number of clusters (Zhang et al., 2021c). fects the intents of utterances. The results in Table 5 demonstrate that (1) RTR and SWR consistently outperform others, which verifies our hypothesis in Section 3.2. (2) Surprisingly, RTR and SWR perform on par with each other. For simplicity, we only report the results with RTR in the main experiments. We have provided simple and effective solutions for two fundamental research questions for new intent discovery (NID): (1) how to learn better utterance representations to provide proper cues for clustering and (2) how to better cluster utterances in the representation space. In the first stage, we use a multi-task pre-training strategy to exploit both external and internal data for representation learning. In the second stage, we perform contrastive learning with mined nearest neighbors to exploit self-supervisory signals in the representation space. Extensive experiments on three intent recognition benchmarks show that our approach can significantly improve the performance of NID in both unsupervised and semi-supervised scenarios. There are two limitations of this work. (1) We have only evaluated on balanced data. However, in real-world applications, most datasets are highly imbalanced. (2) The discovered clusters lack interpretability. Our clustering method can only assign a cluster label to each unlabeled utterance but cannot generate a valid intent name for each cluster. We would like to thank the anonymous reviewers for their valuable comments. This research was supported by the grants of HK ITF UIM/377 and PolyU DaSAIL project P0030935 funded by RGC.
abstain abstain abstain abstain abstain objective objective objective objective other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain objective abstain objective method abstain abstain other abstain abstain abstain abstain objective objective objective result other other other other other other other other other other other other other other objective other other other other other other abstain method other other other other other other other other objective abstain method abstain abstain method method method method method method method method method abstain method other method method abstain abstain abstain method abstain abstain abstain abstain abstain method method method method abstain method other abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain abstain abstain result abstain abstain result abstain result abstain result abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain result objective objective method result abstain method abstain abstain method other other