Recent research considers few-shot intent detection as a meta-learning problem: the model is learning to learn from a consecutive set of small tasks named episodes. In this work, we propose PROTAUGMENT , a meta-learning algorithm for short texts classification applied to the intent detection task. PROTAUGMENT is a novel extension of Prototypical Networks (Snell et al., 2017) that limits over-fitting on the bias introduced by the few-shots classification objective at each episode. It relies on diverse paraphrasing: a conditional language model is first fine-tuned for paraphrasing, and diversity is later introduced at the decoding stage at each meta-learning episode. The diverse paraphrasing is unsupervised as it is applied to unlabelled data and then fueled to the Prototypical Network training objective as a consistency loss. PROTAUGMENT is the state-of-the-art method for intent detection meta-learning, at no extra labeling efforts and without the need to fine-tune a conditional language model on a given application domain. Intent detection, a sub-field of text classification, involves classifying user-generated short-texts into intent classes, usually for conversational agents applications (Casanueva et al., 2020). Since conversational agent applications are domain-specific, intent detection is a challenging task because of labeled data scarcity and the number of classes (in-tents) it usually involves (Dopierre et al., 2020). As a consequence, recent research (Snell et al., 2017; Ren et al., 2018) considers few-shot intent detection as a meta-learning problem: the model is trained to classify user utterances from a consecutive set of small tasks named episodes. Each episode contains a limited number of C classes alongside a limited number of K labeled data for each of the C classes  this is usually referred to as a C -way K -shots setup. At test time, the algorithm is evaluated on classes that were not seen during training. That is the reason why meta-learning is sometimes referred to as learning to learn : it mimics human abilities to learn iteratively from different and small tasks. Meta-learning has successfully been applied to a wide set of NLP tasks: hypernym detection (Yu et al., 2020), low resource machine translation (Gu et al., 2018), machine understanding tasks (Dou et al., 2019) or structured query generation (Huang et al., 2018). Most meta-learning algorithms (Section 2) were developed in the course of the last 5 years. It has recently been empirically demonstrated that comparative studies in follow-up papers of (Snell et al., 2017) are debatable  for short texts classification  because of the two following main issues (Dopierre et al., 2021). First, comparative studies involve simple and limited datasets in terms of number and separability of classes (SNIPS (Coucke et al., 2018), a very popular dataset, includes only 7 classes, with the current best model performing over 99% accuracy (Cao et al., 2020)). Second, as we further better understand (Niven and Kao, 2019), fine-tune (Liu et al., 2019b; Hao et al., 2020) and refine (Khetan and Karnin, 2020) BERT-derived models, it is not clear if the different meta-learning frameworks can be considered state-of-the-art due to their architecture or due to the improvements of available text encoders at the time of conception. (Dopierre et al., 2021) concludes that Prototypical Networks (Snell et al., 2017) (that were using LSTM-based text encoders when introduced in NLP) are actually the state-of-the-art for intent detection when equipped with a fine-tuned BERT text encoder model. Ultimately, improving Prototypical Networks have therefore been proven to be a very challenging task in reality. A cornerstone challenge is that meta-learning models can easily overfit on the biased distribution introduced by a few training examples (Yang et al., 2021). In order to prevent overfitting and inspired by (Xie et al., 2020), we introduce an unsupervised diverse paraphrasing loss in the Prototypical Networks framework. A key idea is consistency learning: by augmenting unlabeled user utterances, PROTAUGMENT enforce a more robust text representation learning. Unfortunately, back-translation is a poor data augmentation strategy for short-texts: neural machine translation provides very similar (if not the same) sentences to the original ones, which hinders its ability to provide diverse augmentations (Section 5.3). Consequently, in this work, we transfer a denoising autoencoder pre-trained on the sequence-to-sequence task (Lewis et al., 2020) to the paraphrase generation task and then use it to generate paraphrases. As fine-tuning is very efficient for such a model, it is not easy to optimize it for diverse paraphrasing. (Goyal and Durrett, 2020) presents an approach for diverse paraphrasing that reorders the original sentence to guide the conditional language model to generate diverse sentences. The diversity in that work is provided by the reordering of the elements, which surprisingly affects the attention mechanism. In (Liu et al., 2020), expression diversity is part of the unsupervised paraphrasing system supported by simulated annealing. Both approaches imply domain transfer, and consequently, as many diverse paraphrasing models to maintain as the number of considered application domains, which do not scale very well. In this work, we instead introduce diversity in the downstream decoding algorithm used for paraphrase generation. Diverse decoding methods are mostly extensions to the beam search algorithm, including noise-based algorithms (Cho, 2016), iterative beam search (Kulikov et al., 2019), clustered beam search (Tam, 2020) and diverse beam search (Vijayakumar et al., 2018). There is no clear optimal solution, the choice is task-specific and dependent on one's tolerance for lower quality outputs as a diversity/fluency trade-off (Ippolito et al., 2019). While diverse beam search allows controlling the diversity/fluency trade-off partially, we further demonstrate that adding constraints to diverse beam search in order to generate tokens not seen in the input sentence (that is, constrained diverse beam search ) is a simple yet powerful strategy to further improve the diversity of the paraphrases. Paired with paraphrasing user utterances and its consistency loss incorporated in Prototypical networks, our model is the best method for intent detection meta-learning on 4 public datasets, with neither extra labeling efforts nor domain-specific conditional language model fine-tuning. We also show that PROTAUGMENT , having access to only 10 samples of each class of the training data, still significantly outperforms a Prototypical Network which is given access to all samples of the same training data. Past works on meta-learning for classification tasks investigate how to best predict a query point's class at an episode scale. This process is bounded to the set of the C classes considered in a given episode. Matching Networks (Vinyals et al., 2016) predict the class of a query point as the average cosine distance between the query vector and all support vectors for each class. Prototypical Networks (Snell et al., 2017) extend Matching Networks: after obtaining support vectors from the encoder, a class prototype is produced via a class-wise vector averaging operation. All query points are then predicted with respect to their distance (cosine or euclidean) to all prototypes. Like Prototypical Networks, Relation Networks (Sung et al., 2018) emerged from Computer Vision application and were later successfully applied to NLP (Zhang et al., 2018). They introduce a relation module, which captures the relationship between data points: instead of using a pre-defined distance (euclidean or cosine most of the time), this approach allows such networks to learn this metric by themselves. This is achieved using either a shallow feed-forward sub-network or a Neural Tensor Layer relation module (Socher et al., 2013) (intermediate learnable matrices). Another extension to Prototypical Networks is provided in (Ren et al., 2018). Unlabeled data are incorporated using two distinct approaches: i) taking unlabeled data from the same classes as the episode or ii) using any unlabeled data and incorporating both a distractor cluster and masking strategy to minimize the impact of distant unlabeled points. The first approach is unrealistic for meta-learning, as it implies knowing the unlabeled data class. The second method assumes that all the noise is centered around a single distractor cluster and introduces an additional hyperparameter for masking  which is hardly fine-tuneable for small few-shot datasets. Meta-learning algorithms are trained using a specific procedure made of consecutive episodes. Let C ep be the set of C classes sampled for the current training episode, such as C ep  C train , where C train is the set of all classes available for training. We note C test , the set of classes used for testing, with C train  C test =  . Each class c  C ep comes with K labeled samples, used as support. The set of C  K samples are usually referred to as S , the support set, so that S = { ( x 1 , y 1 ) , . . . , ( x C  K , y C  K ) } . We denote S c the set of support examples labeled with class c . Each episode comes with a query set Q , which serves as the episode-scale optimization  the model parameters are updated based on the prediction loss on Q , given S as an input. Q c is the set of query examples labeled with class c . In prototypical networks, each class is mapped to a representative point, called prototype . Each sample is first encoded into a vector using an embedding function f  with learnable parameters   this is the function we want to optimize. Using these embeddings, we compute each prototype p c , c  C ep as the mean vector of embedded support points belonging to the class c , as described in Equation 1. p c = 1 K (cid:88) ( x i ,y i )  S c f  ( x i ) (1) Given those prototypes and a distance function d , prototypical networks assign a label to a query point by computing the softmax over distances between this point's embedding and the prototypes, as in Equation 2. In the original paper, (Snell et al., 2017) use the euclidean distance and we also observed consistent slightly worse results with the cosine distance. The supervised loss function  L is the average negative log-probability of the correct class assignments for all query points. At test time, episodes are created using classes from C test , and accuracy is measured as the query points assignments, given prototypes derived from the support points. In this section, we present our semi-supervised approach PROTAUGMENT . Along with the labeled data randomly chosen at each episode, this approach uses U unlabeled data randomly drawn from the whole dataset  that is, data from training, validation, and test labels. We first do a data augmentation step from this unlabeled data, where we obtain M paraphrases for each unlabeled sentence. The m th paraphrase of x will be denoted  x m . Then, given unlabeled data and their paraphrases, we compute a fully unsupervised loss. Finally, we combine both the supervised loss  L (the Prototypical Network loss using labeled data) and unsupervised loss (denoted  L ) and run back-propagation to update the model's parameters. The BART (Lewis et al., 2020) model is a Transformer-based neural machine translation architecture that is trained to remove artificially corrupted text from the input thanks to an autoencoder architecture. While it is trained to reconstruct the original noised input, it can be fine-tuned for task-specific conditional generation by minimizing the cross-entropy loss on new training input-output pairs (Bevilacqua et al., 2020). In PROTAUGMENT , we fine-tune a pre-trained BART model on the paraphrasing task. The paraphrase sentence pairs we use for this task are taken from 3 different paraphrase detection datasets 1 : Quora (Sharma et al., 2019), MSR (Zhao and Wang, 2010), and Google PAWS-Wiki (Yang et al., 2019; Zhang et al., 2019). Those datasets have different sizes, and the largest one  Quora  consist of 149,263 pairs of duplicate questions. To balance turns of sentences (questions/non questions paraphrases), 50% of our fine-tuning paraphrase datasets is made of Quora, 5 . 6% of MSR and 44 . 4% PAWS-Wiki. This yields 94,702 sentence pairs to train the model on the paraphrasing task. We include both code and data on our github repository 2 . Using this fine-trained paraphrasing model, we can generate paraphrases of unlabeled sentences, hopefully having paraphrases representing the same intents as the original sentences. To add some diversity in the generated paraphrases, we use Di-1 we take only pairs that are paraphrases of each other since these are paraphrase detection datasets 2 https://github.com/tdopierre/ProtAugment Loss Annealing Scheduler Paraphrase training Quora MSR PAWS Paraphrase Datasets Cw a y K-shot (Support) Query Episode #1 ... verse Beam Search (DBS) instead of the regular Beam Search. As Vijayakumar et al. (2018) has shown in the original paper, adding a dissimilarity term during the decoding step helps the model produce sequences that are quite far from each other while still retaining the same meaning. The next section describes how we constrained this decoding to enforce even more diversity among generated paraphrases in PROTAUGMENT . While DBS enforces diversity between the generated sentences, it does not ensure diversity between the generated paraphrases and the original sentences. It was formerly designed for tasks that do not need this diversity with the original sentence (translation, image captioning, question gen-eration). To enforce that our generated paraphrases are diverse enough, we further constraint DBS by forbidding using parts of the original sentences. In the following paragraphs, we introduce two forbidding strategies. forbidden at the generation step. The goal here is to force the model to use different words in the generated sentences than it saw in the original sentences. Each word of the input sentence is randomly masked using a probability p mask . The underlying assumption is that forbidding tokens at the beginning of a sentence with a higher probability than the end of the sentence may have a greater impact on the beam search algorithm. Indeed, as the decoding is a conditional task based on prior generated tokens, masking the first tokens may significantly impact diversity. We therefore introduce two additional variants: one where we put more probability on the first tokens and the reverse where there is more weight in the last tokens. To ensure that all three variants mask the same amount of tokens on average, we ensure the area under the curve of the three probability functions are equal to a fixed value noted p mask . Bi-gram Masking Another strategy we consider is to prevent the paraphrasing model from generating the same bi-grams as in the original sentence. This time, we are not masking any single word but forcing the model to change the sentence's structure, which will, hopefully, increase the diversity of the generated paraphrases. After generating paraphrases for each unlabeled sentence, we create unlabeled prototypes. For each unlabeled sentence x u  U , we derive the unlabeled prototype p x u as the average embedding of the paraphrases of x u (Equation 3). After obtaining the unlabeled prototypes, we compute the distances between all unlabeled samples and all unlabeled prototypes. Given such distances, we model the probability of each unlabeled sample being assigned to each unlabeled prototype (Equation 4), as in the supervised part of the Prototypical Networks  except this time, it is fully unsupervised. This probability should be close to 1 between an unlabeled sample and its associated unlabeled prototype and close to 0 otherwise. Given assign probabilities between unlabeled samples and unlabeled prototypes, we can compute a fully unsupervised cross-entropy loss  L , training the model to bring each sentence closer to its augmentations' prototype and further from the prototypes of other unlabeled sentences. Recall that f  is the embedding function with  as learnable parameters (Section 3.2). After obtaining both supervised loss  L and unsupervised loss  L , we combine them into the final loss L using a loss annealing scheduler (see Equation 5), which will gradually incorporate the unsupervised loss as training progresses. The goal here is to mainly use the supervised loss first so that the model gets a sense of the classification task. Then, incorporating more and more knowledge from unlabeled samples will make the model more robust to noise, which is essential as it is constantly tested on classes it has never seen before. We explore three different strategies for gradually increasing the unsupervised contribution: a linear approach (  = 1 ), an aggressive one (  = 0 . 25 ), and a conservative one (  = 4 ). We consider the DialoGLUE benchmark (Mehri et al., 2020), a set of natural language understanding benchmark for task-oriented dialogue, which contains three datasets for intent detection: Banking77 , HWU64 and Clinic150  the three datasets were already available prior the release of DialoGLUE. Additionally, we also consider the Liu57 intent detection dataset, as it contains the same order of magnitude of intent classes and is user-generated as well. All datasets are public and in English. Banking77 The Banking77 dataset (Casanueva et al., 2020) classifies 13 , 083 user utterances related to into 77 different intents. This dataset i) is specific to a single domain (banking) and ii) requires a fine-grained understanding to classify due to intents being very similar. Following (Mehri et al., 2020) and contrary to (Casanueva et al., 2020), we designate a validation set along a training and a testing set for that dataset (Table 1). HWU64 HWU64 (Xingkun Liu and Rieser, 2019) classifies 25 , 716 user utterances with 64 user intents. It features intents spanning across 21 domains (alarm, audio, audiobook, calendar, cooking, datetime, . . . ). When separating training, validation, and test labels, we ensure each domain is rep-Dataset #sentences #classes train/valid/test (total) Available sentences/class #tokens/sentence Banking77 13 , 083 25 / 25 / 27(77) 170  33 11 . Clinic150 This dataset (Larson et al., 2019) classifies 150 user intents in perfectly equally-distributed classes. This chatbot-like style dataset was initially designed to detect out-of-scope queries, though, in our experiments, we discard the out-of-scope class and only keep the 150 labeled classes to work with, as in (Mehri et al., 2020). Liu57 Introduced by Liu et al. (2019a), this intent detection dataset is composed of 54 classes. It was collected on Amazon Mechanical Turk, where workers were asked to formulate queries for a given intent with their own words. It is highly imbal-anced: the most (resp. least) common class holds 5 , 920 (resp. 24 ) samples 5.2 Experimental settings Conditional language model and language model . For the BART fine-tuning process, we used the defaults hyper-parameters reported in (Lewis et al., 2020), and we fine-tuned the BART model for a single epoch (two hours on a Titan RTX GPU). Increasing the number of epochs for fine-tuning BART degrades performances on the intent detection task: the downstream diverse beam search struggles to find diverse enough beam groups since the model perplexity has been lower with further fine-tuning (this is also hinted in (Bevilacqua et al., 2020)). Our text encoder f  is a bert-base model, and the embedding of a given sentence is the last layer hidden state of the first token of this sentence. For each dataset, this model is fine-tuned on the masked language modeling task for 20 epochs. Then, the encoder of our meta learner is initialized using the weights of this fine-tuned model. Datasets From a dataset point-of-view, we create two data profiles: full (all the training dataset is available, the usual meta-learning scenario) and low (only 10 samples are available for each training class, an even more challenging meta-learning scenario in which a model meta-learns on very few samples per training class). All experimental setups are run 5 times. For each run, we randomly select training, validation, and testing classes, as well as the samples for the low setting. We train the few-shot models for a maximum of 10 , 000 C-way K-shots episodes, evaluating and testing every 100 episodes, stopping early if the evaluation accuracy has not progressed for at least 20 evaluations. We evaluate and test using 600 episodes, as in other few-shot works (Snell et al., 2017; Chen et al., 2019). We compare the systems in the following standard few-shot evaluation scenarios: 5 -way 1-shot, and 5 -way 5-shots. Paraphrasing. At each episode, we draw U = 5 unlabeled samples to generate paraphrases from. For the back-translation baseline, we use the publicly available 3 translation models from the Helsinki-NLP team. We use the following pivot languages: fr , es , it , de , nl , which yields 5 augmentations for each unlabeled sentence. For our experiments with Diverse Beam Search, we generate sentences using 15 beams, group them into 5 groups of 3 beams. In each group, we select the generated sentence which is the most different from the input sentence using BLEU as a metric for diversity. This yields M = 5 paraphrases for each unlabeled sentence, as in the back-translation baseline. DBS uses a diversity penalty parameter to penalize words that have already been generated by other beams to enforce diversity. As advised in the original DBS paper (Vijayakumar et al., 2018), we set the diversity penalty to 0 . 5 in our experiments, which provides diversity while limiting model hallucinations. Our Unigram Masking strat-egy's masking probability is set to p mask = 0 . 7 found by linear search from 0 to 1 with steps of 0 . 1 . We evaluate the diversity of paraphrases for each method, and report results for two representative datasets in Table 3 (due to space limitations, the 3 https://huggingface.co/models?search=helsinki-nlp report for all datasets is given in appendix B). For each paraphrasing method and each dataset, metrics are computed over unlabeled sentences and their paraphrases. To assess the diversity of paraphrases generated by the different methods, the popular BLEU metric in Neural Machine Translation is a poor choice (Bawden et al., 2020). We use the bi-gram diversity ( dist-2 ) metric as proposed by (Ippolito et al., 2019), which computes the number of distinct 2-grams divided by the total amount of tokens. We also report the average similarity (denoted use ) within each sentence set, using the Universal Sentence Encoder as an independent sentence encoder. Results show that paraphrases obtained with back-translation are too close to each other, resulting in a high sentence similarity and low bi-gram diversity. On the other hand, DBS generates more diverse sentences with a lower similarity. Our masking strategies strengthen this effect and yield even more diversity. The measured diversity strongly correlates with the average accuracy of the intent detection task (Table 4). In this section, we discuss the accuracy results for the different meta-learners, for the standard 5-way and { 1, 5 } -shots meta-learning scenarios, as provided in Table 4. The reported metric is the accuracy on the test set at the iteration where the validation set's accuracy is maximal. Our DBS+unigram strategy row corresponds to the flat masking strategy, with p mask = 0 . 7 . First, all methods augmented with unsupervised diverse paraphrasing outperform prototypical networks. However, back translation demonstrates only a limited improvement over the vanilla prototypical network due to their narrow diversity for short texts. Using paraphrases from DBS yields better results  about 0.5 points over BT, on average , hinting that using diverse paraphrases in the unsupervised consistency loss allows the few-shot model to build more robust sentence representations and therefore provides improved generalization capacities. Those results are consistent across the different datasets, except for Clinic for which accuracies are all very high, making all methods hardly separable. The dataset is not challenging enough, or in other words, meta-learning is robust to unbalanced short text classification problems given the nature of that dataset. These results illustrate the need for unsupervised paraphrasing and show that using diverse paraphrases provide a significant performance leap. In the 1-shot (resp. 5-shot) scenario, our best meta-learner improves prototypical networks by 5 . 27 (resp. 2 . 85 ) points on average. Remember that these improvements are made in an unsupervised manner hence at no additional cost. Slightly different from to (Xie et al., 2020), we do not find statistical differences depending on the rate at which  L is annealed in PROTAUGMENT loss (   { 0 . 25 , 1 , 4 } ), which makes it easier to tune  our unsupervised loss serves as a consistency regularization. Due to space limitations, this analysis is available in appendix D. Adding our masking strategies on top of DBS has a significant impact on all datasets, with the unigram variant being up about 2 points over the vanilla DBS on average. On all datasets except Clinic, given only 10 labeled samples per class ( low profile), it even outperforms the supervised baseline which is given the full training data ( full profile). This means that PROTAUGMENT does better than prototypical networks with much less  15 times, and up to 47 times, depending on the dataset  labeled sentences per class. Those results indicate that our method more than compensates for the lack of labeled data and that no matter the amount of data available for the training class, there is a performance ceiling you cannot overcome without adding unsupervised knowledge from the validation and test classes. In the full profile, when given all the training data, our method greatly surpasses the Prototypical Network  3.58 points given 1 shot, on average. Moreover, PROTAUGMENT is not only suited for the case where very little training data is available ( low profile): when sampling shots from the entire training dataset ( full profile), it outperforms a fully supervised baseline. Furthermore, note that our method is consistently more stable than the supervised baselines, as its average standard deviation over the different runs is much lower than the vanilla Prototypical Network. We experimented with three variants of the unigram strategy (Section 4.2), each assigning a different drop chance to each token depending on its position in the input sentence. In our experiments, we did not observe any significant difference in performance when putting more weight on the first tokens ( down ), or last tokens ( up ), or the same weight on all tokens ( flat ) (Detailed results in appendix C). We also conducted experiments where we tune the value p mask , from 0 to 1 , selecting 0 . 7 as the best trade-off (Figure 2). This figure also clearly shows that the Clinic dataset is one order of magnitude easier to solve than the other datasets. In this work, we proposed PROTAUGMENT , an architecture for meta-learning for the problem of classifying user-generated short-texts (intents). We first introduced an unsupervised paraphrasing consistency loss in the prototypical network's framework to improve its representational power. Then, while the recent diverse beam search algorithm was designed to enforce diversity between the generated paraphrases, it does not ensure diversity between the generated paraphrases and the original sentences. To make up for the latter, we introduce constraints in the diverse beam search generation, further increasing the diversity. Our thorough evaluation demonstrates that PROTAUGMENT offers a significant leap in accuracy for the most recent and 0.00(DBSbase) 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 p mask 0.800.810.820.830.840.850.860.870.880.890.900.910.920.930.940.950.960.970.980.991.00 DBS + u n i g r a m _ f l a t a cc u r a c y BANKING77 HWU64 Liu Clinic Figure 2: 5-way 1-shot accuracy of DBS-unigram-flat method using different values of p mask . challenging datasets. PROTAUGMENT vastly outperforms prototypical networks, which was found to be the best meta-learning framework for short-texts (Dopierre et al., 2021) against unsupervised-extended Prototypical Networks (Ren et al., 2018), Matching Networks (Vinyals et al., 2016), Relation Networks (Sung et al., 2018), and Induction Networks (Geng et al., 2019), thereby making PROTAUGMENT the new state-of-the-art for this task. We provide the source code of PROTAUGMENT as well as code for evaluations reported in this paper on a public repository 4 4 https://github.com/tdopierre/ProtAugment Acknowledgments We are thankful for the discussion we had with Michele Bevilacqua, Marco Maru, and Roberto Navigli from Sapienza university about diversity in Natural Language Generation. We also would like to thank ANRT 5 for making partnerships between companies and universities happen.
abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain objective abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other method other other abstain other other other objective other other other method abstain abstain abstain method method abstain abstain method method abstain abstain abstain abstain abstain abstain method other abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result result abstain abstain result result abstain abstain abstain abstain abstain abstain abstain objective objective abstain abstain other abstain other other abstain