Few-shot text classification is a fundamental NLP task in which a model aims to classify text into a large number of categories, given only a few training examples per category. This paper explores data augmentationa technique particularly suitable for training with limited datafor this few-shot, highly-multiclass text classification setting. On four diverse text classification tasks, we find that common data augmentation techniques can improve the performance of triplet networks by up to 3.0% on average. To further boost performance, we present a simple training strategy called curriculum data augmentation , which leverages curriculum learning by first training on only original examples and then introducing augmented data as training progresses. We explore a two-stage and a gradual schedule, and find that, compared with standard single-stage training, curriculum data augmentation trains faster, improves performance, and remains robust to high amounts of noising from augmentation. Traditional text classification tasks such as sentiment classification (Socher et al., 2013) typically have few output classes (e.g., in binary classifica-tion), each with many training examples. Many practical scenarios such as relation classification (Han et al., 2018), answer selection (Kumar et al., 2019), and sentence clustering (Mnasri et al., 2017), however, have a converse setup characterized by a large number of output classes (Gupta et al., 2014), often with few training examples per class. This scenario, which we henceforth refer to as few-shot, highly-multiclass text classification , is a common setting in NLP applications and can be challenging due to the scarcity of training data. et al., 2020). In traditional text classification tasks, it has been shown that although performance improvements can be marginal when training data is sufficient, augmentation is especially beneficial in limited data scenarios (Xie et al., 2020). As such, we hypothesize that the few-shot, highly-multiclass text classification scenario is a suitable context for data augmentation. First, we apply popular data augmentation techniques to the common triplet loss (Schroff et al., 2015) approach for few-shot, highly multiclass classification, finding that out-of-the-box augmentation can improve performance noticeably. We then propose a simple curriculum learning strategy called curriculum data augmentation and experiment with two schedules, as shown in Figure 1. A two-stage curriculum, which first trains on original data and then introduces augmented data of fixed temperature (amount of nois-ing), achieves slightly better performance than standard augmentation, while training faster and remaining more robust to high temperatures. A gradual curriculum, which also first trains on original data only but gradually increases augmentation temperature at each subsequent stage, takes longer to converge but improves more than 1% over standard augmentation. Motivation. Inspired by human and animal learning, curriculum learning (Bengio et al., 2009) posits that neural networks train better when examples are not randomly presented but instead organized in a meaningful order that gradually shows more concepts and complexity. Traditionally, curriculum learning approaches first assume that a range of example difficulty exists in the data and then leverage various heuristics to sort examples by difficulty and train models on progressively harder examples (Bengio et al., 2009; Tsvetkov et al., 2016; Weinshall et al., 2018). A newer school of thought, however, has noted that instead of discovering a curriculum in existing data, data can be intentionally modified to dictate an artificial range of difficulty (Korbar et al., 2018; Ganesh and Corso, 2020) this is the approach we will take here. Our approach. Unlike data augmentation in computer vision where augmented data undoubtedly resembles original data, in text, data augmentation techniques might introduce linguistic adversity and therefore can be seen as a form of noising (Li et al., 2017; Wang et al., 2018), where noised data is harder to learn from than unmodified original data. As such, we can create an artificial curriculum in the data by leveraging controlled application of data augmentation, starting by training on only original data and then adding augmented data with a higher levels of noising as training progresses. Specifically, we propose two simple schedules. (1) Two-stage curriculum data augmentation calls for one stage of training with only original data, followed by one stage of training with augmented data of fixed temperature. (2) Gradual curriculum data augmentation involves one stage of training with only original data, followed by multiple stages of training with augmented data where the temperature of augmented data (i.e., fraction of perturbed tokens) gradually increases each stage. We conduct empirical experiments to evaluate curriculum data augmentation on a variety of text classification tasks using a triplet loss model. 1 3.1 Datasets We will consider four diverse few-shot, highly-multiclass text classification scenarios: 1. HUFF ( c = 41 ). The HuffPost Dataset categorizes 200k news headlines from 20122018 into 41 categories such as politics, wellness, entertainment, and travel (Misra, 2018). We use all 41 categories and perform a 70%-30% train-test split by class. 2. FEWREL ( c = 64 ). The FewRel dataset contains sentences categorized by a relationship between its specified head and tail tokens such as capital of,' member of,' and birth name' (Han et al., 2018). We use all 64 classes given in the posted training set, splitting 100 examples per class into a test set, with the remainder of the examples going into the training set. 3. COV-C ( c = 87 ). The COVID-Q dataset clas-sifies questions into 89 clusters where all ques-1 Code is made publicly available at https://github. com/jasonwei20/triplet-loss . tions in a cluster ask about the same thing (Wei et al., 2020). We use the train-test split with three training examples per class as given by the authors. We find that 2 of the 89 classes in the training set actually have only two examples per class instead of the reported three, and so we remove these classes from the training and test sets and use the 87 classes that remain. 4. AMZN ( c = 318 ). The Amazon product review dataset aims to categorize a product into a certain class given a review (Yury, 2020). We only consider the 318 level-3' classes given in this dataset with at least six examples per class. To balance the class distribution during experiments, we randomly sample N c examples per class to be used for training, with N c varying based on the experiment and dataset. Our sampled training sets for COV-C and AMZN have N c = 3 examples per class, and our training sets for HUFF and FEWREL have N c = 10, a common low-resource scenario. 2 For all experiments, we use top-1 accuracy (%) as the evaluation metric. For few-shot, highly-multiclass classification, a common approach is the triplet loss classifier (Schroff et al., 2015), first developed for facial recognition and now also used in NLP (dos Santos et al., 2016; Ein Dor et al., 2018; Lauriola and Mos-chitti, 2020). Specifically addressing few-shot classification, a triplet loss network minimizes distance between examples with the same label and maximizes distance between examples with different labels. During training, given a triplet of (anchor a , positive example p , and negative example n ), a triplet loss network minimizes: L = (cid:88) i d ( a, p )  d ( a, n ) +  , (1) where  is a margin enforced between positive and negative pairs, and d (  ) computes the distance between the input encodings of two examples. To sample triplets, we will consider two strategies: random sampling , which selects triplets randomly, and hard negative mining (Schroff et al., 2015), where triplets are sampled such that d ( a, p ) +  > d ( a, n ) . At evaluation time, a triplet loss classifier returns the class of the example in the training set 2 For COV-C, the given training set size is N c = 3, and for AMZN, N c = 3 is the largest possible such that the training set is balanced by class. with the smallest distance to a given test example. Indeed, both triplet loss and data augmentation target training with limited data, and so combining them seems particularly promising for the the few-shot classification scenario. For our model, we use standard BERT-base with average-pooled encodings and then train a two-layer triplet loss network on top of these encodings. Our triplet loss network architecture contains a linear layer with 200 hidden units, tanh activation, a dropout layer with p = 0 . 4 , and a final linear layer with 40 hidden units. We use cosine distance, a margin of  = 0 . 4 , a batch size of 64 triplets, and a learning rate of 2  10  5 . We implement EDA (Wei and Zou, 2019), a popular combination of token-level augmentation techniques (synonym replacement, random insertion, random swap, random deletion) that defines their temperature parameter 0    1 as the fraction of perturbed tokens, in  4.14.4, and explore four other techniques in  4.5. For the two-stage curriculum, we started by training on original data only, and when validation loss converges, we introduce augmented data of fixed temperature at an augmented to original data ratio of 4:1. For the gradual curriculum, we begin with a temperature of  = 0.0 (equivalent to no augmentation) and then linearly increase the temperature by 0.1 every time validation loss plateaus, up to a final temperature of 0.5. Schedules for each dataset are shown in the Appendix. Figure 2 shows an example training plot with our proposed curriculum schedules. This section compares no augmentation, standard augmentation, and curriculum augmentation for triplet loss networks using two different triplet sampling strategies. Table 1 summarizes these results for five random seeds. We also implement a cross-entropy loss classifier for reference. For triplet loss using random sampling, a model with no augmentation achieved a mean accuracy across our four datasets of 30.2%, and standard augmentation improved performance noticeably by +1.9%. Two-stage curriculum augmentation, which HUFF FEWREL COV-C AMZN c =41 c =64 c =87 c =318 Average  Cross-entropy loss 13.3  2.1 32.4  2.3 26.1  0.8 2.0  0.3 18.5 + standard data augmentation 16.3  2.4 33.0  1.1 24.0  1.6 2.2  0.4 18.9 +0.4 Triplet loss with random sampling 20.9  1.0 43.6  1.2 39.7  1.0 16.4  1.3 30.2 + standard data augmentation 22.2  1.4 44.2  1.6 45.4  1.8 16.5  1.7 32.1 +1.9 + curriculum data augmentation: two-stage 22.3  1.6 44.2  1.8 46.5  1.7 17.2  1.3 32.6 +2.4 + curriculum data augmentation: gradual 23.7  1.2 46.1  0.9 47.1  1.3 17.6  1.0 33.6 +3.4 Triplet loss with hard negative mining 21.0  1.2 44.6  1.2 39.5  1.0 16.2  0.9 30.3 + standard data augmentation 22.6  1.8 45.0  1.6 48.2  0.9 17.4  1.7 33.3 +3.0 + curriculum data augmentation: two-stage 22.6  1.8 45.7  1.4 47.6  1.3 17.9  1.1 33.5 +3.2 + curriculum data augmentation: gradual 23.8  0.9 47.1  1.4 48.9  0.9 18.9  0.9 34.7 +4.4 Table 1: Accuracy (%) on four diverse highly multiclass classification tasks for no augmentation, standard augmentation, and curriculum augmentation. trains for the same number of updates as standard augmentation, achieved a mean accuracy of 32.4%, outperforming standard augmentation by +0.5%. The gradual curriculum further improved +1.0% over the two-stage curriculum. For triplet loss with hard negative mining, standard augmentation substantially improved +3.0% over no augmentation, as adding in augmented data, which is more difficult to classify, likely helped generate a more diverse set of hard negatives. The two-stage curriculum still maintained small improvement over standard augmentation here, and the gradual curriculum provided an even-stronger boost of +4.4% over no augmentation, possibly because increasing the temperature of augmented data over time facilitated hard-negative mining more so than using a constant temperature. Notably, the largest gains for all augmentation types were on COV-C (up to +9.4%). We hypothesize that this occurred not necessarily because of COV-C's smaller data size; rather, there was likely more overfitting to be mitigated by data augmentation as a result of the greater semantic difference between COV-C and the corpus used to pre-train BERT, compared with the other three datasets. This ablation investigates how data augmentation performs for different dataset sizes. Figure 3 shows these results for hard negative mining averaged over HUFF and FEWREL, our two datasets where sufficient data is available. The two-stage curriculum outperformed standard augmentation by a small margin, although both dropped in performance at N c = 20 , consistent with prior findings on the diminished effect of data augmentation for larger datasets (Xie et al., 2020; Andreas, 2020). The gradual curriculum, on the other hand, maintained relatively robust improvement for all dataset sizes explored. Effective curriculum learning necessitates a range of difficulty in training data. In our case, this range is controlled by augmentation temperature , a parameter that dictates how perturbed augmented examples are and therefore affects the distribution of difficulty in training examples. When the distribution of difficulty in data is larger, we should expect Schedule HUFF FEWREL COV-C AMZN Avg. a greater improvement from curriculum learning. Figure 4 compares standard and two-stage curriculum augmentation for various temperatures, with results averaged over all four datasets. At low temperature, augmented examples remained pretty similar to original examples, and so the range of difficulty in examples was small and therefore curriculum learning showed little improvement. At higher temperatures, however, augmented examples became quite different from original examples, and so the range of difficulty in examples was much larger and therefore curriculum data augmentation improved over standard augmentation more. Whereas Wei and Zou (2019) recommend   { 0 . 05 , 0 . 1 } , our curriculum framework liberates us to use much larger  and maintain relatively robust improvements even at   { 0 . 4 , 0 . 5 } when standard augmentation is no longer useful. The gradual curriculum linearly increases temperature  from 0.0 to 0.5 in six stages, and so to isolate the effect of this curriculum, in this section we compare it with a control schedule (where the  in each stage is decided randomly) and an anti-curriculum schedule (where  linearly decreases from 0.5 to 0.0 in six stages). As expected, these results, shown in Table 2, indicate that the curriculum contributes substantively over the control schedule. As our experiments so far have focused on EDA augmentation (Wei and Zou, 2019), this section explores other common techniques in the curriculum framework: (1) Token Substitution replaces words with WordNet synonyms (Zhang et al., 2015); (2) Pervasive Dropout applies word-level dropout with probability p = 0 . 1 (Sennrich et al., 2016a); (3) SwitchOut replaces a token with a randomly token uniformly sampled from the vocabulary (Wang et al., 2018); and (4) Round-Trip Translation translates text into another language and then back into the original language (Sennrich EDA Back-Translation SwitchOut Pervasive Dropout Token Substitution 31 . et al., 2016b). Figure 5 compares standard and two-stage curriculum results averaged over all datasets. EDA improved performance the most, perhaps because it combines four token perturbation functions, creating more diverse noise compared with using a single operation. Our work combines curriculum learning, data augmentation, and triplet loss, and is inspired by prior work in these areas. In vision, several papers have proposed reinforcement learning policies for data augmentation (Cubuk et al., 2019; Ho et al., 2019), and hard negative mining (Schroff et al., 2015; Song et al., 2016) itself can be seen as a form of curriculum learning. In NLP, the work of Kumar et al. (2019) is perhaps most similar to oursthey show that sampling strategies are key for improving performance with triplet loss networks. We see our work as the first to explicitly analyze curriculum learning for data augmentation in text. In closing, we have proposed a curriculum data augmentation framework that is simple yet provides empirical performance improvements, a compelling case for the combination of ideas explored. Our approach exemplifies how data augmentation can create an artificial range of example difficulty that is helpful for curriculum learning, a direction that potentially warrants future research. We thank Ruibo Liu, Weicheng Ma, Jerry Wei, and Chunxiao Zhou for feedback on the manuscript. We also thank Kai Zou for organizational support.
abstain objective result objective objective abstain abstain abstain abstain abstain objective abstain objective objective abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain method abstain abstain method abstain other abstain abstain method method abstain abstain method method method method abstain abstain other method abstain abstain abstain method method abstain method abstain method method method abstain method abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other other objective objective method other other