The few-shot natural language understanding (NLU) task has attracted much recent attention. However, prior methods have been evaluated under a disparate set of protocols, which hinders fair comparison and measuring progress of the field. To address this issue, we introduce an evaluation framework that improves previous evaluation procedures in three key aspects, i.e., test performance, dev-test correlation, and stability. Under this new evaluation framework, we re-evaluate several state-of-the-art few-shot methods for NLU tasks. Our framework reveals new insights: (1) both the absolute performance and relative gap of the methods were not accurately estimated in prior literature; (2) no single method dominates most tasks with consistent performance; (3) improvements of some methods diminish with a larger pretrained model; and (4) gains from different methods are often complementary and the best combined model performs close to a strong fully-supervised baseline. We open-source our toolkit, FewNLU, that implements our evaluation framework along with a number of state-of-the-art methods. 1 2 1 Introduction Few-shot learning for natural language understanding (NLU) has been significantly advanced by pretrained language models (PLMs; Brown et al., 2020; Schick and Schtze, 2021a,b). With the goal of learning a new task with very few (usually less than a hundred) samples, few-shot learning benefits from the prior knowledge stored in PLMs. Various few-shot methods based on PLMs and prompting have been proposed (Liu et al., 2021b; Menon et al., 2021; Gao et al., 2020). The authors have contributed equally to this work. Corresponding Authors. 1 Leaderboard: https://fewnlu.github.io 2 Code available at https://github.com/THUDM/ FewNLU Although the research of few-shot NLU is developing rapidly, the lack of a standard evaluation protocol has become an obstacle hindering fair comparison between various methods on a common ground and measuring progress of the field. While some works (Schick and Schtze, 2021b; Menon et al., 2021) experimented with a fixed set of hyper-parameters, prior work (Perez et al., 2021; Zhang et al., 2020) noted that such a setting might be exposed to the risk of overestimation . 3 Other works (Liu et al., 2021b; Gao et al., 2020; Perez et al., 2021) proposed to use a small development set to select hyper-parameters, but their evaluation protocols vary in a few key aspects (e.g., how to construct data splits), which in fact lead to large differences as we will show (in Section 4.2). The above phenomena highlight the need for a common protocol for the evaluation of few-shot NLU methods. However, the fact that few-shot learning is extremely sensitive to subtle variations of many factors (Dodge et al., 2020; Gao et al., 2020) poses challenges for designing a solid evaluation protocol. In this work, aiming at addressing the aforementioned challenge, we propose an evaluation framework for few-shot NLU. The evaluation framework consists of a repeated procedureselecting a hyper-parameter, selecting a data split, training and evaluating the model. To set up a solid evaluation framework, it is crucial to specify a key design choicehow to construct data splits for model selection. We conduct a comprehensive set of experiments to answer the question. Specifically, we propose a Multi-Splits strategy, which randomly splits the available labeled samples into training and development sets multiple times, followed by aggregating the results from each data split. We show that this simple strategy outperforms several 3 This is because the fixed hyper-parameters are selected according to practical considerations, which are informed by the test set performance from previous evaluations. baseline strategies in three dimensions: (1) the test set performance of the selected hyper-parameters; (2) correlation between development set and true test set performance; and (3) robustness to hyper-parameter settings. We then take a step further to re-evaluate recent state-of-the-art few-shot NLU methods under this common evaluation framework. Our re-evaluation leads to several key findings summarized in Section 2. To aid reproducing our results and benchmarking few-shot NLU methods, we open-source FewNLU, a toolkit that contains implementations of a number of state-of-the-art methods, data processing utilities, as well as our proposed evaluation framework. To sum up, our contributions are as follows. 1. We introduce a new evaluation framework of few-shot NLU. We propose three desiderata of few-shot evaluation and show that our framework outperforms previous ones in these aspects. Thus our framework allows for more reliable comparison of few-shot NLU methods. 2. Under the new evaluation framework, we benchmark the performance of recent methods individually as well as the best performance with a combined approach. These benchmarks reflect the current state of the art and will serve as important baselines for future research. 3. Throughout our exploration, we arrive at several key findings summarized in Section 2. 4. We open-source a toolkit, FewNLU, to facilitate future research with our framework. Finding 1. Our proposed Multi-Splits is a more reliable data-split strategy than several baselines with improvements in (1) test performance, (2) correlation between development and test sets, and (3) stability w.r.t. the number of runs. Finding 2. The absolute performance and the relative gap of few-shot methods were in general not accurately estimated in prior literature. It highlights the importance of evaluation for obtaining reliable conclusions. Moreover, the benefits of some few-shot methods (e.g., ADAPET) decrease on larger pretrained models. Finding 4. No single few-shot method dominates most NLU tasks. This highlights the need for the development of few-shot methods with more consistent and robust performance across tasks. The pretraining-finetuning paradigm (Howard and Ruder, 2018) shows tremendous success in few-shot NLU tasks. Various methods have been developed such as [CLS] classification (Devlin et al., 2018), prompting-based methods with discrete prompts (Schick and Schtze, 2021b; Gao et al., 2020) or continuous prompts (Liu et al., 2021b; Shin et al., 2020; Li and Liang, 2021; Lester et al., 2021), and methods that calibrate the output distribution (Yang et al., 2021; Zhao et al., 2021). The fact that few-shot learning is sensitive to many factors and thus is extremely unstable (Liu et al., 2021a; Lu et al., 2021; Zhang et al., 2020; Dodge et al., 2020) increases the difficulty of few-shot evaluation. Several works address evaluation protocols to mitigate the effects of instability: Gao et al. (2020) and Liu et al. (2021b) adopt a held-out set to select models. Perez et al. (2021) proposed K -fold cross-validation and minimum description length evaluation strategies. Our work differs from these works on few-shot evaluation in several aspects: (1) we propose three metrics to evaluate data split strategies; (2) while most prior work proposed evaluation protocols without justification, we conduct comprehensive experiments to support our key design choice; (3) we formulate a general evaluation framework; (4) our re-evaluation under the proposed framework leads to several key findings. Though there have been a few existing few-shot NLP benchmarks, our work is quite different in terms of the key issues addressed. FLEX (Bragg et al., 2021) and CrossFit (Ye et al., 2021) studied principles of designing tasks, datasets, and metrics. FewGLUE (Schick and Schtze, 2021b) is a dataset proposed for benchmarking few-shot NLU. CLUES (Mukherjee et al., 2021) pays attention to the unified format, metric, and the gap between human and machine performance. While the aforementioned benchmarks focus on what data to use and how to define the task, our work discussed how to evaluate which aims at establishing a proper evaluation protocol for few-shot NLU methods. Since FewNLU is orthogonal to the 502 aforementioned prior work, it can also be employed on the data and tasks proposed in previous work. Formally, for a few-shot NLU task, we have a small labeled set D label = { ( x i , y i ) } Ni and a large test set D test = { ( x test i , y test i ) } i where N is the number of labeled samples, x i is a text input (consisting of one or multiple pieces), and y i  Y is a label. The goal is to finetune a pretrained model with D label to obtain the best performance on D test . An unlabeled set D unlab = { x unlab i } i may additionally be used by semi-supervised few-shot methods (5.1). Our preliminary results (in Appendix A.1) show that using a fixed set of hyper-parameters (Schick and Schtze, 2021a,b) is sub-optimal, and model selection is required. It motivates us to study a more robust evaluation framework for few-shot NLU. The goal of an evaluation framework is twofold: (1) benchmarking few-shot methods for NLU tasks such that they can be fairly compared and evaluated; and (2) obtaining the best few-shot performance in practice. In light of the two aspects, we propose the few-shot evaluation framework in Algorithm 1. The framework searches over a hyper-parameter space H to evaluate a given few-shot method M , obtaining the best hyper-parameter setting h (cid:63) and its test set results. 4 The measurement for each h is estimated by performing training and evaluation on multiple data splits (obtained by splitting D label according to a strategy) and reporting their average dev set results. Finally, the method is evaluated on D test using the checkpoints corresponding to h (cid:63) . For benchmarking, we report the average and standard deviation over multiple test set results. Otherwise, that is, to achieve a model with the best practical performance, we re-run on the entire D label with h (cid:63) . The framework requires specifying a key design choicehow to construct the data splits, which we will discuss in 4.2. 4 For simplicity and ease of use, we use grid search for searching the hyper-parameter space H and identify critical hyper-parameters to limit its size. More complex search methods such as Bayesian Optimization (Snoek et al., 2012) could be used to search over larger hyper-parameter spaces. We first propose the following three key desiderata for the evaluation of different data split strategies. 1. Performance of selected hyper-parameter. A good data split strategy should select a hyper-parameter that can achieve a good test set performance. We use the same metrics as (Schick and Schtze, 2021b), along with standard deviations. 2. Correlation between dev and test sets (over a hyper-parameter distribution) . Since a small dev set is used for model selection, it is important for a good strategy to obtain a high correlation between the performances on the small dev set and test set over a distribution of hyper-parameters. We report the Spearman's rank correlation coefficient for measurement. 3. Stability w.r.t. number of runs K . The choice of the hyper-parameter K should have small impacts on the above two metrics (i.e., performance and correlation). To analyze the stability w.r.t K , we report the standard deviation over multiple different values of K . Besides, it is desirable to have reduced variance when K increases. Thus we report the above two metrics with different 503 values of K and the standard deviation of test scores over K runs. We consider several data split strategies. Some are proposed by previous work, including K -fold cross validation (CV) (Perez et al., 2021), minimum description length (MDL) (Perez et al., 2021), and bagging (BAG) (Breiman, 1996). We also consider two simple strategies worth exploring, including random sampling (RAND) and model-informed splitting (MI). And we propose a new data split strategy, Multi-Splits (MS). Besides, we also experiment a special case of CV when K equals the number of labeled sample, which is leave-of-out cross validation (LOOCV). Since LOOCV takes much longer time and suffers from efficiency problem, we only experimented on several tasks and left the results in Appendix A.2.4. They all fit into the pipeline of the proposed framework in 4.1: 1. K -fold CV equally partitions D label into K folds. Each time, it uses the k th fold for validation and the other K  1 folds for training. 2. MDL assigns half of D label as the joint training data and equally partitions the other half into K folds. Each time, it uses the k th fold for validation, and all its previous k  1 folds together with the joint training data for training. 3. Bagging samples N  r ( r  (0 , 1] is a fixed ratio) examples with replacement from the labeled sample as the training set, leaving samples that do not appear in the train set for validation. 4. Random Sampling performs random sampling without replacement from D label twice, respectively sampling N  r and N  (1  r ) data as the training and development sets. 5. Model-Informed Splitting computes representations of each labeled example using a model, and clusters them into two distinct sets, respectively as the training and development sets. 5 6. Multi-Splits randomly splits D label into training and development sets using a fixed split ratio r . Essentially, these data split strategies differ in several key aspects. 1. For CV and MDL, K controls the number of runs and the split ratio. For Multi-Splits, BAG and RAND, the split ratio is decoupled from K and is controlled by r . For MI, the split ratio and number of runs depend on D label . 5 Specifically, we used a BERT-Base model to encode data and take the [CLS] representations. 2. They use a different amount of data for training and development sets as Table 1 shows. 3. There are cases when CV and MS share the same split ratio. The difference is that MS allows overlap between splits while CV does not. 4. BAG allows duplicated training data, while RAND and Multi-Splits do not. The training and development sets do not overlap for BAG and Multi-Splits but overlap for RAND. In the limit, our Multi-Splits is similar to leave-P -out cross-validation (LPOCV; Celisse, 2014) 6 where LPOCV runs (cid:0) NP (cid:1) times ( P is the number of dev set examples) while Multi-Splits runs K times. As K increases, Multi-Splits gradually approaches LPOCV. Since it is impossible to enumerate the large number of possible splits in practice, Multi-Splits can be viewed as a practical version of LPOCV. Compared to the strategy of (Gao et al., 2020) that uses multiple datasets, our Multi-Splits uses multiple data splits for a single dataset. It is thus more practical as in real-world scenarios, it is hard to obtain multiple labeled datasets for a true few-shot problem; otherwise, it could be formulated as a fully-supervised learning problem. The strategy in (Liu et al., 2021b) is a special case of Multi-Splits when K = 1 , which suffers from higher variance. To evaluate different data split strategies, we experiment on the FewGLUE benchmark (Schick and Schtze, 2021b). We evaluate strategies based on the widely used prompt-based few-shot method PET (Schick and Schtze, 2021b) with DeBERTa as the base model. 7 We run experiments on the same tasks with the same hyper-parameter space 6 LeaveP -out cross-validation uses P data examples as the development set and the remaining data examples as the training set. This is repeated on all ways to cut the labeled dataset in a development set and a training set. 7 We fixed the parameters of DeBERTa's bottom one-third layers due to GPU memory limitations, which did not affect the performance much in our preliminary experiments. to ensure a fair comparison; in this experiment we search learning rate, evaluation ratio, prompt pattern and maximum training step. More experimental details are in Appendix A.2. Table 2, Table 3 and Figure 1 show the main results with 64 labeled samples. It is noteworthy that we also experimented with 32 labeled samples and have observed that varying the number of labeled examples does not affect the following conclusion (see Appendix A.2). Test Performance and Correlation. From both Table 2 and Table 3, we find that Multi-Splits achieves the best average test set performance as well as the best average correlation among all strategies. We analyze them as follows: 8 1. Multi-Splits uses fewer labeled samples for training (i.e., 128) while CV and MDL use more (i.e., 192 and 176). Despite using more training data, both CV and MDL do not perform better. This indicates few-shot performance is limited by not being able to select the best model rather than not having sufficient training data. Both CV and MDL use fewer data for validation (i.e., 64 and 32) than Multi-Splits (i.e., 128), thus leading to poor correlation. 2. Although Multi-Splits and BAG use the same number of training data (i.e., 128), there could be duplication in the training set of BAG, making it 8 In the following explanation, the numbers refer to the total training/development data covering K =4 runs. poor in diversity and further leading to lower test performance, compared to Multi-Splits. This indicates diversity of training sets is crucial when constructing few-shot data splits. 3. RAND uses similar-sized dev and train sets to BAG and MS but performs worse in test performance. Since there could be overlap between train and dev sets, the model may have memorized data, leading to poor test performance. 4. MI constructs very different train and dev sets. Overfitting on one of them and validating on the other pose more challenges for the few-shot method on out-of-distribution tasks. some representative strategies. Both CV and MDL represent strategies whose number of runs are coupled with the size of data split, while Multi-Splits represents strategies that have a fixed ratio and in-dependent K . We observe: (1) Multi-Splits (blue lines) is the most stable in correlation and performance, while other strategies CV and MDL are more sensitive to the choice of K . (2) Multi-Splits shows the smallest variance over multiple runs on both BoolQ and RTE. For COPA, though Multi-Splits shows high variance when K = 2 , the variance becomes smaller with larger K , while CV and MDL suffer from increasing or unstable variance. A possible explanation is that increasing K does not affect the number of training and development examples for Multi-Splits; instead, it increases the confidence of results. An important practical ben-efit of Multi-Splits is that one can always choose to increase K for lower variance. However, for CV and MDL, the sizes of training and development sets are affected by K , where extremely large K value leads to a failure mode and extremely small K leads to unstable results. In practice, it is hard to know which value of K to use a priori. and analysis, we arrive at the following finding. Finding 1. Our proposed Multi-Splits is a more reliable data-split strategy than several baselines with improvements in (1) test performance, (2) correlation between development and test sets, and (3) stability w.r.t. number of runs. Remark Our evaluation framework is better in terms of test performance, dev-test correlation, and stability, which proves it can achieve possible peak performance, reliably select the corresponding hy-perparameters according to dev results without overfitting, and mitigate the effects of randomness to the maximum extent. Therefore, the estimation of our evaluation framework for model performance is more reliable than previous evaluations. We now proceed to re-evaluate state-of-the-art few-shot methods under our evaluation framework with the Multi-Splits strategy. We consider two types: minimal few-shot methods , which only assume access to a small labeled dataset, including Classification (CLS; Devlin et al., 2018), PET (Schick and Schtze, 2021b), ADAPET (Menon et al., 2021), P-tuning (Liu et al., 2021b) and FlipDA (Zhou et al., 2021); and semi-supervised few-shot methods , which allow accessing an additional unlabeled dataset, including PET+MLM (Schick and Schtze, 2021a), iPET (Schick and Schtze, 2021b) and Noisy Student (Xie et al., 2020). The same benchmark datasets, metrics, and hyper-parameter space as in 4.2.3 are used. We use 32 labeled samples for training. We consider two labeling strategies to obtain the pseudo-labels on unlabeled samples used by the semi-supervised methods for self-training, including single-split labeling and cross-split labeling . In the single-split setting (Schick and Schtze, 2021b), pseudo-labels are generated by the models trained on the same data split. In the cross-split setting in our evaluation framework, the pseudo-labels are generated by the models trained on multiple different data splits. More configuration details are in Appendix A.4. Re-Evaluation Results Table 4 shows our reevaluation results. The prompt-based fine-tuning paradigm significantly outperforms the classification fine-tuning on all tasks and on both pretrained models (with an advantage of more than 15 points on average). DeBERTa outperforms ALBERT consistently. We observe significant differences in performance between different prompt-based minimal few-shot methods with ALBERT (e.g., ADAPET and FlipDA outperform PET respectively by about 4 points and 2 points on average) while differences with DeBERTa are slight (e.g., PET, ADAPET, P-tuning, and FlipDA have a performance gap of only about 1.0 points on average). In contrast, semi-supervised few-shot methods (i.e., iPET and Noisy) generally improve 12 points on average compared to minimal few-shot methods on both models. Comparison to Prior Evaluations Since we have proved that our evaluation framework is more reliable in estimating method performance as shown in Section 4.2.4, we conduct experiments to compare the estimates by our evaluation framework and prior evaluations to study whether model performance was accurately estimated in prior work. Table 6 lists the absolute performance from prior evaluations and our evaluation. Results show the absolute performance of few-shot methods in prior evaluations was generally overestimated on RTE 506 BaseModels Few-ShotMethods BoolQ RTE WiC CB MultiRC WSC COPA Avg. The RoBERTa (fully-sup.) results by (Liu et al., 2019). RoBERTa-large has less parameters than DeBERTa-xxlarge-v2. Table 4: Re-evaluation of few-shot methods on ALBERT and DeBERTa under our evaluation framework with Multi-Splits strategy on test set of our setup. For iPET and Noisy Student, (cross) and (single) respectively means cross-split labeling and single-split labeling strategies as introduced in 5.2. Our Best (few-shot) is the results achieved by a combination method as introduced in 5.4. Globally best results for each task are in bold. Best results for minimal few-shot methods are underlined. (cid:58)(cid:58)(cid:58) Best (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) results (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) semi-supervised (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) few-shot (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) methods are marked with wavelines. and COPA. Similar findings have been highlighted in prior works (Perez et al., 2021; Zhang et al., 2020), and our evaluation framework confirms the findings under a more reliable setup. This results from a more reliable evaluation procedure that emphasizes dev-test correlation to prevent overfitting (discussed in Section 4.2). Besides, the relative gaps between different methods were not accurately estimated by the prior reported numbers. For example, according to the reported results in prior works, ADAPET outperforms P-Tuning on COPA and P-Tuning beats ADAPET on WiC, while our evaluation reveals the opposite. On one hand, this is because prior results were obtained under a less reliable evaluation procedure (discussed in Section 4.2). Deviation in the estimates of absolute performance contributes to inaccuracy in the estimates of relative performance. On the other, prior experiments were not conducted under a shared evaluation procedure. These two factors are corrected by our re-evaluation under the more reliable proposed framework. To sum up, our re-evaluation compares all methods on a common ground, revealing the following: Finding 2. The absolute performance and the relative gap of few-shot methods were in general not accurately estimated in prior literature. This is corrected by our new evaluation framework with improved reliability. It highlights the importance of evaluation for obtaining reliable conclusions. Moreover, the benefits of some few-shot methods (e.g., ADAPET) decrease on larger pretrained models like DeBERTa. We further explore the best few-shot performance by combining various methods, and evaluating under our evaluation framework. For combined options, we consider five minimal few-shot methods (i.e., CLS, PET, ADAPET, P-tuning, and FlipDA), five training paradigms (i.e., single-run, iPET (sin-gle/cross), and Noisy Student (single/cross)), and the addition of a regularized loss (+MLM). We experiment with all possible combinations and report the best for each task. Best (few-shot) in Table 4 achieves the best results on all tasks among all methods. Existing few-shot methods can be practically used in combination. Compared to RoBERTa (fully-sup) (Liu et al., 2019), the performance gap has been further narrowed to 2.89 points on average. 9 Compared to DeBERTa (fully-sup), there is still a sizeable gap between few-shot and fully-supervised systems. We list the best-performing combination for each task in Table 5. The best combinations are very different across tasks, and there is no single method that dominates most tasks. PET and ADAPET as well as iPET and Noisy Student are about equally preferred while cross-split labeling and no regularization term perform better. We thus recommend future work to focus on the development of methods that achieve consistent and robust performance across tasks. We summarize the following findings: Finding 3. Gains of different methods are largely complementary. A combination of methods largely outperforms individual methods, performing close to a strong fully-supervised baseline on RoBERTa. However, there is still a sizeable gap between the best few-shot and the fully-supervised system. Finding 4. No single few-shot method dominates most NLU tasks. This highlights the need for the development of few-shot methods with more consistent and robust performance across tasks. We open-source FewNLU, an integrated toolkit designed for few-shot NLU. It contains implemen-9 Note that the gap could be larger since RoBERTa-Large has a smaller number of parameters than DeBERTa, and RoBERTa (fully-sup) does not incorporate additional beneficial techniques such as ensembling or self-training. tations of state-of-the-art methods, data processing utilities, a standardized few-shot training framework, and most importantly, our proposed evaluation framework. Figure 2 shows the architecture. We hope FewNLU could facilitate benchmarking few-shot learning methods for NLU tasks and ex-pendit the research in this field. We introduce an evaluation framework, re-evaluate a number of few-shot learning methods under the evaluation framework with a novel Multi-Splits strategy, and release a few-shot toolkit. Apart from this, we also aim at advancing the development of few-shot learning by sharing several new experimental findings. We identify several new directions for future work: (1) In practice, how to define the hyper-parameter search space a priori is a challenge. (2) It is critical for the community to iterate and converge on a common evaluation framework. (3) Few-shot natural language generation might also be studied in a similar framework. We thank Dani Yogatama for valuable feedback on a draft of this paper. Tang is funded by NSFC for Distinguished Young Scholar (61825602). Zheng, Ding, Tang, and Yang are funded by the National Key R&D Program of China (2020AAA0105200) and supported by Beijing Academy of Artificial Intelligence (BAAI). Zheng is Funded by China Postdoctoral Science Foundation (2021M690471). Zhou and Li are supported in part by the National Natural Science Foundation of China Grant 62161146004, Turing AI Institute of Nanjing and Xi'an Institute for Interdisciplinary Information Core Technology.
abstain abstain result objective other method abstain abstain abstain objective abstain other abstain abstain abstain abstain objective abstain abstain method objective result abstain method objective objective objective objective abstain objective objective objective abstain abstain abstain abstain abstain abstain other other other other other objective abstain other other other objective other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method method abstain method method abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain method method abstain method method abstain method abstain result abstain abstain result abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain method abstain abstain abstain abstain objective objective objective abstain abstain other other other other other