Prompt-based tuning for pre-trained language models (PLMs) has shown its effectiveness in few-shot learning. Typically, prompt-based tuning wraps the input text into a cloze question. To make predictions, the model maps the output words to labels via a verbalizer , which is either manually designed or automatically built. However, manual verbalizers heavily depend on domain-specific prior knowledge and human efforts, while finding appropriate label words automatically still remains challenging. In this work, we propose the prototypical verbalizer (ProtoVerb) which is built directly from training data. Specifically, ProtoVerb learns prototype vectors as verbalizers by contrastive learning. In this way, the prototypes summarize training instances and are able to enclose rich class-level semantics. We conduct experiments on both topic classification and entity typing tasks, and the results demonstrate that ProtoVerb significantly outperforms current automatic verbalizers, especially when training data is extremely scarce. More surprisingly, ProtoVerb consistently boosts prompt-based tuning even on untuned PLMs, indicating an elegant non-tuning way to utilize PLMs. Our codes are avaliable at https: //github.com/thunlp/OpenPrompt . The massive-scale pre-trained language models (PLMs) (Han et al., 2021a) have been proven to be backbones for solving a variety of NLP tasks (Kowsari et al., 2019; Rajpurkar et al., 2016). To further adapt these PLMs to downstream tasks such as classification, traditional approaches fine-tune the language models through an extra classifier (Howard and Ruder, 2018). However, when task-specific data is limited (Bragg et al., 2021), training the extra classifier effectively is challenging due to the gap between pre-training tasks (e.g.,  Corresponding author: Z.Liu (liuzy@tsinghua.edu.cn) A [MASK] news : Tokyo Olympic Daily Preview, July 26th. masked language modeling) and fine-tuning tasks (e.g., classification and regression). This gap impedes the fast adaptation of PLMs to downstream tasks. Recently, prompt-based tuning (Schick and Schtze, 2021; Liu et al., 2021) has risen to be a powerful way for few-shot learning by bridging the gap between the pre-training stage and downstream task stage. In prompt-based tuning, the input texts are wrapped with task-specific templates to re-formalize the original task as a cloze-style task. For example, in topic classification task, we can use template  <text> This topic is about [MASK] , where <text> is the placeholder for input sentences. The PLMs are asked to infer the words to fill in [MASK] and the words are further mapped to corresponding labels through a verbalizer (e.g. sports for label Sports). Verbaliz-7014 ers are of great importance in prompt-based tuning (Gao et al., 2021) since they are the bridges between model outputs and the final predictions. How to build effective verbalizers for prompt-based tuningespecially for many-class classification, is a critical issue in prompt-based tuning. Typically, most current works adopt three kinds of verbalizers: manual verbalizers, search-based verbalizers, and soft verbalizers. We show them by an example in Figure 1. Human-designed manual verbalizers pick some label words (e.g. label names) to depict classes. These verbalizers are powerful across multiple tasks (Schick and Schtze, 2021). Despite their success, a major drawback roots in the strong assumption that we own precise understandings of downstream tasks and are able to sum up each class with several words. Without task-specific prior knowledge, selecting appropriate label words is non-trivial. Further, they also need intensive human labors when facing many classes. To mitigate these issues, search-based verbalizers aim at finding suitable label words from vocabulary with algorithms (Schick et al., 2020; Shin et al., 2020; Gao et al., 2021) and soft verbalizers use trainable tokens which are optimized during tuning (Hambardzumyan et al., 2021; Zhang et al., 2021). However, it is challenging to search or optimize adequately in a large vocabulary or embedding space under a low-data regime, making automatic verbalizers suboptimal compared with manual ones. Intuitively, class proxies in verbalizers should encapsulate class-level semantic features, which are expressed implicitly by instances. To obtain these semantic representatives with few data, one promising approach is computing central points of class instances, namely prototypes , as approximation. To this end, we manage to estimate prototype vectors for each class to serve as verbalizer. Summarized from instances, prototypes are supposed to establish concepts similar with human-designed labels. In this work, we introduce prototypes into this problem and propose prototypical verbalizer (Pro-toVerb), which learns class prototypes from training data to build verbalizers automatically. For prototype learning, inspired by the idea of PCL (Li et al., 2021), ProtoVerb trains the prototype vectors by contrastive learning with the InfoNCE estimator (Oord et al., 2018). Specifically, our optimization objective includes two components: The first part is an instance-instance loss to cluster intra-class instances and separate inter-class instances; The second part is an instance-prototype loss which enforces the prototypes to be center points of classes. Compared with other verbalizer construction methods, ProtoVerb learns continuous vectors straight from training instances efficiently, which makes it a plug-in-and-play algorithm with high flexibility. To verify the effectiveness of ProtoVerb, we conduct extensive experiments on topic classification and entity typing tasks. We study two different settings where ProtoVerb can work: (1) When manual verbalizers are available, ProtoVerb can play as an extra verbalizer in the inference stage. Results show that ProtoVerb consistently improves the classification performance with low cost, and even untuned PLMs benefit largely. (2) Consider a realistic setting where only a limited number of samples are provided with no manual verbalizers, ProtoVerb also produces verbalizers of high quality. Experimental results demonstrate that ProtoVerb significantly outperforms existing search-based and soft verbalizers. Despite the success of PLMs (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2019) in massive NLP tasks, few-shot fine-tuning of PLMs was suboptimal due to the gap between pre-training and downstream tasks. Inspired by the in context learning proposed by GPT-3 (Brown et al., 2020), stimulating model knowledge with a few prompts has recently received much attention. A series of prompt-based work on knowledge probing (Trinh and Le, 2018; Petroni et al., 2019; Davison et al., 2019), text classification (Schick and Schtze, 2021; Gao et al., 2021), relation extraction (Han et al., 2021b), and entity typing (Ding et al., 2021a) emerge and achieve impressive progress. Typically, a piece of prompt contains a template and a verbalizer. Early prompts employ human-picked prompts which demand human knowledge and manual efforts. To alleviate this issue, later works explore automatic designing and optimizing prompts (Liu et al., 2021; Gao et al., 2021; Zhang et al., 2021). Recently research works further propose continuous prompts to replace the discrete phrases (Lester et al., 2021; Li and Liang, 2021). However, the designation of verbalizers, an important part of prompts, is less ex-7015 plored. Verbalizers bridge between model outputs and labels and make great impact on prompt-based tuning (Gao et al., 2021). With task-specific knowledge, human-picked words are widely used and proved effective (Schick and Schtze, 2021). The major drawback of manual verbalizers is the assumption that we possess sufficient knowledge of downstream tasks, which is not always satisfied. To avoid intensive human labor and expert knowledge dependency in manual verbalizers, some works explore search-based verbalizers (Schick et al., 2020; Gao et al., 2021; Shin et al., 2020) that identify label words automatically with training data. However, with a large vocabulary and few examples, it is non-trivial to find suitable words. Another line of researches focuses on soft verbalizers (Ham-bardzumyan et al., 2021; Zhang et al., 2021), which insert continuous embeddings as soft labels. The label embeddings are optimized along with model tuning. Similarly, soft verbalizers require abundant data for sufficient optimization, which can not be satisfied with the few-shot setting. In contrast, our approach learns prototype vectors from scratch, hence is more effective for few-shot tuning. In few-shot learning, prototype-based metric-learning methods have been promising approaches for their simplicity and effectiveness. Prototypical Networks (ProtoNet) (Snell et al., 2017) is the pioneering work that introduces prototypes into deep learning. Specifically, ProtoNet calculates prototype vectors by taking the average of instance vectors and makes predictions by metric-based comparisons between prototypes and query instances. A set of following works concentrates on the advancement of prototype estimation (Li et al., 2021; Gao et al., 2019; Ding et al., 2021c). Among them, PCL (Li et al., 2021) achieves remarkable results on self-supervised few-shot learning by using prototypes as latent variables and inspires us in designing training objectives. The success of prototype-based models indicates that prototypes, which are representative embeddings of instances from the same classes, encapsulate some class-level semantic features. Inspired by the intrinsic similarity of prototypes and verbalizers, we find it natural and elegant to introduce prototypes into verbalizer construction for prompt-based tuning. Given a pre-trained language model M , our goal is to tune it for specific downstream tasks. Take N way K shot few-shot text classification as an example, the support set for class n D n = { x n 1 ,    , x nK } contains K sentences. We aim to predict the label y  Y for each sentence, where Y is the label set with N distinct classes. For a sentence concatenated with special tokens x = { [CLS] , t 1 ,    , t T , [SEP] } , language model M encodes it into hidden representations { h [CLS] , h 1 ,    , h T , h [SEP] } . Conventional fine-tuning trains an extra classifier F over the [CLS] embedding h [CLS] and output the probability distribution on label set Y . The vanilla prompt-based tuning converts the downstream task to a cloze-style mask language modeling problem. For example, to formulate the text classification task, we can modify the original input x with a template T (  ) = A [MASK] news: to get the prompt input T ( x ) = A [MASK] news: x . With T ( x ) , M produces the hidden vector at the [MASK] position h [MASK] . To calculate the probability distribution over the label set, a manual verbalizer stores a set of label words V and the score for label y is PM ( y | x ) = g ( PM ( [MASK] = v |T ( x )) | v  V y ) , (2) where V y is the label words of y and g (  ) is to aggregate multiple scores. In previous sections, we introduce the general pipeline of prompt-based tuning. As manually defining or automatically searching for appropriate verbalizers can be challenging, here we propose to learn prototypes directly from training instances. Inspired by PCL (Li et al., 2021), the prototypes are trained with contrastive learning. As shown in 7016 A [MASK] news : Stocks Fall as Oil Hits High. Figure 2, we first get the hidden states of [MASK] tokens to represent instances, then project them to another embedding space for prototype learning. The prototypes are used as verbalizers for prediction. Next, we will introduce the learning and inference stages of ProtoVerb in detail. Given a piece of training text x wrapped with a template, we take the last layer's hidden state of the [MASK] token h [MASK] as the initial representation of the text. With an encoder E  (  ) parameterized by  , the instance representation of x is v = E  ( x ) = Wh [MASK] . In practice, we simply adopt a linear encoder with weight W . To measure the similarity between instances, we adopt cosine similarity function S (  ) , where S ( v i , v j ) = v i || v i ||  v j || v j || . With the instance representation and similarity function, we discuss how to define our training objective. Denote C = { c 1 ,    , c N } as the set of prototype vectors. Intuitively, there are two goals we need to achieve by optimization: (1) For instance-instance pairs, intra-class pairs should get higher similarity scores than inter-class pairs. (2) For instance-prototype pairs, the similarity scores between prototype c n and instances of class n should be higher than c n and other instances. To realize these two goals, we define the objective function based on the InfoNCE estimator (Oord et al., 2018), which is widely adopted in contrastive learning. For the instance-instance objective, we minimize the following loss function L ins =  1 N 2 K 2 (cid:88) n (cid:88) i,j log exp S ( v ni , v nj ) (cid:80) n  ,j  exp S ( v ni , v n  j  ) , (5) where ( v ni , v nj ) are instance pairs of the same class. This loss function maximizes intra-class similarity and minimizes inter-class similarity between instances. Similarly, the instance-prototype loss function is defined as L proto =  1 N 2 K (cid:88) i,n log exp S ( v ni , c n ) (cid:80) n  exp S ( v ni , c n  ) , (6) and v ni is of class n . This objective forces each prototype to lie at the center point of its instances. Overall, combining the instance-instance loss and instance-prototype loss, our final training objective is L = L ins + L proto . During inference, following the same metric, we calculate the similarity scores of query and prototypes. The probability score for class k is PM ( y k | x ) = exp S ( v , c k ) (cid:80) k  exp S ( v , c k  ) . (8) Then we make prediction by arg max function (cid:101) y = arg max k PM ( y k | x ) . (9) When there are other verbalizers (e.g. manual verbalizers), we first process the logits from different verbalizers with a standard scaler (minus mean then divide by standard deviation). Then we take the mean value of the scores to get the final score. We conduct extensive few-shot learning experiments to illustrate the effectiveness of ProtoVerb. In this section, we first introduce the experimental settings in use. Then we present and discuss the experiment results. Verbalizers in many-class classification tasks are difficult to get precise definitions. Hence we adopt three topic classification datasets: AG's News, Yahoo (Zhang et al., 2015), and DBPedia (Lehmann et al., 2015) and one entity typing dataset: FewNERD (Ding et al., 2021d) as benchmarks, and their statistics are summarized in Table 1. To focus on the verbalizer and alleviate the influence of templates, we adopt multiple fixed manual templates. For topic classification, following (Hu et al., 2021), we use four templates on each dataset. For entity typing, we use three templates from (Ding et al., 2021a). Details about the templates can be found in Appendix A. Dataset Task #Class #Test AG's News TC 4 7,600 DBPedia TC 14 70,000 Yahoo TC 10 60,000 FewNERD ET 66 96,901 Table 1: Dataset statistics. Under the few-shot setting, we randomly sample k = 1 , 2 , 4 , 8 , 16 instances in each class from the training set and test the model on the entire test set. As for the evaluation metric, we use accuracy in all experiments. For the different usages of ProtoVerb, we consider two specific settings: (1) ProtoVerb as a single verbalizer ( 5.5). When manual verbalizers are not available, we can tune the model with ProtoVerb. Under this setting, we want to evaluate the performance of ProtoVerb compared with other automatic verbalizer construction methods. (2) ProtoVerb as an extra verbalizer ( 5.6). Naturally, we suppose that there exists a manual verbalizer and we append ProtoVerb to strengthen the performance. Under this setting, ProtoVerb is a plug-in-and-play component and does not participate in the tuning process. We compare ProtoVerb with manual verbalizers and other verbalizer ensembles. All our models and baselines are implemented with PyTorch (Paszke et al., 2019) framework, Hugging-face transformers (Wolf et al., 2020), and OpenPrompt toolkit (Ding et al., 2021b). We optimize PLMs with AdamW optimizer (Loshchilov and Hutter, 2019). For prototype learning, we set the prototype dimension to 128 and optimize the loss function with Adam optimizer (Kingma and Ba, 2015). For topic classification, we use RoBERTa-large (Liu et al., 2019) as our PLM backbone and tune the model for 5 epochs. The batchsize is 2 and the learning rate is 3e-5. For entity typing, we tune a BERT-base (Devlin et al., 2019) model for 30 epochs and set the batchsize to 16. The learning rate here is 5e-5. The vanilla prompt-based tuning method fuses the input text with a task-specific template and maps the model outputs to labels through a verbalizer. For fair comparisons, all our baselines and proposed models are built on this pipeline and they merely differ from the verbalizers in use. Manual verbalizers (ManualVerb) are defined by human with domain knowledge. Here we simply employ the verbalizers provided by OpenPrompt (Ding et al., 2021b). Search-based verbalizers (SearchVerb) search for suitable words from vocabulary automatically. We adopt the implementation in PETAL (Schick et al., 2020), which finds the words that maximize the likelihood of the training data. To combine SearchVerb with ManualVerb, we merge their verbalizer words together. Soft verbalizers (SoftVerb) introduce trainable tokens as verbalizers in prompt-based tuning. We follow the approach in WARP (Hambardzumyan et al., 2021) that applies soft tokens as a linear decoding layer, and the token embeddings are learned along with model tuning. Note that the templates in WARP are also trainable, but here we only use its soft verbalizers. In single verbalizer experiments, we initialize the token embeddings randomly for fairness. And in extra verbalizer experiments, they are initialized with label names. Table 2 presents the performance of different verbalizers. Overall, ManualVerb is the most powerful verbalizer, which is reasonable because it is picked by human with domain knowledge. ProtoVerb outperforms SearchVerb and SoftVerb remarkably and consistently, especially when only 1 or 2 instances per class are given. The poor performances of the two baselines under extreme data scarcity corroborate the issues we claim in  1. As the training data become sufficient, ProtoVerb gets comparable or even exceeding scores compared with ManualVerb, showing that ProtoVerb is able to learn prototypes that well represent the classes. At the same time, the gaps between ManualVerb and other verbalizers narrow, which also indicates that we can summarize data across various ways. Across tasks, ProtoVerb gets better results on topic classification than entity typing. A possible reason is that FewNERD is a fine-grained entity typing dataset, in which the differences across classes are subtle. For example, it is hard for ProtoVerb K Method AG DB Yahoo Few 0 ManualVerb 75.13 67.06 43.11 20.00 1 Fine-tuning 25.45 10.80 10.59 7.48 ManualVerb 76.67 85.47 50.22 41.68 SearchVerb+ 51.82 81.31 43.24 35.64 SoftVerb+ 76.34 85.85 49.11 37.66 ProtoVerb+ 77.71 88.16 50.08 43.20 w/o tuning 76.28 78.32 45.01 29.51 2 Fine-tuning 25.78 49.01 11.26 19.03 ManualVerb 81.06 93.61 58.65 46.44 SearchVerb+ 77.56 91.79 52.46 42.13 SoftVerb+ 79.95 93.68 55.73 42.17 ProtoVerb+ 84.09 94.77 59.33 48.69 w/o tuning 82.13 86.11 50.34 34.44 4 Fine-tuning 28.14 94.08 26.02 20.98 ManualVerb 84.73 95.83 61.41 52.54 SearchVerb+ 81.25 95.16 58.98 50.61 SoftVerb+ 84.22 94.90 59.01 49.45 ProtoVerb+ 85.71 96.74 66.14 54.16 w/o tuning 83.05 89.56 55.59 35.55 8 Fine-tuning 72.78 96.83 54.76 49.77 ManualVerb 85.85 96.46 64.12 56.59 SearchVerb+ 85.68 97.57 65.32 56.58 SoftVerb+ 86.54 97.40 63.48 54.30 ProtoVerb+ 87.25 97.64 66.61 58.30 w/o tuning 83.79 92.61 59.42 34.37 16 Fine-tuning 84.14 97.25 64.27 52.66 ManualVerb 84.74 96.05 58.77 61.17 SearchVerb+ 85.30 95.08 59.34 61.70 SoftVerb+ 85.65 96.34 58.68 59.23 ProtoVerb+ 87.98 97.22 65.65 62.55 w/o tuning 84.78 93.46 60.89 33.96 Table 3: Results for multiple verbalizer experiments. to discriminate between person-artist/author and person-director with only a few instances. However, ProtoVerb can also catch up with ManualVerb with enough samples. Table 3 shows the experiment results when we ensemble manual verbalizers with automatic verbalizers. The ensembled versions are denoted as SearchVerb+, SoftVerb+, and ProtoVerb+ respectively. From the table, we have the following observations: (1) Basically, prompt-based tuning outperforms fine-tuning by a large margin with few 7019 samples (1  2 per class). When sufficient training data is available, fine-tuning models will produce comparable results. (2) Overall, ProtoVerb+ certainly improves the performance of prompt-based tuning under most cases, which demonstrates the effectiveness of ProtoVerb+. At the same time, SearchVerb+ and SoftVerb+ seldom show enhancement compared with ManualVerb. As ProtoVerb+ does not introduce any external knowledge, this illustrates that ProtoVerb+ provides a better way to utilize training data. Finally, we also present the results of applying ProtoVerb+ on untuned PLMs. It is worth noting that even for untuned models, ProtoVerb+ also boosts them considerably on all tasks. For example on DBPedia, showing only one instance per class to PLMs with ProtoVerb+ leads to 11.26% absolute accuracy improvement. On topic classification, when more training samples are given, untuned PLMs achieve competitive scores. This observation indicates a new cost-efficient way to leverage training data, which we highlight as valuable for future study of none-tuning methods for PLMs. Compared to the in context learning in GPT-3 (Brown et al., 2020), ProtoVerb+ is not limited by input length and can deal with arbitrary number of samples. We further study this fixed model scenario in  6.1. In this section, we discuss several analytical top-ics for further understandings of ProtoVerb. For simplicity, we conduct experiments on AG's News dataset. In  5.6, we see ProtoVerb is still powerful with fixed PLMs. For further comparisons, we conduct experiments to quantitatively evaluate verbalizers when PLMs are fixed. Figure 3 gives the results. To clarify, using ManualVerb on fixed PLMs equals the zero-shot setting, which we plot with a dashed line. Meanwhile, different from  5.6, ProtoVerb here is a single verbalizer. From the figure we can conclude that (1) Similar with  5.5, ProtoVerb outperforms SoftVerb and SearchVerb by a large margin under low-shot settings. Notably, ProtoVerb exceeds ManualVerb with only 2 shots per class, illustrating the experessive power of prototypes. (2) SoftVerb is also better than SearchVerb under this setting, demonstrating that tunable verbalizers could exploit training data better with PLMs fixed. To validate the effect of each part in the loss function, we conduct an ablation study on AG's News dataset. For comparison, we consider two variants of prototype calculation methods: (1) ProtoVerb with L proto only. (2) Following ProtoNet (Snell et al., 2017), take the average of instance embeddings for prototype embeddings. Table 4 shows the results. Compared to taking the mean embedding 7020 Class K = 1 K = 16 World Qaida, Syria, Iraq, Nusra, TPP Taliban, Iraq, Afghan, militants, rebellion Sports Steelers, Raptors, Knicks, Dodgers ball, ESPN, baseball, Fifa, Sports Business cash, earnings, Securities, NYSE Dow, dividend, investing, markets Tech LTE, Tel, Huawei, Mbps, VPN Vault, IBM, Qualcomm, Technologies Table 6: Words that are most similar with prototypes of each class on AG's News. vectors directly, optimizing the embedding vectors of prototypes using our loss functions leads to better performances and stability. Adding L ins is also beneficial, meaning that L ins helps ProtoVerb in learning instance embeddings. Noisy data are commonly seen as threats in real-world datasets for few-shot learning systems. For automatic verbalizers, noisy data are more harmful because of the effect on both the quality of verbalizers and the training process. In this section, we evaluate the robustness of different automatic verbalizers against noisy samples on AG's News. For training stability, we set K = 8 , 16 . Table 5 presents the accuracy drop when there are 1, 2, or 3 samples having wrong labels. It is clearly seen that a limited number of noisy samples will hinder the performance greatly, showing the vulnerability of automatic verbalizers. Meanwhile, we can also find that ProtoVerb is more robust than baseline methods when facing noisy samples. Since ProtoVerb learns continuous prototype vectors, their meanings are implicit. Here we manage to investigate which words are most similar to the learned prototypes. Due to word embeddings and prototype vectors lying in different embedding spaces, we can not directly calculate their similarity. Hence we use the vocabulary as the input texts (one word at a time) to get the top-scored word for each class. On AG's News dataset, we collect some most similar words for each class and list them in Table 6. To investigate the property of prototypes learned with different numbers of samples, we present words for K = 1 and K = 16 . With the table, we see that: (1) Even when only one example is available, the learned prototypes are meaningful. Most of the similar words are proper nouns and entity names closely related to class topics. For example, Steelers, Raptors, Knicks, and Dodgers are all baseball or basketball teams that appear frequently in sports news. We attribute this to prompt mechanism that allows PLMs to extract the most conclusive information and fill the [MASK] with it. Then the relevant words are also included. (2) With more training instances, prototypes show diverse interests. Despite entity names, more con-ceptual words show up on the list, such as ball and Sports for class Sports. We interpret this as the summarization and abstraction ability of prototypes. Given many instances, prototypes are enforced to capture their common features, hence some abstract concepts are found automatically. In this way, ProtoVerb encapsulates class-level, rather than entity-level, semantics, which leads to better performance on unseen data. To give further analyses for the inner workings of prototypes, we measure the similarity between ProtoVerb and ManualVerb to see whether ProtoVerb is able to learn abstract concepts as humans do. On AG's News dataset, we calculate the similarity scores between prototypes and manual verbalizers and normalize the scores using the softmax function across the four classes. In Figure 4 we plot the scores with various shots. It is clearly seen that the similarity of prototypes and corresponding verbal-7021 izers are above average (0.25). As shot increases, the scores also gradually grow, which illustrates that prototypes can capture the conceptual information better from more instances. This observation matches our findings in  6.4. Among the four classes, Business and Sports get higher scores than World and Tech. A reasonable guess is that World and Tech news includes diverse sub-topics that are hard to summarize. In this paper, we propose a novel approach for automatic verbalizer construction in prompt-based tuning. The proposed ProtoVerb learns class prototypes from training instances using contrastive learning. We explore the performance of ProtoVerb on few-shot topic classification and entity typing tasks. As a single verbalizer, ProtoVerb outperforms state-of-the-art automatic verbalizers considerably. Working together with manual verbalizers, ProtoVerb can also consistently improve prompt-based tuning with minor effort. The results validate the effectiveness of ProtoVerb. Our analysis further reveals the intrinsic properties of prototypes. For future work, we will focus on extending ProtoVerb for effective non-tuning algorithms of PLMs and prompt-tuning with soft templates. Moreover, we are finding proper ways to combine label words and prototypes for verbalizer construction. This work is supported by the National Key R&D Program of China (No. 2020AAA0106502), Institute for Guo Qiang at Tsinghua University, Beijing Academy of Artificial Intelligence (BAAI), and International Innovation Center of Tsinghua University, Shanghai, China. Ganqu Cui and Shengding Hu conducted the experiments. Ganqu Cui, Shengding Hu, Ning Ding and Zhiyuan Liu wrote the paper. Longtao Huang provided valuable advices to the research. Our work explores how to stimulate large PLMs with few samples. We conduct experiments under the few-shot setting, where requires less training time and fewer resources than normal full-data setting. Also, we open up our codes and hyperpa-rameters to facilitate future reproduction without repeated energy cost.
abstain abstain abstain abstain objective abstain abstain objective abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective abstain result method abstain abstain abstain other other other other other other other other other other abstain other other other other other objective other other other other abstain other abstain objective other objective other other other method other other abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain method abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain result abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain method result abstain abstain abstain abstain result abstain abstain method method abstain abstain other method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain objective abstain objective abstain abstain abstain result objective result other other other other abstain method abstain