Tuning pre-trained language models (PLMs) with task-specific prompts has been a promising approach for text classification. Particularly, previous studies suggest that prompt-tuning has remarkable superiority in the low-data scenario over the generic fine-tuning methods with extra classifiers. The core idea of prompt-tuning is to insert text pieces, i.e., template, to the input and transform a classification problem into a masked language modeling problem, where a crucial step is to construct a projection, i.e., verbalizer, between a label space and a label word space. A verbalizer is usually handcrafted or searched by gradient descent, which may lack coverage and bring considerable bias and high variances to the results. In this work, we focus on incorporating external knowledge into the verbalizer, forming a knowledgeable prompt-tuning (KPT), to improve and stabilize prompt-tuning. Specifically, we expand the label word space of the verbalizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the expanded label word space. Extensive experiments on zero and few-shot text classification tasks demonstrate the effectiveness of knowledgeable prompt-tuning. Our source code is publicly available at https://github.com/ thunlp/KnowledgeablePromptTuning . Recent years have witnessed the prominence of Pre-trained Language Models (PLMs) (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Raffel et al., 2020; Xu et al., 2021) due to their superior performance on a wide range of language-related downstream tasks such as text classification (Kowsari et al., 2019), question answering (Ra-jpurkar et al., 2016), and machine reading comprehension (Nguyen et al., 2016). To fathom the prinCorresponding authors: Z.Liu (liuzy@tsinghua.edu.cn), H.Wang (huadw2012@163.com) ciples of such effectiveness of PLMs, researchers have conducted extensive studies and suggested that PLMs have obtained rich knowledge during pre-training (Petroni et al., 2019; Davison et al., 2019). Hence, how to stimulate and exploit such knowledge is receiving increasing attention. One conventional approach to achieve that is fine-tuning (Devlin et al., 2019), where we add extra classifiers on the top of PLMs and further train the models under classification objectives. Fine-tuning has achieved satisfying results on supervised tasks. However, since the extra classifier requires adequate training instances to tune, it is still challenging to apply fine-tuning in few-shot learning (Brown et al., 2020) and zero-shot learning (Yin et al., 2019) scenarios. Originated from GPT-3 (Brown et al., 2020) and LAMA (Petroni et al., 2019, 2020), a series of studies using prompts (Schick and Schtze, 2021a; Liu et al., 2021) for model tuning bridge the gap between pre-training objective and down-stream tasks, and demonstrate that such discrete or continuous prompts induce better performances for PLMs on few-shot and zero-shot tasks. A typical way to use prompts is to wrap the input sentence into a natural language template and let the PLM conduct masked language modeling. For instance, to classify the topic of a sentence x : What's the relation between speed and accelera-tion? into the S CIENCE  category, we wrap it into a template: A [MASK] question: x . The prediction is made based on the probability that the word science is filled in the  [MASK]  token. The mapping from label words (e.g., science ) to the specific class (e.g., class SCIENCE ) is called the verbalizer (Schick and Schtze, 2021a), which bridges a projection between the vocabulary and the label space and has a great influence on the performance of classification (Gao et al., 2021). Most existing works use manual verbalizers (Schick and Schtze, 2021a,b), in which the 2225 designers manually think up a single word to indicate each class. To ease the human effort of designing the class name, some works propose to learn the label words using discrete search (Schick et al., 2020) or gradient descent (Liu et al., 2021; Ham-bardzumyan et al., 2021). However, the learned-from-scratch verbalizer, lack of human prior knowledge, is still considerably inferior to the manual verbalizers (see Appendix A for pilot experiments), especially in few-shot setting, and even not applicable in zero-shot setting, which leaves the manual verbalizer a decent choice in many cases. However, manual verbalizers usually determine the predictions based on limited information. For instance, in the above example, the mapping {science}  SCIENCE means that only predicting the word science for the [MASK] token is regarded as correct during inference, regardless of the predictions on other relevant words such as physics and maths, which are also informative. Such handcrafted one-one mapping limits the coverage of label words, thus lacking enough information for prediction and introducing bias into the verbalizer. Therefore, manual verbalizers are hard to be optimal in text classification, where the semantics of label words are crucial for predictions. The optimization-based expansion, though can be combined with manual verbalizers to yield better performance, only induces a few words or embeddings that are close to the class name in terms of word sense or embedding distance. Thus they are difficult to infer words across granularities (e.g. from science to physics). If we can expand the verbalizer of the above example into { science, physics }  SCIENCE , the probability of making correct predictions will be considerably enhanced. Therefore, to improve the coverage and reduce the bias of the manual verbalizer, we present to incorporate external knowledge into the verbalizers to facilitate prompt-tuning, namely, knowledgeable prompt-tuning (KPT). Since our expansion is not based on optimization, it will also be more favorable for zero-shot learning. Specifically, KPT contains three steps: construction, refinement, and utilization. (1) Firstly, in the construction stage, we use external KBs to generate a set of label words for each label (in  3.2). Note that the expanded label words are not simply synonyms of each other, but cover different granularities and perspectives, thus are more comprehensive and unbiased than the class name. (2) Secondly, to cope with the noise in the unsupervised expansion of label words, we propose four refinement methods, namely, frequency refinement, relevance refinement, contextualized calibration, and learnable refinement (in  3.3), whose effectiveness is studied thoroughly in  4. (3) Finally, we apply either a vanilla average loss function or a weighted average loss function for the utilization of expanded verbalizers, which map the scores on a set of label words to the scores of the labels. We conduct extensive experiments on zero-shot and few-shot text classification tasks. The empirical results show that KPT can reduce the error rate of classification by 16%, 18%, 10%, 7% on average in 0, 1, 5, 10 shot experiments, respectively, which shows the effectiveness of KPT. In addition to the performance boost, KPT also reduces the prediction variances consistently in few-shot experiments and yields more stable performances. Two groups of research are related to KPT: prompt-tuning, and the verbalizer construction. Prompt-tuning. Since the emergence of GPT-3 (Brown et al., 2020), prompt-tuning has received considerable attention. GPT-3 (Brown et al., 2020) demonstrates that with prompt-tuning and in-context learning, the large-scale language models can achieve superior performance in the low-data regime. The following works (Schick and Schtze, 2021a,b) argue that small-scale language models (Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020) can also achieve decent performance using prompt-tuning. Prompt-tuning has been applied to a large variety of tasks such as Text Classification (Schick and Schtze, 2021a), Natural Language Understanding (Schick and Schtze, 2021b; Liu et al., 2021) , Relation Extraction (Han et al., 2021; Chen et al., 2021), and Knowledge Probing (Petroni et al., 2019; Liu et al., 2021), etc. Verbalizer Construction. As introduced in  1, the verbalizer is an important component in prompt-tuning and has a strong influence on the performance of prompt-tuning (Holtzman et al., 2021; Gao et al., 2021). Most works use human-written verbalizers (Schick and Schtze, 2021a), which are highly biased towards personal vocabulary and do not have enough coverage. Some other studies (Gao et al., 2021; Shin et al., 2020; Liu et al., 2021; Schick et al., 2020) design automatic verbalizer searching methods for better ver-2226 science, mathematics, biology, research, knowledge, physics, knowledge, innovate, calculation, ..., agrobiology, mscience, euclid, orthogon, einstein, bioacoustics, chemo, axiom, nomenclature What's the relation between speed and acceleration? balizer choices, however, their methods require adequate training set and validation set for optimization. Moreover, the automatically determined verbalizers are usually synonym of the class name, which differs from our intuition of expanding the verbalizer with a set of diverse and comprehensive label words using external KB. Schick et al. (2020) and Shin et al. (2020) also try multiple label words for each class. The optimal size of their label words set for each class is generally less than 10, which lacks coverage when used in text classification tasks. In this section, we present our methods to incorporate external knowledge into a prompt verbalizer. We first introduce the overall paradigm of prompt-tuning and then elucidate how to construct, refine and utilize the knowledgeable prompt. Let M be a language model pre-trained on large scale corpora. In text classification task, an input sequence x = ( x 0 , x 1 , ..., x n ) is classified into a class label y  Y . Prompt-tuning formalizes the classification task into a masked language modeling problem. Specifically, prompt-tuning wraps the input sequence with a template , which is a piece of natural language text. For example, assuming we need to classify the sentence x =What's the relation between speed and acceleration? into label SCIENCE (labeled as 1) or SPORTS (labeled as 2), we wrap it into x p = [CLS] A [MASK] question : x Then M gives the probability of each word v in the vocabulary being filled in [MASK] token PM ( [MASK] = v | x p ) . To map the probabilities of words into the probabilities of labels, we define a verbalizer as a mapping f from a few words in the vocabulary, which form the label word set V , to the label space Y , i.e., f : V (cid:55) Y . We use V y to denote the subset of V that is mapped into a specific label y ,  y Y V y = V . Then the probability of label y , i.e., P ( y | x p ) , is calculated as P ( y | x p )= g (cid:0) PM ( [MASK] = v | x p ) | v  V y (cid:1) , (1) where g is a function transforming the probability of label words into the probability of the label. In the above example, regular prompt-tuning may define V 1 = { science } , V 2 = {sports } and g as an identity function, then if the probability of science is larger than sports, we classify the instance into SCIENCE . We propose KPT, which mainly focuses on using external knowledge to improve verbalizers in prompt-tuning. In KPT , we use KBs to generate multiple label words related to each class y , e.g., V 1 = {science,physics, ...}. And we propose four refinement methods to eliminate the noise in the expanded V . Finally, we explore the vanilla average and weighted average approaches for the 2227 utilization of the expanded V . The process of predicting masked words based on the context is not a single-choice procedure, that is, there is no standard correct answer, but abundant words may fit this context. Therefore, the label words mapped by a verbalizer should be equipped by two attributes: wide coverage and little subjective bias . Such a comprehensive projection is crucial to the imitation of pre-training, which is the essence of prompt-tuning. Fortunately, external structured knowledge could simultaneously meet both requirements. In this section, we introduce how we use external knowledge for two text classification tasks: topic classification and sentiment classification. For topic classification, the core issue is to extract label words related to the topic from all aspects and granularities. From this perspective, we choose Related Words 1 , a knowledge graph G aggregated from multiple resources, including word embeddings, ConceptNet (Speer et al., 2017), WordNet (Pedersen et al., 2004), etc., as our external KB. The edges denote "relevance" relations and are annotated with relevance scores. We presume the the name of each class v 0 is correct and use them as the anchor node to get the neighborhood nodes NG ( v 0 ) whose scores are larger than a threshold  as the related words 2 . Thus, each class is mapped into a set of label words V y = NG ( v 0 )  { v 0 } . For binary sentiment classification, the primary goal is to extend the binary sentiment to sentiment of more granualities and aspects. We use the sentiment dictionary summarized by previous researchers 3 , 4 . Several examples of the label words in the KPT are in Table 1. Although we have constructed a knowledgeable verbalizer that contains comprehensive label words, the collected label words can be very noisy since the vocabulary of the KB is not tailored for the PLM. Thus it is necessary to refine such verbalizer by retaining high-quality words. In this section, 1 https://relatedwords.org 2 We take  = 0 in the experiments 3 https://www.enchantedlearning.com/wordlist/positivewords.shtml 4 https://www.enchantedlearning.com/wordlist/negativewords.shtml we propose four refinement methods addressing different problems of the noisy label words. Frequency Refinement. The first problem is to handle the rare words. We assume that several words in the KB are rare to the PLM, thus the prediction probabilities on these words tend to be inaccurate. Instead of using a word-frequency dictionary, we propose to use contextualized prior of the label words to remove these words. Specifically, given a text classification task, we denote the distribution of the sentences x in the corpus as D . For each sentence in the distribution, we wrap it into the template and calculate the predicted probability for each label word v in the masked position PM ( [MASK] = v | x p ) . By taking the expectation of the probability over the entire distribution of sentences, we can get the prior distribution of the label words in the masked position. We formalize it as PD ( v )= E x D PM ( [MASK] = v | x p ) . Empirically, we found that using a small-size unlabeled support set  C sampled from the training set and with labels removed, will yield a satisfying estimate of the above expectation. Thus, assuming that the input samples { x   C} have a uniform prior distribution, the contextualized prior is approximated by PD ( v )  1 |  C| (cid:88) x   CPM ( [MASK] = v | x p ) . Then we remove the label words whose prior probabilities are less than a threshold. Details can be found in Appendix C. Relevance Refinement. As our construction of knowledgeable label words is fully unsupervised, some label words may be more relevant to their belonging class than the others. To measure the relevance of a label word to each class, we obtain the prediction probability of the label word on the support set  C as the vector representation q v of the label words, i.e., q v 's i -th element is q vi = PM ([ MASK ] = v | x i p ) , x i   C , (4) where x i p represents the sentence x i combined with the template p. To estimate the class's representation, we presume that the name of each class v 0 , such as science for SCIENCE , though lack of coverage, is very relevant to the class. Then we use the vector representation q v 0 of the these names as the 2228 Dataset Label Label Words AG's News POLITICS politics, government, diplomatic, law, aristotle, diplomatical, governance ... class's representation q y . Therefore the relevance score between a label word v and a class y is calculated as the cosine similarity between the two representation: r ( v, y ) = cos( q v , q y ) = cos( q v , q v 0 ) . Moreover, some label words may contribute positively to multiple classes, resulting in confusion between classes. For example, the potential label word physiology of class SCIENCE may also be assigned with a high probability in a sentence of class SPORTS . To mitigate such confusion and filter the less relevant label words, we design a metric that favors the label word with high relevance merely to its belonging class and low relevance to other classes: R ( v ) = r ( v, f ( v )) |Y|  1 (cid:80) y Y ,y  = f ( v ) ( r ( v, y )) , (6) where f ( v ) is the corresponding class of v . Ideally, a good label word should at least has a higher relevance score for its belonging class than the average relevance score for the other classes. Therefore, we remove the label words with R ( v ) < 1 . In practice, we have a slight modification to Equation (6), please refer to appendix C for details. Essentially, this Relevance Refinement adopts the idea of the classical TF-IDF (Jones, 1972) algorithm which estimates the relevance of a word to a document. It prefers to use a word that is relevant to a specific document while irrelevant to other documents as the keyword of the document. In KPT, a class is analogous to a document, while a label word is comparable to the word in the document. From this perspective, equation (6) is a variant of TF-IDF metric. Contextualized Calibration. The third problem is the drastic difference in the prior probabilities of label words. As previous works (Zhao et al., 2021; Holtzman et al., 2021) have shown, some label words are less likely to be predicted than the others, regardless of the label of input sentences, resulting in a biased prediction. In our setting, the label words in the KB tend to have more diverse prior probabilities, resulting in a severer problem (see Table 2). Therefore, we use the contextualized prior of label words to calibrate the predicted distribution, namely, contextualized calibration (CC):  PM ( [MASK] = v | x p )  PM ( [MASK] = v | x p ) PD ( v ) (7) where PD ( v ) is the prior probability of the label word. Learnable Refinement . In few-shot learning, the refinement can be strengthen by a learning process. Specifically we assign a learnable weight w v to each label word v (may be already refined by the previous methods). The weights form a vector w  R |V| , which is initialized to be a zero vector. The weights are normalized within each V y :  v = exp( w v ) (cid:80) u V y exp( w u ) . Intuitively, in the training process, a small weight is expected to be learned for a noisy label word to minimize its influence on the prediction. Note that in few-shot setting, calibration may not be necessary because the probability of a label word can be trained to the desired magnitude, i.e.,  PM ( [MASK] = v | x p ) = PM ( [MASK] = v | x p ) . In addition to these refinement methods, since many label words are out-of-vocabulary for the PLM and are split into multiple tokens by the tokenizer. For these words, we simply use the average prediction score of each token as the prediction score for the word. The influence of this simple approach is studied in Appendix D.3. The final problem is how to map the predicted probability on each refined label word to the decision of the class label y . predicting the label. Therefore, we use the average of the predicted scores on V y as the predicted score for label y . The predicted label  y is  y =argmax y Y (cid:80) v V y  PM ( [MASK] = v | x p ) |V y | . Weighted Average. In few-shot setting, supported by the Learnable Refinement, we adopt a weighted average of label words' scores as the prediction score. The refinement weights  v are used as the weights for averaging. Thus, the predicted  y is  y = argmax y Y exp (cid:0) s ( y | x p ) (cid:1) (cid:80) y  exp (cid:0) s ( y  | x p ) (cid:1) , (10) where s ( y | x p ) is s ( y | x p )= (cid:88) v V y  v log PM ( [MASK] = v | x p ) . (11) This objective function is suitable for continuous optimization by applying a cross-entropy loss on the predicted probability. We provide a theoretical illustration of the KPT framwork in Appendix B. We evaluate KPT on five text classification datasets to demonstrate the effectiveness of incorporating external knowledge into prompt-tuning. We carry out experiments on three topic classification datasets: AG's News (Zhang et al., 2015), DBPedia (Lehmann et al., 2015), and Yahoo (Zhang et al., 2015), and two sentiment classification datasets: IMDB (Maas et al., 2011) and Amazon (McAuley and Leskovec, 2013). The statistics of the datasets are shown in Table 7. The detailed information and the statistics of each dataset is in Appendix E. We test all prompt-based methods using four manual templates and report both the average results (with standard error) of the four templates and the results of the best template (shown in (brack-ets) ). The reasons for using manual templates and the specific templates for each dataset are in Appendix E. 4.2 Experiment Settings Our experiments are based on OpenPrompt (Ding et al., 2021), which is an open-source toolkit to conduct prompt learning. For the PLM, we use RoBERTa large (Liu et al., 2019) for all experiments. For test metrics, we use Micro-F1 in all experiments. For all zero-shot experiments, we repeat the experiments 3 times using different random seeds if randomness is introduced in the experiments, and for all few-shot experiments, we repeat 5 times. Note that considering the four templates and five/three random seeds, each reported score of prompt-based methods is the average of 20/12 experiments , which greatly reduces the randomness of the evaluation results. For the refinement based on the support set  C , the size of the unlabeled support set |  C| is 200. For few-shot learning, we conduct 1, 5, 10, and 20-shot experiments. For a k -shot experiment, we sample k instances of each class from the original training set to form the few-shot training set and sample another k instances per class to form the validation set. We tune the entire model for 5 epochs and choose the checkpoint with the best validation performance to test. Other hyper-parameters can be found in Appendix F. 4.3 Baselines In this subsection, we introduce the baselines we compare with. To better understand our proposed methods, we also compare within the performance of KPT using different configuration. Fine-tuning (FT). Traditional fine-tuning method inputs the hidden embedding of [CLS] token of the PLM into the classification layer to make predictions. Note that fine-tuning can not be applied to the zero-shot setting, since the classification layer is randomly initialized. Prompt-tuning (PT). The regular prompt-tuning method uses the class name as the only label word for each class, which is used in PET (Schick and Schtze, 2021a) and most existing works. For a fair comparison, we do not use the tricks in PET, such as self-training and prompt ensemble, which are orthogonal to our contributions. Automatic Verbalizer (AUTO). The automatic verbalizer is proposed by PETAL (Schick et al., 2020), which uses labeled data to select the most informative label words inside a PLM's vocabulary. It is targeted at the situation when no manually defined class names are available. It's not obvious how to combine it with the manually de-2230 Method AG's News DBPedia Yahoo Amazon IMDB PT 75.1  6.2 (79.0) 66.6  2.3 (68.4) 45.4  7.0 (52.0) 80.2  8.8 (87.8) 86.4  4.0 (92.0) PT+CC 79.9  0.7 (81.0) 73.9  4.9 (82.6) 58.0  1.4 (58.8) 91.4  1.6 (93.5) 91.6  3.0 (93.7) KPT 84.8  1.2 ( 86.7 ) 82.2  5.4 ( 87.4 ) 61.6  2.2 ( 63.8 ) 92.8  1.2 ( 94.6 ) 91.6  2.7 (94.0) -FR 82.7  1.5 (85.0) 81.8  4.6 (86.2) 60.9  1.5 (62.7) 92.8  1.2 ( 94.6 ) 91.6  2.8 ( 94.1 ) -RR 81.4  1.5 (83.7) 81.4  4.5 (85.8) 60.1  1.0 (61.4) 92.8  1.2 ( 94.6 ) 91.6  2.8 ( 94.1 ) -CC 55.5  2.8 (58.3) 64.5  6.8 (73.0) 42.4  5.0 (46.8) 86.2  5.7 (92.5) 90.3  2.8 ( 94.1 ) Table 2: Results of zero-shot text classification. fined class name to boost the performance, and how it can be applied in a zero-shot setting. Therefore we only compare it in the few-shot setting with no class name information given. 2021). They use a continuous vector for each class and use the dot product between the masked language model output and the class vector to produce the probability for each class. In our experiments, its class vectors are initialized with the class names' word embedding, since it is more effective with 2231 manual class names as the initial values (see Appendix A). As an optimization-based method, Soft Verbalizer is not applicable in the zero-shot setting. PT+CC. For zero-shot setting, we further introduce PT combined with our proposed contextualized calibration 5 as a baseline to see how much improvement is made by contextualized calibration instead of knowledgeable verbalizers. For KPT , we experiment with different variants to better understand the proposed methods such as refinement. -FR , -RR , -CC and -LR is the variant that does not conduct Frequency Refinement, Relevance Refinement, Contextualized Calibration, and Learnable Refinement, respectively. In few-shot experiments, we presume that the supervised training data can train the output probability of each label word to the desired magnitude, thus we don't use CC and FR in the KPT . This decision is justified in Appendix D.2. In this subsection, we introduce the specific results and provide possible insights of KPT . Zero-shot. From Table 2, we see that all the variants of KPT , except for KPT-CC, consistently outperforms PT and PT+CC baselines, which indicates the effectiveness of our methods. Comparison between PT and PT+CC proves that Contextualized Calibration is very effective in the zero-shot setting. The results of KPT-FR-RR-CC, which is the variant without any refinement, reveal the label noise is severe in the automatically constructed knowledgeable label words. The gap between KPT-FR-RR and KPT-FR-RR-CC is larger than the gap between PT+CC and PT, demonstrating the drastic difference in the prior probabilities of the knowledgeable label words as we hypothesized in  3.3. Comparison between KPT, KPT-FR, KPT-FR-RR proves the effectiveness of the refinement methods. For the analysis regarding each type of classification task, we observe that the performance boost compared to the baselines in topic classification is higher than sentiment classification, which we conjecture that topic classification requires more external knowledge than sentiment classification. While CC offers huge improvement (on average +13%) over PT baseline, the incorporation of external knowledge further improves over PT+CC up to 11% on DBPedia, and 6% on AG's News and Yahoo. We also observe that the improvement brought 5 The same support sets are used as KPT . by the refinement methods is more noticeable for topic classification tasks. By looking at the fraction of label words maintained after the refinement process (See appendix D.4), we conjecture that the sentiment dictionary that we used in sentiment classification tasks contains little noise. Moreover, the improvement brought by the refinement process justifies the resilience of our methods to recover from noisy label words. Few-shot. From Table 3, we first find out that prompt-based methods win over fine-tuning by a dramatic margin under nearly all situations. The gap enlarges as the shot becomes fewer. Comparing the baseline methods, the Soft Verbalizer (SOFT) generally wins over the Manual Verbalizer(PT) by a slight margin. However, automatic verbalizer (AUTO), although free of manual effort, lags behind the other verbalizers especially in a low-shot setting. The reason is obvious since the selection of label words among the vocabulary becomes inaccurate when labeled data is limited. When comparing KPT with the baseline methods, we find KPT or its variants consistently outperform all baseline methods. On average, 17.8% , 10.3%, and 7.4% error rate reduction from the best baseline methods are achieved on 1, 5, and 10 shot experiments, respectively. Comparing within the variants of KPT , we find that RR and LR are generally effective across shots on topic classification dataset, while in sentiment classification dataset, KPT works well without the refinements, which is consistent with our previous assumptions that the sentiment dictionary has little noise. Note that the KPT-RR variant does not utilize any unlabeled support set  C since we do not conduct CC and FR by default in few-shot learning. This variant is still superior to the baseline methods in most cases. In terms of variance, we can see that KPT enjoys smaller variances than baseline methods in most cases, demonstrating that the better coverage of label words stabilizes the training. For 20-shot experiments, we can see that the gap between different methods narrows as the training data becomes sufficient. However, KPT and its variants still win by a consistent margin over the baseline methods. Surprisingly, with more training data, LR does not become more powerful as we may hypothesize. We conjecture that it is because all label words, even with some noise, can serve as training objectives for prompt tuning. This perspective is similar to Gao et al. (2021) that using bad 2232 as a label word for the class positive can still preform classification although the performance degrades. Ablation studies about our refinement methods have been shown in the previous section. In this section and Appendix D, we conduct more in-depth analyses on the proposed methods. One advantage of KPT is that it can generate diverse label words across different granularities. To specifically quantify such diversity, we conduct a case study. For the correctly predicted sentences of a class y , we count the frequency of label words v  V y appearing in the top-5 predictions for the [MASK] position. Then we report the top-15 frequent label words in Figure 2. Due to space limit, only the results of SPORTS and BUSINESS category of AG's News are shown. As shown in Figure 2, a diversity of label words, instead of mainly the original class names, are predicted. And the predicted label words cover various aspects of the corresponding topic. For example, for the topic SPORTS , the predicted leagues, football, and coach are related to it from different angles. In addition to the visualization, we study the influence of the support set's size on zero-shot text classification in Appendix D.1. Then we justify that few-shot learning via labeled data eases the need for calibration and frequency-based refinement in Appendix D.2. We also demonstrate that our approach to handling the out-of-vocabulary (OOV) words is reasonable in Appendix D.3. Moreover, we take a closer look at the refinement process by analyzing the fraction of label words maintained during refinement in Appendix D.4. Finally, we discuss the potential use of the proposed methods when knowledge bases resources are not readily available in Appendix D.5. In this paper, we propose KPT , which expands the verbalizer in prompt-tuning using the external KB. To better utilize the KB, we propose refinement methods for the knowledgeable verbalizer. The experiments show the potential of KPT in both zero-shot settings and few-shot settings. For future work, there are open questions related to our research for investigation: (1) Better approaches for combining KB and prompt-tuning in terms of template construction and verbalizer design. (2) Incorporating external knowledge into prompt-tuning for other tasks such as text generation. Zhiyuan Liu, Huadong Wang and Shengding Hu proposed the idea and led the research. Shengding Hu designed the methods and conducted the experiments. Ning Ding and Shengding Hu wrote the abstract, introduction (Section 1) and method (Sec-tion 3) part of the paper. Shengding Hu finished the other parts. Ning Ding, Huadong Wang and Zhiyuan Liu thoroughly revised the whole paper. Jingang Wang, Juanzi Li, Wei Wu and Maosong Sun proofread the paper and provided valuable comments. This work is supported by the National Key R&D Program of China (No. 2020AAA0106502), Institute Guo Qiang at Tsinghua University, NExT++ project from the National Research Foundation, Prime Minister's Office, Singapore under its IRC@Singapore Funding Initiative, and International Innovation Center of Tsinghua University, Shanghai, China. This work proposes knowledgeable prompt tuning which uses external knowledge bases to construct the verbalizer. Users should be aware of the potential error in the external KBs, or even the injection of malicious words.
abstain abstain abstain abstain result method abstain other other abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain method abstain objective method method abstain abstain other other other other other other other other other other other objective other other method abstain abstain abstain abstain abstain method method method abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain objective abstain method abstain other abstain abstain method abstain abstain method method abstain abstain abstain method abstain abstain other abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain other abstain abstain method abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain result abstain result abstain result abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain method abstain objective objective abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain