Current methods for few-shot fine-tuning of pretrained masked language models (PLMs) require carefully engineered prompts and verbalizers for each new task to convert examples into a cloze-format that the PLM can score. In this work, we propose PERFECT , a simple and efficient method for few-shot fine-tuning of PLMs without relying on any such handcrafting , which is highly effective given as few as 32 data points. PERFECT makes two key design choices: First, we show that manually engineered task prompts can be replaced with task-specific adapters that enable sample-efficient fine-tuning and reduce memory and storage costs by roughly factors of 5 and 100, respectively. Second, instead of using handcrafted verbalizers, we learn new multi-token label embeddings during fine-tuning, which are not tied to the model vocabulary and which allow us to avoid complex auto-regressive decoding. These embeddings are not only learnable from limited data but also enable nearly 100x faster training and inference. Experiments on a wide range of few shot NLP tasks demonstrate that PERFECT , while being simple and efficient, also outperforms existing state-of-the-art few-shot learning methods. Our code is publicly available at https://github.com/ facebookresearch/perfect.git . Recent methods for few-shot language model tuning obtain impressive performance but require careful engineering of prompts and verbalizers to convert inputs to a cloze-format (Taylor, 1953) that can be scored with pre-trained language models (PLMs) (Radford et al., 2018; Radford et al.; Brown et al., 2020; Schick and Schtze, 2021a,b). For example, as Figure 1 shows, a sentiment classifier can be designed by inserting the input text x in a prompt template  x It was [MASK] where verbalizers (e.g., great' and terrible') are substituted for the [MASK] to score target task labels (positive' or negative'). In this paper, we show that such engineering is [CLS] The restaurant had excellent foods. It was [MASK] [SEP] Pretrained Language Model Input Pattern MLM Head terrible great Verbalizers positive negative Labels Figure 1: Existing few-shot fine-tuning methods require manual engineering to reduce new tasks to masked language modeling. not needed for few-shot learning and instead can be replaced with simple methods for data-efficient fine-tuning with as few as 32 end-task examples. More specifically, we propose PERFECT , a Prompt-free and Efficient paRadigm for FEw-shot Cloze-based fine-Tuning. To remove handcrafted patterns, PERFECT uses task-specific adapter layers (Houlsby et al., 2019; Pfeiffer et al., 2020) (3.1). Freezing the underlying PLM with millions or billions of parameters (Liu et al., 2019; Raffel et al., 2020), and only tuning adapters with very few new parameters saves on memory and storage costs (4.2), while allowing very sample-efficient tuning (4). It also stabilizes the training by increasing the worst-case performance and decreasing variance across the choice of examples in the few shot training sets (4.3). To remove handcrafted verbalizers (with variable token lengths), we introduce a new multi-token fixed-length classifier scheme that learns task label embeddings which are independent from the language model vocabulary during fine-tuning (3.2). We show (4) that this approach is sample efficient and outperforms carefully engineered verbalizers from random initialization (4). It also allows us to avoid previously used expensive auto-regressive decoding schemes (Schick and Schtze, 2021b), by leveraging prototypical networks (Snell et al., 2017) over multiple tokens. Overall, these changes enable up to 100x faster learning and inference (4.2). PERFECT has several advantages: It avoids engineering patterns and verbalizers for each new task, which can be cumbersome. Recent work has shown that even some intentionally irrelevant or misleading prompts can perform as well as more interpretable ones (Webson and Pavlick, 2021). Unlike the zero-shot or extreme few-shot case, where prompting might be essential, we argue in this paper that all you need is tens of training examples to avoid these challenges by adopting PERFECT or a similar data-efficient learning method. Experiments on a wide variety of NLP tasks demonstrate that PERFECT outperforms state-of-the-art prompt-based methods while being significantly more efficient in inference and training time, storage, and memory usage (4.2). To the best of our knowledge, we are the first to propose a few-shot learning method using the MLM objective in PLMs that provide state-of-the-art results while removing all per-task manual engineering. Problem formulation: We consider a general problem of fine-tuning language models in a few-shot setting, on a small training set with K unique classes and N examples per class, such that the total number of examples is |D| = N  K . Let D =  Kk =1 D k be the given training set, where D k = { ( x ik ,y ik ) } Ni =1 shows the set of examples labeled with class k and y ik  Y is the corresponding label, where |Y| = K . We additionally assume access to a development set with the same size as the training data. Note that larger validation sets can grant a substantial advantage (Perez et al., 2021), and thus it is important to use a limited validation size to be in line with the goal of few-shot learning. Unless specified otherwise, in this work, we use 16 training examples ( N = 16 ) and a validation set with 16 examples, for a total of 32-shot learning. Recent work has shown that fine-tuning all parameters of PLMs with a large number of parameters in low-resource datasets can lead to a sub-optimal solution (Peters et al., 2019; Dodge et al., 2020). As shown in Figure 2, Rebuffi et al. (2018) and Houlsby et al. (2019) suggest an efficient alternative, by inserting small task-specific modules called adapters within layers of a PLMs. They then only train the newly added adapters and layer normalization, while fixing the remaining parameters of a PLM. b) a feed-forward block, where both modules are followed by a skip connection. As depicted in Figure 2, adapters are normally inserted after each of these blocks before the skip connection. Adapters are bottleneck architectures. By keeping input and output dimensions the same, they introduce no additional architectural changes. Each adapter, A ( . )  RH , consists of a down-projection, D ( . )  RH  B , a non-linearity, such as GeLU (Hendrycks and Gimpel, 2016), and an up-projection U ( . )  RB  H , where H is the dimension of input hidden states x , and B is the bottleneck size. Formally defined as: A ( x )= U ( GeLU ( D ( x )))+ x , (1) 2.2 Prompt-based Fine-tuning Standard Fine-tuning: In standard fine-tuning with PLMs (Devlin et al., 2019), first a special [CLS] token is appended to the input x , and then the PLM maps it to a sequence of hidden representations h = ( h 1 ,..., h S ) with h i  RH , where H is the hidden dimension, and S is the maximum sequence length. Then, a classifier, softmax ( WT h [CLS] ) , using the embedding of the classification token ( h [CLS] ), is trained end-to-end for each downstream task. The main drawback of this approach is the discrepancy between the pre-training and fine-tuning phases since PLMs have been trained to predict mask tokens in a masked language modeling task (Devlin et al., 2019). 2021a,b; Gao et al., 2021) formulates tasks in a cloze-format (Taylor, 1953). This way, the model can predict targets with a masked language modeling (MLM) objective . For example, as shown in Figure 1, for a sentiment classification task, inputs are converted to: x prompt = [CLS] x . Then, the PLM determines which verbalizer (e.g., great' and terrible') is the most likely substitute for the mask in the x prompt . This subsequently determines the score of targets (positive' or negative'). In detail: Training strategy: Let M : Y V be a mapping from target labels to individual words in a PLM's vocabulary. We refer to this mapping as verbalizers . Then the input is converted to x prompt = T ( x ) by appending a pattern and a mask token to x so that it has the format of a masked language modeling input. Then, the classification task is converted to a MLM objective (Tam et al., 2021; Schick and Schtze, 2021a), and the PLM computes the probability of the label y as: p ( y | x )= p ( [MASK] = M ( y ) | x prompt ) = exp( WT M ( y ) h [MASK] ) (cid:80) v  V exp( W Tv  h [MASK] ) , (2) where h [MASK] is the last hidden representation of the mask, and W v shows the output embedding of the PLM for each verbalizer v V . For many tasks, verbalizers have multiple tokens. Schick and Schtze (2021b) extended (2) to multiple mask tokens by adding the maximum number of mask tokens M needed to express the outputs (verbalizers) for a task. In that case, Schick and Schtze (2021b) computes the probability of each class as the summation of the log probabilities of each token in the corresponding verbalizer, and then they add a hinge loss to ensure a margin between the correct verbalizer and the incorrect ones. Inference strategy: During inference, the model needs to select which verbalizer to use in the given context. Schick and Schtze (2021b) predicts the verbalizer tokens in an autoregressive fashion. They first trim the number of mask tokens from M to each candidate verbalizer's token length and compute the probability of each mask token. They then choose the predicted token with the highest probability and replace the corresponding mask token. Conditioning MASK 1 CLS Multi-head Attention Adapter + PLML aye r Layer norm Feed forward + Layer norm Embedding Layer SEP TOK 1 Adapter MASK Embedding 1 Hinge Loss D es i r e d L a b e l s E s t i m a t e d L a b e l s TOKNMASKMMASK Embedding MWM Label Embedding W 1 Input Masks Figure 3: We remove handcrafted patterns and verbalizers. on this new token, the probabilities of the remaining mask positions are recomputed. They repeat this autoregressive decoding until they fill all mask positions. This inference strategy is very slow, as the number of forward passes increases with the number of classes and the number of verbalizer's tokens. This formulation obtained impressive few-shot performance with PLMs. However, the success of this approach heavily relies on engineering handcrafted patterns and verbalizers . Coming up with suitable verbalizers and patterns can be difficult (Mishra et al., 2022b,a). Additionally, the performance is sensitive to the wording of patterns (Zhao et al., 2021; Perez et al., 2021; Schick and Schtze, 2021a; Jiang et al., 2020) or to the chosen verbalizers (Webson and Pavlick, 2021). In addition, handcrafted verbalizers cause problems for efficient training: a) they require updating the PLM embedding layer, causing large memory overhead; b) fine-tuning PLMs also requires a very small learning rate (usually 10  5 ), which slows down tuning the parameters of the verbalizers; c) modeling verbalizers as one of the tokens of 3640 the PLM vocabulary (perhaps unintentionally) impacts the input representation during tuning; d) verbalizers have variable token lengths, complicating the implementation in a vectorized format, thereby making it challenging to efficiently fine-tune PLMs. We propose PERFECT , a verbalizer and pattern free few-shot learning method. We design PERFECT to be close to the pre-training phase, similar to the PET family of models (Schick and Schtze, 2021b; Gao et al., 2021), while replacing handcrafted patterns and verbalizers with new components that are designed to describe the task and learn the labels. As shown in Figure 3, we first convert each input x input to its masked language modeling (MLM) input containing M mask tokens [MASK] 1 with no added patterns, denoted as x masked = T  ( x input ) . 2 PERFECT then trains a classifier per-token and optimizes the average multi-class hinge loss over each mask position. Three main components play a role in the success of PERFECT : a) a pattern-free task description, where we use task-specific adapters to efficiently tell the model about the given task, replacing previously manually engineered patterns (3.1), b) multi-token label-embedding as an efficient mechanism to learn the label representations, removing manually designed verbalizers (3.2). c) an efficient inference strategy building on top of the idea of prototypical networks (Snell et al., 2017) (3.4), which replaces prior iterative autoregressive decoding methods (Schick and Schtze, 2021b). As shown in Figure 3, we fix the underlying PLM model and only optimize the new parameters that we add (green boxes). This includes the task-specific adapters to adapt the representations for a given task and the multi-token label representations. We detail each of these components below. We use task-specific adapter layers to provide the model with learned, implicit task descriptions. Adapters additionally bring multiple other benefits: a) fine-tuning all weights of PLMs with millions or billions of parameters is sample-inefficient, and can be unstable in low-resource settings (Dodge et al., 1 We discuss the general case with inserting multiple masks; for some datasets this improves performance (4.3.1). 2 We insert mask tokens after the input string in single-sentence benchmarks, and after the first sentence in the case of sentence-pair datasets and encode both sentences as a single input, which we found to perform the best (Appendix C). 2020); adapters allow sample-efficient fine-tuning, by keeping the underlying PLM fixed, b) adapters reduce the storage and memory footprints (4.2), c) they also increase stability and performance (4), making them an excellent choice for few-shot fine-tuning. To our knowledge, this is the first approach for using task-specific adapters to effectively and efficiently remove patterns in few-shot learning. Experimental results in 4 show its effectiveness compared to handcrafted patterns and soft prompts (Li and Liang, 2021; Lester et al., 2021). We freeze the weights of the PLM's embedding layer and introduce a separate label embedding L  RK  M  H , which is a multi-token label representation where M is the number of tokens representing each label, K indicates the number of classes, H is the input hidden dimension. Using a fixed number of tokens M for each label, versus variable-token length verbalizers used in prior work (Schick and Schtze, 2021a,b) substantially simplifies the implementation and accelerates the training (4.2). As shown in Figure 3, we optimize label embeddings so that the PLM predicts the correct label, and optimize adapters to adapt the PLM for the given task. For label embeddings, PERFECT trains a classifier per token and optimizes the average multi-class hinge loss over all mask positions. Given x masked , let h [MASK] i be the embedding of its i -th mask token from the last layer of the PLM encoder. Additionally, let f ( . ) : RH  RK be a per-token classifier that computes the predictions by multiplying the mask token embedding with its corresponding label embedding. Formally defined as: t i = f ( h [MASK] i )= LT i h [MASK] i , where L i  RK  H shows the label embedding for the i -th mask position. Then, for each mask position, we optimize a multi-class hinge loss between their scores t i and labels. Formally defined as: L ( x ,y,i )= (cid:80) Kk =1 ,k  = y max(0 ,m  t iy + t ik ) K , where t ik shows the k -th element of t i , representing the score corresponding to class k , and m is the margin, which we fix to the default value of m = 1 . Then, the final loss is computed by averaging the loss 3641 over all mask tokens and training samples: L = 1 M |D| (cid:88) ( x ,y ) D M (cid:88) i =1 L ( x ,y,i ) (3) 3.4 Inference with PERFECT During evaluation, instead of relying on the prior iterative autoregressive decoding schemes (Schick and Schtze, 2021b), we classify a query point by finding the nearest class prototype to the mask token embeddings: y =argmax y Y max i { 1 ,...,M } (cid:16) exp  d ( h qi , c iy ) (cid:17) , (4) where d is squared euclidean distance, 3 h qi indicates the embedding of the i -th mask position for the query sample q , and c iy  RD is the prototype representation of the i -th mask token with class label y , i.e., the mean embedding of i -th mask position in all training samples with label y : c iy = 1 |D y | (cid:88) b D y h bi , (5) where h bi shows the embedding of i -th mask position for training sample b , and D y is the training instances with class y . This strategy closely follows prototypical networks (Snell et al., 2017), but applied across multiple tokens. We choose this form of inference because prototypical networks are known to be sample efficient and robust (Snell et al., 2017), and because it substantially speeds up evaluation compared to prior methods (4.2). We conduct extensive experiments on a variety of NLP datasets to evaluate the performance of PERFECT and compare it with state-of-the-art few-shot learning. Datasets: We consider 7 tasks and 12 datasets: 1) the sentiment analysis datasets SST-2 (Socher et al., 2013), SST-5 (Socher et al., 2013), MR (Pang and Lee, 2005), and CR (Hu and Liu, 2004), 2) the subjectivity classification dataset SUBJ (Pang and Lee, 2004), 3) the question classification dataset TREC (Voorhees and Tice, 2000), 4) the natural language inference datasets CB (De Marneffe et al., 2019) and RTE (Wang et al., 2019a), 5) the question answering dataset QNLI (Rajpurkar et al., 2016), 6) the word sense disambiguation dataset WiC (Pilehvar 3 We also tried with cosine similarity but found a slight improvement with squared Euclidean distance (Snell et al., 2017). and Camacho-Collados, 2019), 7) the paraphrase detection datasets MRPC (Dolan and Brockett, 2005) and QQP. 4 See datasets statistics in Appendix A. For MR, CR, SST-5, SUBJ, and TREC, we test on the original test sets, while for other datasets, since test sets are not publicly available, we test on the original validation set. We sample 16 instances per label from the training set to form training and validation sets. Baselines We compare with the state-of-the-art few-shot learning of PET and fine-tuning: PET (Schick and Schtze, 2021a,b) is the state-of-the-art few-shot learning method that employs carefully crafted verbalizers and patterns. We report the best (PET-best) and average (PET-average) results among all patterns and verbalizers. 5 FINETUNE The standard fine-tuning (Devlin et al., 2019), with adding a classifier on top of the [CLS] token and fine-tuning all parameters. Our method We study the performance of PERFECT and perform an extensive ablation study to show the effectiveness of our design choices: PERFECT -rand We randomly initialize the label embedding L from a normal distribution N (0 , ) with  = 10  4 (chosen based on validation performance, see Appendix D) without relying on any handcrafted patterns and verbalizers . As an ablation, we study the following two variants: PERFECT -init We initialize the label embedding with the token embeddings of manually designed verbalizers in the PLM's vocabulary to study the impact of engineered verbalizers. prompt+mte To compare the impact of adapters versus soft prompt-tuning for few-shot learning, we append trainable continuous prompt embeddings to the input (Lester et al., 2021). Then we only tune the soft prompt and multi-token label embeddings (mte). bitfit+mte Following Cai et al. (2020) and Rav-fogel et al. (2021), we tune biases as an alternative to adapters. We additionally tune multi-token label embeddings. Logan IV et al. (2021) Following Logan IV et al. (2021), we remove patterns and tune the biases in the PET. Experimental details: We use the RoBERTa large model (Liu et al., 2019) (355M parameters) as the underlying PLM for all methods. We use the HuggingFace PyTorch implementation (Wolf et al., 2020). For 4 https://quoradata.quora.com/ 5 For a controlled study, we use the MLM variant shown in (2), which has been shown to perform the best (Tam et al., 2021). the baselines, we used the carefully manually designed patterns and verbalizers in Gao et al. (2021), Min et al. (2021), and Schick and Schtze (2021b) (usually 5 different options per datasets; see Appendix B). We evaluate all methods using 5 different random samples to create the training/validation sets and 4 different random seeds for training. Therefore, for PET-average, we report the results on 20 x 5 (number of patterns and verbalizers) = 100 runs, while for PET-best and our method, we report the results over 20 runs. The variance in few-shot learning methods is usually high (Perez et al., 2021; Zhao et al., 2021; Lu et al., 2021). Therefore, we report average, worst-case performance, and standard deviation across all runs, where the last two values can be important for risk-sensitive applications (Asri et al., 2016). Table 1 shows the performance of all methods. PERFECT obtains state-of-the-art results, improving the performance compared to PET-average by +1.1 and +4.6 points for single-sentence and sentence-pair datasets respectively. It even outperforms PET-best, where we report the best performance of PET across multiple manually engineered patterns and verbalizers. Moreover, PERFECT generally improves the minimum performance and reduces standard deviation substantially. Finally, PERFECT is also significantly more efficient: reducing the training and inference time, memory usage, and storage costs (see 4.2). PET-best improves the results over PET-average showing that PET is unstable to the choice of patterns and verbalizers; this difference is more severe for sentence-pair benchmarks. This might be because the position of the mask highly impacts the results, and the patterns used for sentence-pair datasets in Schick and Schtze (2021b) exploits this variation by putting the mask in multiple locations (see Appendix B). Removing patterns and tuning biases in Logan IV et al. (2021) is not expressive enough and performs substantially worse than PERFECT on average. embedding with handcrafted verbalizers in PERFECT -init, it consistently obtains lower performance, demonstrating that PERFECT is able to obtain state-of-the-art performance with learning from pure random initialization . We argue that initializing randomly close to zero (with low variance  =10  4 ), as done in our case, slightly improves performance, which perhaps is not satisfied when initializing from the manually engineered verbalizers (see Appendix D). As a second ablation, when learning patterns with optimizing soft prompts in prompt+mte, we observe high sensitivity to learning rate, as also confirmed in Li and Liang (2021) and Mahabadi et al. (2021a). We experimented with multiple learning rates but performance consistently lags behind PERFECT -rand. This can be explained by the low flexibility of such methods as all the information regarding specifying patterns needs to be contained in the prefixes. As a result, the method only allows limited interaction with the rest of the model parameters, and obtaining good performance requires very large models (Lester et al., 2021). In addition, increasing the sequence length leads to memory overhead (Mahabadi et al., 2021a), and the number of prompt tokens is capped by the number of tokens that can fit in the maximum input length, which can be a limitation for tasks requiring large contexts. As a third ablation, tuning biases with optimizing soft prompts in bitfit+mte obtains lower performance compared to PERFECT , showing that adapters are a better alternative compared to tuning biases to learn task descriptions for few-shot learning. In this section, we compare the efficiency of PERFECT with the state-of-the-art few-shot learning method, PET. To this end, we train all methods for ten epochs on the 500-sampled QNLI dataset. We select the largest batch size for each method that fits a fixed budget of the GPU memory (40 GB). Due to the auto-regressive inference strategy of PET (Schick and Schtze, 2021b), all prior work implemented it with a batch size of 1 (Perez et al., 2021; Schick and Schtze, 2021b; Tam et al., 2021). Additionally, since PET deals with verbalizers of variable lengths, it is hard to implement their training phase in batch mode. We specifically choose QNLI to have verbalizers of the same length and enable batching for comparison purposes (referred to as PET in batch ). However, verbalizers are still not of fixed-length for most other tasks, and this speed-up does not apply generally to PET. In Table 2, for each method we report the percentage of trained parameters, memory usage, training time, and inference time. PERFECT reduces the number of trained parameters, and therefore the storage requirement, by 99.08%. It additionally reduces the memory requirement by 21.93% compared to PET. PERFECT speeds up training substantially, by 97.22% relative to the original PET's implementation, and 30.85% to our implementation of PET. This is because adapter-based tuning saves on memory and allows training with larger batch sizes. In addition, PERFECT is significantly faster during inference time (96.76% less inference time relative to PET). Note that although prompt+mte and bitfit+mte can also reduce the storage costs, by having 0.02M and 0.32 M trainable parameters respectively, they are not expressive enough to learn task descriptions, and their performance substantially lags behind PERFECT (see Table 1). Overall, given the size of PLMs with millions and billions of parameters (Liu et al., 2019; Raffel et al., 2020), efficient few-shot learning methods are of paramount importance for practical applications. PERFECT not only outperforms the state-of-the-art in terms of accuracy and generally improves the stability (Table 1), but also is significantly more efficient in runtime, storage, and memory. Can task-specific adapters replace manually engineered patterns? PERFECT is a pattern-free approach and employs adapters to provide the PLMs with task descriptions implicitly. In this section, we study the contribution of replacing manual patterns with adapters in isolation without considering our other contributions in representing labels, training, and inference. In PET (Schick and Schtze, 2021a,b), 3644 Dataset PET-Average Pattern-Free SST-2 89.7/81.0/2.4 90.5/87.8/1.2 CR 88.4/68.8/3.0 89.8/87.0/1.4 MR 85.9/79.0/2.1 86.4/83.0/1.8 SST-5 45.9/40.3/2.4 44.8/40.0/2.4 SUBJ 88.1/79.6/2.4 85.3/74.7/3.8 TREC 85.0/70.6/4.5 87.9/84.6/1.8 CB 86.9/73.2/5.1 93.0/89.3/1.9 RTE 60.1/49.5/4.7 63.7/56.3/4.1 QNLI 66.5/55.7/6.2 71.3/65.8/2.5 MRPC 62.1/38.2/6.8 66.0/54.4/5.6 QQP 63.4/44.7/7.9 71.8/64.3/3.7 WiC 51.0/46.1/2.6 53.7/50.3/2.0 Avg 72.8/60.6/4.2 75.4/69.8/2.7 Table 3: Average performance of PET with five different patterns vs. Pattern-Free that replaces handcrafted patterns with task-specific adapters. we replace the handcrafted patterns with task-specific adapters ( Pattern-Free ) while keeping the verbalizers and the training and inference intact 6 and train it with a similar setup as in 4. Table 3 shows the results. While PET is very sensitive to the choice of prompts, adapters provide an efficient alternative to learn patterns robustly by improving the performance (average and worst-case) and reducing the standard deviation. This finding demonstrates that task-specific adapters can effectively replace manually engineered prompts. Additionally, they also save on the training budget by at least 1 / number of patterns (normally 1/5) by not requiring running the method for different choices of patterns, and by freezing most parameters, this saves on memory and offers additional speed-up. Impact of Removing Adapters To study the impact of adapters in learning patterns, we remove adapters, while keeping the label embedding. Handcrafted patterns are not included and we tune all parameters of the model. Table 4 shows the results. Adding adapters for learning patterns contributes to the performance by improving the average performance, and making the model robust by improving the minimum performance and reducing the standard deviation. This is because training PLMs with millions of parameters is sample-inefficient and unstable on resource-limited datasets (Dodge 6 Since we don't have patterns, in the case of multiple sets of verbalizers, we use the first set of verbalizers as a random choice. Dataset PERFECT -Adapters SST-2 90.7/88.2/1.2 88.2/81.9/2.3 CR 90.0/85.5/1.4 89.2/83.1/1.7 MR 86.3/81.4/1.6 82.5/78.2/2.5 SST-5 42.7/35.1/2.9 40.6/33.6/3.3 SUBJ 89.1/82.8/2.1 89.7/85.0/1.9 TREC 90.6/81.6/3.2 89.8/74.2/4.3 CB 90.3/83.9 /3.5 89.6 /83.9/2.8 RTE 60.4/53.1/ 4.7 61.7/53.8 /5.1 QNLI 74.1/60.3/4.6 73.2/56.3/5.8 MRPC 67.8 /54.7/5.7 68.0 /54.2/6.1 QQP 71.2/64.2/3.5 71.0/62.0/3.7 WiC 53.8/47.0/3.0 52.5/46.9/3.0 Avg 75.6/68.1/3.1 74.7/66.1/3.5 Table 4: Performance of PERFECT w/o adapters, -Adapters . We report the average performance/worst-case perfor-mance/and the standard deviation. et al., 2020; Zhang et al., 2020; Mosbach et al., 2021). However, by using adapters, we substantially reduce the number of trainable parameters, allowing the model to be better tuned in a few-shot setting. Impact of the number of masks In Table 1, to compare our design with PET in isolation, we fixed the number of mask tokens as the maximum number inserted by PET. In table 5, we study the impact of varying the number of inserted mask tokens for a random selection of six tasks. For most tasks, having two mask tokens performs the best, while for MR and RTE, having one, and for MRPC, inserting ten masks improves the results substantially. The number of required masks might be correlated with the difficulty of the task. PERFECT is designed to be general, enabling having multiple mask tokens. Adapter Layers: Mahabadi et al. (2021b) and stn et al. (2020) proposed to generate adapters' weights using hypernetworks (Ha et al., 2017), where Mahabadi et al. (2021b) proposed to share a small hypernetwork to generate conditional adapter weights efficiently for each transformer layer and task. Mahabadi et al. (2021a) proposed compacter layers by building on top of ideas of parameterized hypercomplex layers (Zhang et al., 2021) and low-rank methods (Li et al., 2018; Aghajanyan et al., 2021), as an efficient fine-tuning method for PLMs. We are the first to employ adapters to replace handcrafted patterns for few-shot learning. Few-shot Learning with PLMs: Le Scao and Rush (2021) showed that prompting provides substantial improvements compared to fine-tuning, especially in low-resource settings. Subsequently, researchers continuously tried to address the challenges of manually engineered patterns and verbalizers: a) Learning the patterns in a continuous space (Li and Liang, 2021; Qin and Eisner, 2021; Lester et al., 2021), while freezing PLM for efficiency, has the problem that, in most cases, such an approach only works with very large scale PLMs (Lester et al., 2021), and lags behind full fine-tuning in a general setting, while being inefficient and not as effective compared to adapters (Ma-habadi et al., 2021a). b) Optimizing patterns in a discrete space (Shin et al., 2020; Jiang et al., 2020; Gao et al., 2021) has the problem that such methods are computationally costly. c) Automatically finding verbalizers in a discrete way (Schick et al., 2020; Schick and Schtze, 2021a) is computationally expensive and does not perform as well as manually designed ones. d) Removing manually designed patterns (Logan IV et al., 2021) substantially lags behind the expert-designed ones. Our proposed method, PERFECT , does not rely on any handcrafted patterns and verbalizers. We proposed PERFECT , a simple and efficient method for few-shot learning with pre-trained language models without relying on handcrafted patterns and verbalizers. PERFECT employs task-specific adapters to learn task descriptions implicitly, replacing previous handcrafted patterns, and a continuous multi-token label embedding to represent the output classes. Through extensive experiments over 12 NLP benchmarks, we demonstrate that PERFECT , despite being far simpler and more efficient than recent few-shot learning methods, produces state-of-the-art results. Overall, the simplicity and effectiveness of PERFECT make it a promising approach for few-shot learning with PLMs. The authors would like to thank Sebastian Ruder and Marius Mosbach for their comments on drafts of this paper. This research was partly supported by the Swiss National Science Foundation under grant number 200021_178862.
abstain objective objective objective abstain abstain other abstain abstain result abstain abstain objective abstain abstain abstain objective result method abstain abstain abstain method abstain objective method other method other method other other other other other other other other other other other abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain method abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain other abstain method abstain result abstain result abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain other result abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain other other objective other other other other other other objective objective abstain objective abstain other other