We introduce a noisy channel approach for language model prompting in few-shot text classification. Instead of computing the likelihood of the label given the input (referred as direct models), channel models compute the conditional probability of the input given the label, and are thereby required to explain every word in the input. We use channel models for recently proposed few-shot learning methods with no or very limited updates to the language model parameters, via either in-context demonstration or prompt tuning. Our experiments show that, for both methods, channel models significantly outperform their direct counterparts, which we attribute to their stability, i.e., lower variance and higher worst-case accuracy. We also present extensive ablations that provide recommendations for when to use channel prompt tuning instead of other competitive methods (e.g., direct head tuning): channel prompt tuning is preferred when the number of training examples is small, labels in the training data are imbalanced, or generalization to unseen labels is required. Prompting large language models, by prepending natural language text or continuous vectors (called prompts ) to the input, has shown to be promising in few-shot learning (Brown et al., 2020). Prior work has proposed methods for finding better prompt (Shin et al., 2020; Li and Liang, 2021; Lester et al., 2021) or better scoring of the output from the model (Zhao et al., 2021; Holtzman et al., 2021). These studies directly predict target tokens to determine the prediction for an end task. Despite promising results, they can be unstable with high variance across different verbalizers (text expression for labels) and seeds, and the worst-case performance is often close to random (Perez et al., 2021; Lu et al., 2021). In this paper, we introduce alternative channel models for prompted few-shot text classification LM =( A three-hour cinema master class. ,  It was great. ) ( x , y ) A three-hour cinema master class. with large language models, inspired by noisy channel models in machine translation (Brown et al., 1993; Koehn et al., 2003; Yu et al., 2017; Yee et al., 2019) and their extensions to other tasks (Yogatama et al., 2017; Lewis and Fan, 2018). Unlike direct models that compute the conditional probability of the label token given the input, channel models compute the conditional probability of the input given the output (Figure 1). Intuitively, channel models are required to explain every word in the input, potentially amplifying training signals in the low data regime. We study the impact of channel models for language model prompting where the parameters of the language model are frozen. In particular, we compare channel models with their direct counterparts for (1) demonstration methods, either concatenation-based (Brown et al., 2020) or our proposed, ensemble-based (Section 4.1.3), and (2) prompt tuning (Lester et al., 2021). Our experiments on eleven text classification datasets show that channel models outperform their direct counterparts by a large margin. We attribute the strong performance of channel models to their stability: they have lower variance and significantly higher worst-case accuracy then their direct counterparts over different verbalizers and seeds. We additionally find a direct model with head tuning tuning the LM head while freezing other parametersis surprisingly effective, often outperforming direct models with other forms of tuning. While different methods are preferred given different conditions, the channel model with prompt tuning (denoted as channel prompt tuning) significantly outperforms all direct baselines when (1) the training data is imbalanced, or (2) generalization to unseen labels is required. In summary, our contributions are three-fold: 1. We introduce a noisy channel approach for language model prompting in few-shot text classification, showing that they significantly outperform their direct counterparts for both demonstration methods and prompt tuning. 2. We find particularly strong performance of channel models over direct models when the training data is imbalanced or generalization to unseen labels is required. 3. Based on extensive ablations, we provide recommendations between different models (di-rect vs. channel and prompt tuning vs. head tuning) based on given conditions such as the target task, the size of training data, the number of classes, the balance between labels in the training data, and whether generalization to unseen labels is required. Let x and y be the input and the output, respectively. The most widely used models, denoted as direct models, compute P ( y | x ) . In contrast, noisy channel models maximize P ( x | y ) P ( y ) (Shannon, 1948; Brown et al., 1993). 1 While the noisy channel approach has been the most successful in machine translation (Yamada and Knight, 2001; Koehn et al., 2003; Yu et al., 2017; Yee et al., 2019), it has also been studied in more general NLP tasks. Prior work provides a theoretical analysis that channel models approach their asymptotic errors more rapidly than their direct counterparts (Ng and Jordan, 2002), and empirically shows that channel models are more robust to distribution shift in text classification (Yogatama et al., 2017) or question answering (Lewis and Fan, 2018), and in a few-shot setup (Ding and Gimpel, 2019). 1 We follow Yu et al. (2017); Yee et al. (2019) in using the terms direct models and channel models. They are often referred as discriminative models and generative models in prior work (Yogatama et al., 2017; Lewis and Fan, 2018). In principle, these two distinctions are not always equivalent, e.g., a model that computes P ( x, y ) = P ( y | x ) P ( x ) is generative but not a channel model. In this paper, we explore channel models using a large language model on a wide range of text classification tasks, focusing on prompt-based few-shot learning. Prior work in few-shot learning has used different approaches, including semi-supervised learning with data augmentation or consistency training (Miyato et al., 2017; Clark et al., 2018; Xie et al., 2020; Chen et al., 2020) and meta learning (Finn et al., 2017; Huang et al., 2018; Bansal et al., 2020). Recent work has introduced prompting (or priming ) of a large language model. For example, Brown et al. (2020) proposes to use a concatenation of training examples as a demonstration, so that when it is prepended to the input and is fed to the model, the model returns the output following the pattern in the training examples. This is especially attractive as it eliminates the need for updating parameters of the language model, which is often expensive and impractical. Subsequent work proposes alternative ways of scoring labels through better model calibration (Zhao et al., 2021; Holtzman et al., 2021), or learning better prompts, either in a discrete space (Shin et al., 2020; Jiang et al., 2020; Gao et al., 2021) or in a continuous space (Li and Liang, 2021; Lester et al., 2021; Liu et al., 2021; Zhong et al., 2021; Qin and Eisner, 2021). Almost all of them are direct models, computing the likelihood of y given x with the prompts. Our work is closely related to two recent papers. Tam et al. (2021) studies a label-conditioning objective for masked language models; although this is not strictly a generative channel model, conditioning on the output y is similar to our work. However, they are still optimizing a discriminative objective, and inference at test time is the same as with the direct model. Holtzman et al. (2021) explores zero-shot models that compute the probability of x given y based on Pointwise Mutual Information, but with a restriction that the input and the output are interchangeable. To the best of our knowledge, our work is the first that uses a noisy channel model for few-shot language model prompting for classification, and also the first to draw the connection with the noisy channel literature. We focus on text classification tasks. The goal is to learn a task function f : X  C , where X is the 5317 Method Zero-shot Concat-based Demonstrations Ensemble-based Demonstrations Direct PLM ( v ( c i ) | x ) PLM ( v ( c i ) | x 1 , v ( c 1 ) ...x k , v ( c k ) , x )  Kj =1 PLM ( v ( c i ) | x j , v ( c j ) , x ) Direct++ PLM ( v ( c i ) | x ) PLM ( v ( c i ) | NULL ) PLM ( v ( c i ) | x 1 ,v ( c 1 ) ...x k ,v ( c k ) ,x ) PLM ( v ( c i ) | x 1 ,v ( c 1 ) ...x k ,v ( c k ) , NULL )  Kj =1 PLM ( v ( c i ) | x j ,v ( c j ) ,x ) PLM ( v ( c i ) | x j ,v ( c j ) , NULL ) Channel PLM ( x | v ( c i )) PLM ( x | x 1 , v ( c 1 ) ...x k , v ( c k ) , v ( c i ))  Kj =1 PLM ( x | v ( c j ) , x j , v ( c i )) Table 1: Comparison of zero-shot, concat-based demonstrations, and ensemble-based demonstrations (Section 4.1). set of all natural language texts and C = { c 1 ...c m } is a set of labels. We consider three formulations. Direct computes distributions of labels c i  C given the input x  X : P ( c i | x ) . This is the most widely used method in modern neural networks. Direct++ is a stronger direct model that computes P ( c i | x ) P ( c i | NULL ) instead of P ( c i | x ) , following the method from Holtzman et al. (2021) and the nonparametric method from Zhao et al. (2021). This approach is motivated by the fact that language models can be poorly calibrated and suffer from competition between different strings with the same meaning. This approach is used for the demonstration methods in Section 4.1. Channel uses Bayes' rule to reparameterize P ( c i | x ) as P ( x | c i ) P ( c i ) P ( x ) . As we are generally interested in argmax c i C P ( x | c i ) P ( c i ) P ( x ) and P ( x ) is independent from c i , it is sufficient to model P ( x | c i ) P ( c i ) . We assume P ( c i ) = 1 |C| and only compute P ( x | c i ) . We explore direct and channel models using a causal language model (LM) PLM that gives the conditional probability of the text y when followed by x . More precisely, given the text x = x 1 ...x t x and y = y 1 ...y t y ( x 1 ...x t x , y 1 ...y t y  V , where V is the vocabulary set), PLM ( y | x ) indicates  t y t (cid:48) =1 PLM ( y t (cid:48) | x 1 ...x t x y 1 ...y t (cid:48)  1 ) . 2 When learning a task function f : X  C , we also assume a pre-defined verbalizer v : C  X which maps each label into a natural language expression. For example, if the task is sentiment analysis with C = { c + , c  } , an example input text x would be A three-hour cinema master class and an example v would have v ( c + ) = It was great and v ( c  ) = It was terrible. In a few-shot setup, we are also given a set of K training examples 2 In practice, we use length normalization that was found to be effective by Holtzman et al. (2021). We are interested in methods where there are no trainable parameters (Section 4.1) or the number of trainable parameters is very small, typically less than 0.01% of the total (Section 4.2). This follows prior observations that updating and saving a large number of parameters for every task is expensive and often infeasible (Rebuffi et al., 2017; Houlsby et al., 2019; Lester et al., 2021). In demonstration methods, there are no trainable parameters. We explore three ways of making a prediction, as summarized in Table 1. 4.1.1 Zero-shot We follow Brown et al. (2020) in computing P ( c i | x ) and P ( x | c i ) as PLM ( v ( c i ) | x ) and PLM ( x | v ( c i )) , respectively. For example, given x = A three-hour cinema master class, the direct model compares the probabilities of It was great and It was terrible when following A three-hour cinema master class, while the channel model considers the probabilities of A three-hour cinema master class when following It was great or It was terrible. We follow the few-shot learning method in Brown et al. (2020). The key idea is to prepend a concatenation of K training examples to the input so that a language model can learn the task setup from the input. The original method was used for a direct model, but can be naturally extended for a channel model. Concretely, P ( c i | x ) in direct models is obtained via PLM ( v ( c i ) | x 1 , v ( c 1 ) ,    , x K , v ( c K ) , x ) , and P ( x | c i ) in channel models is obtained via PLM ( x | v ( c 1 ) , x 1 ,    , v ( c K ) , x K , v ( c i )) . a stronger direct model. Instead of concatenating K training examples as one sequence and getting output probabilities from an LM once, we obtain output probabilities from an LMK times conditioned on one training example at a time, and multiply the resulting probabilities. Specifically, P ( c i | x ) is computed via  Kj =1 PLM ( v ( c i ) | x j , v ( c j ) , x ) and P ( x | c i ) is computed via  Kj =1 PLM ( x | v ( c j ) , x j , v ( c i )) . This method also reduces the memory consumption the concat-based method uses O ( K 2 ) while this method uses O ( K ) and eliminates the dependency on the ordering of training examples, which has been shown to significantly impact the model performance (Zhao et al., 2021; Lu et al., 2021). We also explore methods that tune a very limited number of model parameters, as summarized in Figure 2. We study head tuning (Section 4.2.1) and transformation tuning (Section 4.2.2) for direct models. We also consider prompt tuning (Sec-tion 4.2.3) for both direct and channel models, which we refer as direct prompt tuning and channel prompt tuning, respectively. All models share the same input-output interface with the zero-shot setup in Table 1 during training and inference. Head tuning finetunes the headthe matrix in the LM which transforms the hidden representation from the last transformer layer to the logit values. Let O  R |V| h be the head and h x  R h be the hidden representations from the last transformer layer given x , PLM ( v i | x ) for a token v i  V is computed via an i -th element of Softmax( Oh x ) . We finetune O while freezing all other parameters of the LM. Although O is tied with the embedding matrix of the LM during language model pretraining, we separate them during head tuning. 3 4.2.2 Transformation tuning As an alternative to head tuning, we transform O with a new transformation matrix U  R h  h . Specifically, PLM ( v i | x ) for a token v i  V is computed via an i -th element of Softmax( OUh x ) . We train U , initialized from an identity matrix, and freeze other parameters including O . Prompt tuning is the method that has recently gathered much attention (Li and Liang, 2021; Lester et al., 2021; Liu et al., 2021). The key idea is to consider the LM as a black-box model and instead learn continuous prompt embeddings. We follow the method from Lester et al. (2021) where n prompt tokens u 1 ...u n are prepended to the input, and the embeddings of u 1 ...u n are learned. In other words, direct models compute P ( c i | x ) = PLM ( v ( c i ) | u 1 ...u n , x ) , and channel models compute P ( x | c i ) = PLM ( x | u 1 ...u n , v ( c i )) . The parameters in the LM are frozen except the embeddings of u 1 ...u n . 4 5 Experimental Setup 5.1 Datasets We report results for eleven text classification datasets, following Zhang et al. (2015) and Gao 3 This is different from head tuning from prior work, e.g., Le Scao and Rush (2021), which finetunes  PLM and uses a separate, randomly initialized head instead of the LM head. 4 This is different from prompt tuning in Gao et al. (2021); Liu et al. (2021) which jointly trains prompt embeddings and the parameters of the LM. et al. (2021): SST-2 (Socher et al., 2013), SST-5 (Socher et al., 2013), MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), Amazon (McAuley and Leskovec, 2013), Yelp (Zhang et al., 2015), TREC (Voorhees and Tice, 2000), AGNews (Zhang et al., 2015), Yahoo (Zhang et al., 2015), DBPedia (Lehmann et al., 2015) and Subj (Pang and Lee, 2004). The datasets include a varied number of classes per task, from 2 to 14. See Table 10 in Appendix A for dataset samples. For few-shot learning, we primarily use training set size K = 16 , but explore K = { 4 , 16 , 64 , Full } in the ablations. We sample the K examples uniformly from the true distribution of the training data. We relax the assumption from prior work of an equal number of training examples per label (Gao et al., 2021; Logan IV et al., 2021), for more realistic and challenging evaluation. We follow all the hyperameters and details from prior work (Appendix B) which eliminates the need for a held-out validation set. The very limited data is better used for training rather than validation, and cross-validation is less helpful when the validation set is extremely small (Perez et al., 2021). We use GPT-2 (Radford et al., 2019) for the LM. We primarily use GPT-2 Large but also experiment with varying sizes (Small, Medium, Large and X-Large) for the ablations in Appendix C. While we only experiment with GPT-2, our experiments are easily extendable to other causal language models. We use accuracy as a metric for all datasets. We experiment with 4 different verbalizers (taken from Gao et al. (2021); full list provided in Appendix A), 5 different random seeds for sampling training data, and 4 different random seeds for training. We then report Average accuracy and Worst-case accuracy . 5 We consider the worst-case accuracy to be as important as the average accuracy given significantly high variance of few-shot learning models, as shown in previous work (Zhao et al., 2021; Perez et al., 2021). The worst-case accuracy is likely of more interest in high-risk applications (Asri et al., 2016; Guo et al., 2017). This section reports results from demonstration methods (Section 6.1), tuning methods (Sec-tion 6.2) and ablations (Section 6.3). Discussion is provided in Section 7. Direct vs. Direct++ Direct++ significantly outperforms the naive direct model across all setups, indicating that using P ( c i | x ) P ( c i | NULL ) instead of P ( c i | x ) is highly beneficial as claimed by Holtzman et al. (2021); Zhao et al. (2021). Concat vs. Ensemble Our proposed, ensemble-based method is better than the concat-based method in direct models, by 7% absolute in the average accuracy and the worst-case accuracy, when macro-averaged across all datasets. In contrast, the ensemble-based method is not always better in channel models; it is better only on the datasets with long inputs. We conjecture that the ensemble-based method may suffer when labels in the training data are not balanced, which direct++ explicitly takes into account as described in Zhao et al. (2021). Direct++ vs. Channel In a few-shot setting, channel models outperform direct models in almost all cases. The strongest channel model outperforms the strongest direct model by 3.1% and 7.2% absolute, in terms of the average accuracy and the worst-case accuracy, respectively. 5 We also report standard deviation and best-case accuracy in the Appendix. Standard deviation and the best-case accuracy are reported in Table 11 and Table 12 in the Appendix. They indicate strong performance of channel models can be attributed to their low variance. The highest best-case accuracy is achieved by di-rect++ on most datasets, but it has a higher variance, having lower average and the worst-case accuracy than channel models. Zero-shot vs. Few-shot Performance of direct models sometimes degrades in a few-shot setting, which is also observed by prior work (Zhao et al., 2021). This is likely because demonstrations provided by the training data may cause the model to be miscalibrated and easily biased by the choice of demonstrations. However, channel models achieve few-shot performance that is significantly better than zero-shot methods on all datasets. Comparison when prompt tuning When using prompt tuning, channel models consistently outperform direct models by a large margin on all datasets. Improvements are 13.3% and 23.5% absolute in the average and the worst-case accuracy, respectively. Standard deviation and the best-case accuracy are reported in Table 13 in the Appendix. Consistent with the findings in Section 6.1, the strong performance of channel prompt tuning can be explained by the low variance of channel prompt tuning. Direct prompt tuning often achieves higher best-case accuracy; however, due to its high variance, its overall accuracy is lower, with significantly lower worst-case accuracy. Head tuning vs. prompt tuning We find that head tuning is a very strong method, despite often being omitted as a baseline in prior work. It significantly outperforms direct prompt tuning in all cases. It also outperforms channel prompt tuning on some datasets, particularly significantly on TREC and Subj. For these datasets, the taskfinding the type of the answer to the question or identifying the subjectivity of the statementis inherently different from language modeling, and likely benefits from directly updating the LM parameters, rather than using the LM as a black box. are achieved on Yahoo and DBPedia. In fact, on these datasets, channel prompt tuning even outperforms all finetuning finetuning all parameters of the LMwhich achieves 48.9/43.8 on Yahoo and 66.3/50.4 on DBPedia. We conjecture that using K = 16 on these datasets naturally requires generalization to unseen labels due to the large number of classes ( |C| = 10 and 14 ), where channel prompt tuning significantly outperforms direct models, as we show in Section 6.4. For the ablations, we report experiments on SST-2, MR, TREC and AGNews, using one train seed (instead of four), and four verbalizers and five data seeds (as in main experiments). Varying the number of training examples We vary the number of training examples ( K ) and report the average accuracy in Figure 3. All methods achieve higher accuracy as K increases. While we confirm strong performance of channel prompt tuning with K  16 , head tuning outperforms channel head tuning when K = 64 . When K = Full , both direct prompt tuning and head tuning outperform channel prompt tuning. We think this is because (1) training signals amplified by channel models (Lewis and Fan, 2018) are more significant when K is small, and (2) channel models are more beneficial when labels on the training data are imbalanced (confirmed in the next ablation), which is more likely to happen with smaller K . It is also worth noting that our experiment with K = Full confirms the finding from Lester et al. (2021) that direct prompt tuning matches the performance of all finetuningfinetuning all parameters of the LMwhile being much more parameter-Direct All Direct Head Direct Prompt Channel Prompt 50 70 90 0 0.125 0.25 0.375 0.5 A cc u r ac y ( % ) p-0 0.125 0.25 0.375 0.5 p-No Upsample Upsample K=16 K=64 Figure 4: Impact of imbalance in labels. efficient. This only holds with K = Full ; in a few-shot setup, all finetuning significantly outperforms other methods. This contradicts traditional analysis that having less trainable parameters is better when the training data is scarce (Ng and Jordan, 2002). It is likely because such analysis did not take into account language model pretraining, which gives supervision to the model yet is not the training data for an end task. Impact of imbalance in labels On binary datasets (SST-2 and MR), we vary the label imbalance in the training data with K = { 16 , 64 } . Specifically, let C = { c + , c  } and p  = |{ ( x, c )  D| c = c  }| / |D| , i.e., the ratio of c  in the training data. We vary p  to be { 0 , 0 . 125 , 0 . 250 , 0 . 375 , 0 . 5 } . p  = 0 . 5 means the labels are perfectly balanced, and p  = 0 means that labels in the training data only include c + . We additionally compare with upsampling baselines 5322 Data Zero-shot Finetuning Direct++ Channel DirectAll DirectHead DirectTrans DirectPrompt ChannelPrompt SST-2 80.3/76.9 77.1/74.8 50.2/49.1 50.2/49.1 50.2/49.1 50.2/49.1 85.5 / 82.5 SST-5 33.3/28.8 29.2/27.7 40.1 / 34.8 34.3/28.0 32.6/24.5 30.0/18.1 37.5/32.6 MR 77.4/73.2 74.3/69.3 50.0/50.0 50.0/50.0 50.0/50.0 50.0/50.0 80.9 / 74.8 CR 77.9/69.7 65.8/60.2 50.0/50.0 50.0/50.0 50.0/50.0 50.0/50.0 80.9 / 74.8 TREC 27.7/12.6 30.5/19.4 50.8 /31.0 44.8/29.6 44.6/ 32.8 33.9/17.4 34.3/26.0 Subj 52.0/48.8 57.8/51.5 50.0/50.0 50.0/50.0 50.0/50.0 50.0/50.0 66.6 / 57.6 Table 5: Model performance when there is at least one label at test time that was unseen during training. where we upsample training examples with infrequent labels so that the model has seen an equal number of examples per label during training. Results are reported in Figure 4. All direct models are sensitive to the imbalance in training data, even though they benefit from upsampling when p  is small. Channel prompt tuning is insensitive to the imbalance, and significantly outperforms direct models when p  is small; it even outperforms all finetuning when p  < 0 . 25 . When p  is near to 0.5, direct head tuning matches or outperforms channel prompt tuning. It is also worth noting that direct prompt tuning with upsampling matches or outperforms all finetuning and head tuning when p  is small. We experiment with a challenging scenario where the model must generalize to unseen labels. While it may be seen as an extreme scenario, this is often a practical setting, e.g., the problem is defined with a set of labels but later an addition of the new label may be needed. First, we sample K training examples as in main experiments but excluding one random label, so that at least one label at test time was unseen during training. Table 5 reports the results. All direct models are unable to predict the label that is unseen at training time. However, channel prompt tuning can predict unseen labels and achieves considerably better performance than zero-shot. It outperforms all finetuning on 2-way classification datasets, and outperforms head tuning on five datasets except for TREC on which head tuning achieves very strong performance on seen labels. Next, we run zero-shot transfer learning, where the model is trained on one dataset and is tested on another dataset. Here, head tuning is not applicable when the labels are not shared between two datasets. Figure 5 shows the results. Channel prompt tuning outperforms all direct models including all finetuning on all datasets except for TREC. It is particularly competitive when the tasks are inherently similar, e.g., transfer between 2-way sentiment analysis and 5-way sentiment analysis in the first three figures. In fact, in such cases, perfor-5323 mance is close to the models trained on in-domain data. When tasks are inherently different, e.g., the rest of the figures in Figure 5, gains over zero-shot performance are relatively small; we think more work should be done to make cross-task transfer better and to discover when it is possible. In this work, we introduced a noisy channel approach for few-shot text classification through LM prompting, where we either provide demonstrations to the LM or tune the prompt embeddings in the continuous space. Our experiments on eleven datasets show that channel models significantly outperform their direct counterparts, mainly because of their stability, i.e., lower variance and better worst-case accuracy. We also found that direct head tuning is more competitive than previously thought, and different methods are preferred given different conditions. Specifically, channel prompt tuning is preferred in the following scenarios. K is small Channel prompt tuning is more competitive when there are fewer training examples. We hypothesize two reasons: (1) Channel models are more stable (i.e., achieve low variance and high worst-case accuracy), unlike direct models that are highly unstable with small k (Zhao et al., 2021; Perez et al., 2021; Lu et al., 2021). (2) Channel models provide more signals by requiring the model to explain the input word-by-word (as claimed in Lewis and Fan (2018)) which is beneficial in the low data regime. Data is imbalanced or |C| is large When the training data is even slightly imbalanced, no direct models are competitive. We think this is because the LM head relies too much on unconditional distributions of labels. Channel prompt tuning is less sensitive because labels are only a conditioning variable. Label imbalance in the training data is a real-world problem, especially when k is small and |C| is large. We thus suggest this is an important area for future work. Generalization to unseen labels is required All direct models are unable to predict labels that are unseen during training, indicating that they overfit in the label space. In contrast, channel models can predict unseen labels, likely because the label space is indirectly modeled. This is in line with prior work that shows channel models are more competitive under a distribution shift (Yogatama et al., 2017; Lewis and Fan, 2018). Task is closer to language modeling If the task is too different from language modeling even with carefully chosen verbalizers (e.g., TREC and Subj), head tuning outperforms prompt tuning. This is likely because it benefits from directly updating the parameters of the LM. This may mean that causal LMs are not suitable for all tasks, or we need more sophisticated methods to apply causal LMs for such tasks without updating the LM parameters. Limitations and future work While we show that channel models are competitive in few-shot text classification, there are limitations that provide avenues for future work. First, it is not as easy to use channel models for non classification tasks where modeling prior distributions is non-trivial. We think future work can obtain the prior with a separate model and incorporate it to the conditional LM as done by Lewis and Fan (2018), potentially with beam search decoding as in Yu et al. (2017); Yee et al. (2019). Second, while this paper focuses on causal LMs, it is an open question how to use a channel model with masked LMs. Although we think channel models are not inherently restricted to causal LMs, the specific way in which existing masked LMs are pretrained makes it hard to use channel models without updating the LM parameters, e.g., masked LMs are not trained to generate long sentences. One recent approach uses a label-conditioning objective (Tam et al., 2021) as a clever way to introduce a channel-like model with existing masked LMs. Extending and further integrating these different approaches would be important for using channel models in a wider range of scenarios. We thank Ari Holtzman, Eric Wallace, Gabriel Il-harco, Jungsoo Park, Myle Ott, Peter West and Ves Stoyanov for their helpful comments and discussion. This research was supported by NSF IIS-2044660, ONR N00014-18-1-2826, an Allen Distinguished Investigator Award, and a Sloan Fellowship.
abstain abstain objective result method abstain abstain abstain abstain abstain abstain abstain abstain method abstain result abstain result abstain objective result method other other other other other method other other objective other other other other other other abstain abstain other other objective method other abstain method abstain abstain abstain abstain abstain abstain abstain method method other method abstain method method abstain abstain method abstain method abstain abstain other abstain method abstain abstain method method abstain abstain abstain method method method abstain method abstain abstain method abstain abstain method abstain other abstain abstain abstain method abstain method abstain method method method method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain result result abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method result abstain result method method abstain abstain other other