The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFFb etter few-shot fine-tuning of language models 1 a suite of simple and complementary techniques for fine-tuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning. 2 1 Introduction The GPT-3 model (Brown et al., 2020) has made waves in the NLP community by demonstrating astounding few-shot capabilities on myriad language understanding tasks. Given only a natural language prompt and a few demonstrations of the task, GPT-3 is able to make accurate predictions without updating any of the weights of its underlying lan-* The first two authors contributed equally. guage model. However, while remarkable, GPT-3 consists of 175B parameters, which makes it challenging to use in most real-wold applications. In this work, we study a more practical scenario in which we only assume access to a moderately-sized language model such as BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019), and a small number of examples (i.e., a few-shot setting), which we can use to fine-tune the weights of the language model. This setting is appealing as (1) such models can be trained on typical research hardware; (2) few-shot settings are realistic, as it is generally both easy to acquire a few annotations (e.g., 32 examples) and efficient to train on them; and (3) updating parameters typically leads to better performance. Inspired by GPT-3's findings, we propose several novel strategies for expanding its few-shot learning abilities to our setting, considering both classification andfor the first timeregression. First, we follow the route of prompt-based prediction, first developed by the GPT series (Radford et al., 2018, 2019; Brown et al., 2020) for zero-shot prediction and recently studied by PET (Schick and Schutze, 2021a,b) for fine-tuning. Prompt-based prediction treats the downstream task as a (masked) language modeling problem, where the model directly generates a textual response (referred to as a label word ) to a given prompt defined by a task-specific template (see Figure 1(c)). Finding the right prompts, however, is an artrequiring both domain expertise and an understanding of the language model's inner workings. Even if significant effort is invested, manual prompts are likely to be suboptimal. We address this issue by introducing automatic prompt generation, including a pruned brute-force search to identify the best working label words, and a novel decoding objective to automatically generate templates using the generative T5 model (Raffel et al., 2020)all of which only require the few-shot training data. This allows us MLMhead  no utterly   MLMhead great (label:positive) terrible (label:negative)  label:positive label:negative  CLShead [CLS] No reason to watch . to cheaply obtain effective prompts that match or outperform our manually chosen ones. Second, we adopt the idea of incorporating demonstrations as additional context. GPT-3's naive in-context learning paradigm picks up to 32 randomly sampled examples, and concatenates them with the input. This method is not guaranteed to prioritize the most informative demonstrations, and mixing random examples from different classes together creates long contexts which can be hard to learn from. Additionally, the number of usable demonstrations is bounded by the model's maximum input length. We develop a more refined strategy, where, for each input, we randomly sample a single example at a time from each class to create multiple, minimal demonstration sets . We also devise a novel sampling strategy that pairs inputs with similar examples, thereby providing the model with more discriminative comparisons. We present a systematic evaluation for analyzing few-shot performance on 8 single-sentence and 7 sentence-pair NLP tasks. We observe that given a small number of training examples, (1) prompt-based fine-tuning largely outperforms standard fine-tuning; (2) our automatic prompt search method matches or outperforms manual prompts; and (3) incorporating demonstrations is effective for fine-tuning, and boosts few-shot performance. Together, these simple-yet-effective methods contribute towards a dramatic improvement across the tasks we evaluate on, and we obtain gains up to 30% absolute improvement (11% on average) compared to standard fine-tuning. For instance, we find that a RoBERTa-large model achieves around 90% accuracy on most binary sentence classification tasks, while only relying on 32 training examples. We refer to our approach as LM-BFF, better few-shot fine-tuning of language models: a strong, task-agnostic method for few-shot learning. Language model prompting. The GPT series (Radford et al., 2018, 2019; Brown et al., 2020) fueled the development of prompt-based learning, and we follow many of its core concepts. We are also greatly inspired by the recent PET work (Schick and Schutze, 2021a,b), although they mainly focus on a semi-supervised setting where a large set of unlabeled examples are provided. We only use a few annotated examples as supervision, and also explore automatically generated prompts and fine-tuning with demonstrations. Furthermore, we deviate from their evaluation by providing a more rigorous framework, as we will discuss in 3. Finally, there is a large body of work on prompting for mining knowledge from pre-trained models (Trinh and Le, 2018; Petroni et al., 2019; Davison et al., 2019; Talmor et al., 2020, inter alia ). Different from these works, we focus on leveraging prompting for fine-tuning on downstream tasks. Automatic prompt search. Schick and Schutze (2021a) and Schick et al. (2020) explore ways of identifying label words automatically, however, none of these results lead to better performance compared to hand-picked ones. In contrast, our method searches over both templates and label words, and is able to match or outperform our manual prompts. Several other attempts have been made in additionyet these approaches either operate in limited domains, such as finding patterns to express specific relations (Jiang et al., 2020), or require a large number of examples for gradient-guided search (Shin et al., 2020; Zhong et al., 2021). Our approach aims to develop general-purpose search methods that rely only on a few annotations. Fine-tuning of language models. A number of recent studies have focused on better methods for fine-tuning language models (Howard and Ruder, 2018; Dodge et al., 2020; Lee et al., 2020; Zhang et al., 2021). These works mainly focus on optimization and regularization techniques to stabilize fine-tuning. Here we use standard optimization techniques, and instead mainly focus our efforts on better prompt-based fine-tuning in a more extreme few-shot setting. We anticipate that results of these studies are largely complementary to ours. Few-shot learning. Broadly speaking, our setting is also connected to other few-shot learning paradigms in NLP, including (1) semi-supervised learning (Miyato et al., 2017; Xie et al., 2020; Chen et al., 2020), where a set of unlabeled examples are given; (2) meta-learning (Yu et al., 2018; Han et al., 2018; Bansal et al., 2020a,b; Bao et al., 2020), where a set of auxiliary tasks are given; and (3) intermediate training (Phang et al., 2018; Yin et al., 2020), where a related, intermediate task is given. We deviate from these settings by making minimal assumptions about available resources: we only assume a few annotated examples and a pre-trained language model. Our focus is on understanding how far we can push without any other advantages. Task formulation. In this work, we assume access to a pre-trained language model L that we wish to fine-tune on a task D with a label space Y . For the task, we only assume K training examples per class 3 for the task's training set D train , such that the total number of examples is K tot = K  |Y| , and D train = { ( x i in , y i ) } K tot i =1 . Our goal is then to develop task-agnostic learning strategies that generalize well to an unseen test set ( x test in , y test )  D test . For model selection and hyper-parameter tuning, we assume a development set D dev , of the same size as the few-shot training set, i.e., |D dev | = |D train | . This distinction is important: using a larger development set confers a significant advantage (see our 3 For regression, we partition the data into two classes according to being above or below the median value. experiments in Appendix A), and subverts our initial goal of learning from limited data. 4 For all of the following experiments (unless specified other-wise), we take L = RoBERTa-large and K = 16 . Evaluation datasets. We conduct a systematic study across 8 single-sentence and 7 sentence-pair English tasks, including 8 tasks from the GLUE benchmark (Wang et al., 2019), SNLI (Bowman et al., 2015), and 6 other popular sentence classification tasks (SST-5, MR, CR, MPQA, Subj, TREC). All of the dataset details are provided in Appendix B. For single-sentence tasks, the goal is to make a prediction based on an input sentence x in = x 1 , such as whether a movie review is positive or not. For sentence-pair tasks, the goal is to take a pair of input sentences x in = ( x 1 , x 2 ) and predict the relationship between them. We also interchangeably refer to the inputs as < S 1 > or ( < S 1 > , < S 2 > ). Note that we mainly use SST-2 and SNLI for pilot experiments and model development, making it close to a true few-shot setting, at least for all the other datasets we evaluate on. Evaluation protocol. Systematically evaluating few-shot performance can be tricky. It is wellknown that fine-tuning on small datasets can suffer from instability (Dodge et al., 2020; Zhang et al., 2021), and results may change dramatically given a new split of data. To account for this, we measure average performance across 5 different randomly sampled D train and D dev splits. This issue has also been discussed in Schick and Schutze (2021b) they suggest using a fixed set of training examples. We argue that sampling multiple splits gives a more robust measure of performance, and a better estimate of the variance. We also observe that hyper-parameters can make a significant difference, thus we sweep multiple hyper-parameters for each data sample, and take the best setting as measured on the D dev of that sample (see Appendix C.1). Given a masked language model L , we first convert input x in to a token sequence  x , and the language model L then maps  x to a sequence of hidden vectors { h k  R d } . During standard fine-tuning, we usually take  x single = [CLS] x 1 [SEP] or  x pair = [CLS] x 1 [SEP] x 2 [SEP] . For down-4 In contrast, Schick and Sch  utze (2021a,b) do not use a development set, and adopt a set of hyper-parameters based on practical considerations. This is akin to shooting in the dark on a setting that we show can have unintuitive outcomes. stream classification tasks with a label space Y , we train a task-specific head, softmax( W o h [CLS] ) , by maximizing the log-probability of the correct label, where h [CLS] is the hidden vector of [CLS] , and W o  R |Y| d is a set of randomly initialized parameters introduced at the start of fine-tuning. Similarly, for a regression task, we can introduce w o  R d and optimize the mean squared error between w o  h [CLS] and the gold label. In either case, the number of new parameters can be substantial for example, a simple binary classification task will introduce 2,048 new parameters for a RoBERTa-large modelmaking it challenging to learn from a small amount of annotated data (e.g., 32 examples). An alternative approach to solving this problem is prompt-based fine-tuning , in which L is directly tasked with auto-completing natural language prompts. For instance, we can formulate a binary sentiment classification task using a prompt with input x 1 (e.g.,  No reason to watch it . ) as: x prompt = [CLS] x 1 It was [MASK] . and let L decide whether it is more appropriate to fill in  great (positive) or  terrible (negative) for [MASK] . We now formalize this approach for classification and regression (4.1 and 4.2), and discuss the importance of prompt selection (4.3). Let M : Y  V be a mapping from the task label space to individual words 5 in the vocabulary 5 More generally, we can consider a one-to-many mapping M : Y  2 |Y| in which we map labels to sets of words. However, we did not find significant gains in our experiments. V of L . Then for each x in , let the manipulation x prompt = T ( x in ) be a masked language modeling (MLM) input which contains one [MASK] token. In this way, we can treat our task as an MLM, and model the probability of predicting class y  Y as: p where h [MASK] is the hidden vector of [MASK] and w v denotes the pre-softmax vector corresponding to v  V . When supervised examples { ( x in , y ) } are available, L can be fine-tuned to minimize the cross-entropy loss. It is important to note that this approach re-uses the pre-trained weights w v and does not introduce any new parameters. It also reduces the gap between pre-training and fine-tuning, making it more effective in few-shot scenarios. We assume the same basic setup as in classification, but treat the label space Y as a bounded interval [ v l , v u ] . Inspired by Mettes et al. (2019), we model the problem as an interpolation between two opposing poles, { y l , y u } , with values v l and v u respectively. For instance, we can formulate our previous sentiment analysis task as a regression problem in the range [0 , 1] , where we slide between  terrible ( v l = 0 ) and  great ( v u = 1 ). In this way, we can express y as a mixture model : y = v l  p ( y l | x in ) + v u  p ( y u | x in ) , (2) M : { y l , y u }  V , and model p ( y u | x in ) the same as Eq. (1). We fine-tune L to minimize the KL-divergence between the inferred p ( y u | x in ) and the observed mixture weight, ( y  v l ) / ( v u  v l ) . 4.3 Manual prompts: the good and the bad The key challenge is to construct the template T and label words M ( Y ) we refer to these two together as a prompt P . Previous works (Schick and Schutze, 2021a,b) hand-craft both the templates and label words, which usually requires domain expertise and trial-and-error. Table 1 summarizes manual templates and label words chosen for each dataset in our experiments. These templates and label words were designed by intuition, and by considering formats used in previous literature. To better understand what constitutes a good template or label word, we conduct a pilot study on SST-2 and SNLI. Table 2 shows that different prompts can lead to substantial differences in final accuracy. Specifically, when a template is fixed, the better the label words match the semantic classes, the better the final accuracy is ( great / terrible > good / bad > cat / dog ). In extreme cases where we swap plausible label words (e.g., terrible / great ), we achieve the worst overall performance. 6 Furthermore, with the same set of label words, even a small change in the template can make a difference. For example, for SNLI, if we put [MASK] at the end, or swap sentence order, we observe a > 10% drop. The above evidence clearly underlines the 6 It is unclear, however, why RoBERTa thinks that cat is more positive than dog. importance of selecting good templates and label words. Searching for prompts, however, is hard, as the search space can be very largeespecially for the template. Even worse, we only have a few examples to use to guide our search, which can easily overfit. We will address these issues next. We now explore principled ways of automating the search process for label words (5.1) and templates (5.2). Our goals are to reduce the human involvement required to design prompts, and to find more optimal settings than those that we manually choose. Here, we assume a classification task, but the process for regression is analogous. We first study how to construct a label word mapping M that maximizes accuracy on D dev after fine-tuning, given a fixed template T . Naively searching all possible assignments, however, is (1) generally intractable, as the search space is exponential in the number of classes; and (2) prone to overfitting, as we will tend to uncover spurious correlations given only a few annotations. As a simple solution, for each class c  Y , we construct a pruned set V c  V of the top k vocabulary words based on their conditional likelihood using the initial L . That is, let D c train  D train be the subset of all examples of class c . We take V c as Top k v V where PL denotes the output probability distribution of L . To further narrow down the search space, we find the top n assignments over the pruned space that maximize zero-shot accuracy on D train (both n and k are hyper-parameters, see Appendix C.2). Then we fine-tune all top n assignments, and rerank to find the best one using D dev . This approach is similar to the automatic verbalizer search methods in Schick and Schutze (2021a); Schick et al. (2020), except that we use a much simpler search process (brute-force) and also apply re-ranking which we find to be quite helpful. Next, we study how to generate a diverse set of templates {T } automatically from a fixed set of label words M ( Y ) . To address this challenging problem, we propose to use T5 (Raffel et al., 2020), Best template Generated templates Training examples for label:negative T5  Training examples for label:positive  Decode < S 1 > A [MASK] one. a large pre-trained text-to-text Transformer. T5 is pre-trained to fill in missing spans (replaced by T5 mask tokens, e.g., <X> or <Y> ) in its input. For example, given the input  Thank you <X> me to your party <Y> week , T5 is trained to generate  <X> for inviting <Y> last <Z> , meaning that  for inviting  is the replacement for <X> and  last  is the replacement for <Y> . This is well suited for prompt generation: we can simply take input sentences from D train and let the T5 model construct the template T , without having to specify a pre-defined number of tokens for it. < S 1 >  <X> M ( y ) <Y> < S 1 > , < S 1 >  < S 1 > <X> M ( y ) <Y> , < S 1 > , < S 2 >  < S 1 > <X> M ( y ) <Y> < S 2 > . As shown in Figure 2, we rely on the T5 model to fill in the placeholders. When decoding, our goal here is to find an output that can work well for all examples in D train , i.e., the output template T that maximizes (cid:80) ( x in ,y ) D train log P T5 ( T | T g ( x in , y )) , where P T5 denotes the output probability distribution of T5. It can be decomposed according to: |T | |T | We use beam search to decode multiple template candidates. Concretely, we use a wide beam width (e.g., 100) to cheaply obtain a large set of diverse templates. We then fine-tune each generated template on D train and use D dev to either pick the single template with the best performance (Table 3), or 7 We consider putting the label word both before and after the input sentence for single-sentence tasks. However, we find that it is always better to put the label words in the middle (between the two sentences) for sentence-pair tasks. the top k templates to use as an ensemble (Table 4). Though it might appear to be expensive to fine-tune the model on each individual template, this is fast in practice due to the small size of D train , and is also fully automated: making it easy to use, compared to manually tuning prompts for each dataset. In this section, we study whether we can leverage demonstrations when fine-tuning medium-sized LMs, and find better ways to exploit them. GPT-3's naive approach to in-context learning simply involves concatenating the input with up to 32 examples randomly drawn from the training set. This approach is suboptimal as (1) the number of available demonstrations is bounded by the model's maximum input length; 8 and (2) mixing numerous random examples from different classes together creates extremely long contexts which can be hard to leverage, especially for a smaller model. To address these issues, we propose a simpler solution: at each training step, we randomly sample one 9 example (cid:0) x ( c ) in , y ( c ) (cid:1) D train from each class, convert it into T (cid:0) x ( c ) in (cid:1) with [MASK] replaced by M ( y ( c ) ) we denote this as  T (cid:0) x ( c ) in , y ( c ) (cid:1) and then concatenate them with x in (Figure 1(c)): T Here  denotes concatenation of input sequences. During both training and inference we sample multiple demonstration sets for each x in . Note that both x in and demonstration examples are sampled from the same set D train during training. At testing time, we still sample demonstration sets from D train and ensemble predictions across all sets. We observe that controlling the construction of the demonstration examples { ( x ( c ) in , y ( c ) ) } is crucial for good final performance. For example, if the set of contrastive demonstrations x ( c ) in are all dramatically differentfrom each other, or from the query x in then it becomes challenging for the language model to decipher meaningful patterns. As a result, the model may simply ignore 8 GPT-3 uses a context size of 2,048 while most smaller language models (e.g., RoBERTa) have a context size of 512. 9 We also explored sampling multiple examples per class, but did not observe any improvements. the context, or even get confused by the additional examples. To address this issue, we devise a simple strategy in which we only sample examples that are semantically close to x in . Specifically, we use a pre-trained SBERT (Reimers and Gurevych, 2019) model to obtain embeddings for all input sentences (for sentence-pair tasks, we use the concatenation of the two sentences). Here we just feed the raw sentences without the templates into SBERT. For each query x in and each label c  Y , we sort all training instances with the label x  D c train by their similarity score to the query cos( e ( x in ) , e ( x )) , and only sample from the top r = 50% instances for each class to use as demonstrations. We present our main results, and address several research questions pertaining to our LM-BFF approach. Implementation details are in Appendix C. 7.1 Main results We use a RoBERTa-large model and set K = 16 in our experiments. A comparison of using RoBERTa vs BERT can be found in Appendix D. For automatic prompt search, in our main table we report automatic template search only (which consistently performs the best, see Table 5). To put our results in perspective, we compare to a number of baselines, namely (1) standard fine-tuning in our few-shot setting; (2) standard fine-tuning using the full training set; (3) simply taking the most frequent class (measured on the full training set); (4) prompt-based zero-shot prediction where we take our manual prompts and use L out-of-the-box without using any training examples; and (5) GPT-3 in-context learning, where we use the same prompt-based zero-shot setting, but augment the context with randomly sampled 32 demonstrations (and still use RoBERTa-large, not GPT-3). Single-prompt results. Table 3 shows our main results using a single prompt, either from our manually designed ones (Table 1) , or the best generated ones. First, prompt-based zero-shot prediction achieves much better performance than the majority class, showing the pre-encoded knowledge in RoBERTa. Also, GPT-3 in-context learning does not always improve over zero-shot prediction, likely because smaller language models are not expressive enough to use off-the-shelf like GPT-3. Second, prompt-based fine-tuning can greatly outperform standard fine-tuning, both when using a manual prompt or a generated one. CoLA is one interesting exception, as the input may be a nongrammatical sentence which is out of the distribution of L . Generally, our automatically searched templates can achieve comparable or even higher results than manual ones, especially for tasks in which constructing strong manual templates is less intuitive (e.g., TREC, QNLI and MRPC). Finally, using demonstrations in context leads to consistent gains in a majority of tasks. In summary, our combined solutionfine-tuning with automatically searched templates and sampled demonstration setsachieves a 30% gain on SNLI compared to standard fine-tuning, and 11% gain on average. Ensemble results. An advantage of automatic prompt search is that we can generate as many prompts as we want, train individual models, and create large ensembles. PET (Schick and Schutze, 2021a,b) also ensembles multiple models trained with manual prompts. 10 In Table 4, we make a direct comparison of our searched prompts and PET's manual prompts on MNLI and RTE (two 10 They then use unlabeled data and distillation to get a single model, which is outside of our scope. datasets that we evaluate in common). 11 As the results show, an ensemble with multiple templates always improves performance. An ensemble of the same number of automatic templates achieves comparable or better performance than the ensemble of PET's manual prompts. Increasing the number of automatic templates brings further gains. Table 5 gives the results of using manual vs automatic prompts. For automatic prompts, we compare template search (Auto T), label word search (Auto L), and a joint variant (Auto T + L) in which we start from manual label words, apply Auto T, and then Auto L. In most cases, Auto T achieves comparable or higher performance than manual ones, and is consistently the best variant. Auto L outperforms manual prompts on TREC and MRPCbut is considerably worse on SNLI. Auto T + L is often better than Auto L, but only sometimes better than Auto T. Table 6 shows examples from Auto T and Auto L (A full list in Appendix E). Auto T templates generally fit the context and label words well, but can contain biased peculiarities (e.g.,  { Yes/No } , no  in SNLI). For Auto L words, things are mixed: while most look intuitively reasonable, there are also some mysterious abnormalities (e.g.,  Hi  for the entailment class in SNLI). 11 In the PET NLI templates, the hypothesis is put before the premise, which we actually found to be suboptimal. In our experiments, we swap the two and get better results. Table 7 compares the performance of demonstrations using uniform sampling to selective sampling by SBERT. We acknowledge that SBERT is trained on SNLI and MNLI datasets, thus we also tried a simple sentence encoder using mean pooling of hidden representations from RoBERTa-large. We find that in either case, using selective sampling outperforms uniform sampling, highlighting the importance of sampling similar examples for incorporating demonstrations in context. Figure 3 illustrates how standard fine-tuning and our LM-BFF compare as K increases. For a simple task such as SST-2 (also see MR, CR and MPQA in Table 3), despite using only 32 total examples, LM-BFF has already nearly saturated its performance and is comparable to standard fine-tuning over the entire dataset. On the harder task of SNLI, LM-BFF continues to improve as K increases while still maintaining a performance gap over standard fine-tuning, until the two converge around K = 256 . Reformulating NLP tasks as MLM has exciting implications for few-shot learning, but also has limitations. First, while LM-BFF greatly outperforms standard fine-tuning, Table 3 shows that, overall, the performance still substantially lags behind fine-tuning with thousands of examples, especially for harder tasks. Additionally, just like standard fine-tuning, our results also suffer from high variance. As described in 2, several recent studies have tried to counter instability in few-shot fine-tuning and we expect these methods to also help here. With respect to automatic prompt generation, despite its effectiveness, we still find it practically challenging to expand the search space, or generalize well based on only approximately 32 examples. This is partly due to our lingering reliance on some manual designeither manual templates (for label word search) or manual label words (for template search), which allows us to get our search off the ground, but does also bias it towards areas of the search space that we might have already imagined. Finally, it is important to clarify that LM-BFF favors certain tasks which (1) can be naturally posed as a fill-in-the-blank problem; (2) have relatively short input sequences; and (3) do not contain many output classes. Issues (2) and (3) might be ameliorated with longer-context language models (e.g., Beltagy et al., 2020). For tasks that are not straightforward to formulate in prompting, such as structured prediction, issue (1) is more fundamental. We leave it as an open question for future work. In this paper we presented LM-BFF, a set of simple but effective techniques for fine-tuning language models using only a few examples. Our approach proposes to (1) use prompt-based fine-tuning with automatically searched prompts; and (2) include selected task demonstrations (training examples) as part of the input context. We show that our method outperforms vanilla fine-tuning by up to 30% (and 11 % on average). We concluded by discussing the limitations of our approach, and posed open questions for future study. We thank the members of Princeton, MIT, Ts-inghua NLP groups and the anonymous reviewers for their valuable feedback. TG is supported by a Graduate Fellowship at Princeton University and AF is supported by an NSF Graduate Research Fellowship. This research is also partly supported by a Google Research Scholar Award.
abstain method method objective method objective abstain abstain abstain abstain abstain method abstain objective abstain abstain abstain abstain abstain abstain abstain result method abstain abstain abstain objective objective method result objective result abstain other method method objective method other method other other objective other objective other other other method abstain other other method abstain abstain method method method method method method abstain method abstain abstain method method abstain abstain abstain method abstain method method abstain method abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain method abstain abstain abstain abstain result result result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain result abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain method objective result abstain other other other