In text classification tasks, useful information is encoded in the label names. Label semantic aware systems have leveraged this information for improved text classification performance during fine-tuning and prediction. However, use of label-semantics during pre-training has not been extensively explored. We therefore propose L abel S emantic A ware P re-training (LSAP) to improve the generalization and data efficiency of text classification systems. LSAP incorporates label semantics into pre-trained generative models (T5 in our case) by performing secondary pre-training on labeled sentences from a variety of domains. As domain-general pre-training requires large amounts of data, we develop a filtering and labeling pipeline to automatically create sentence-label pairs from unlabeled text. We perform experiments on intent (ATIS, Snips, TOPv2) and topic classification (AG News, Yahoo! An-swers). LSAP obtains significant accuracy improvements over state-of-the-art models for few-shot text classification while maintaining performance comparable to state of the art in high-resource settings. Large pre-trained language models have enabled better performance on many NLP tasksespecially in few-shot settings (Brown et al., 2020; Schick and Schutze, 2021a; Wu and Dredze, 2020). More informative representations of textual inputs often leads to much higher downstream performance on NLP applications, which explains the rapid and general adoption of models such as (Ro)BERT(a) (Devlin et al., 2019; Liu et al., 2019), GPT-2 (Radford et al., 2019), and T5 (Raffel et al., 2020). However, while these models are often used to effectively encode inputs, fewer works have attempted to give models access to informative representations of labels as well. Work done as an intern at Amazon Web Services. Most discriminative approaches to text classification only give the model access to label indices. A recent stream of work has obtained significant improvements in structured prediction tasks by using sequence-to-sequence (seq2seq) models to generate labels (Athiwaratkun et al., 2020; Paolini et al., 2021). Yet these generative approaches make use of label semanticsthe meaning of class label namesonly during fine-tuning and prediction. Thus, we propose L abel S emantic A ware P retraining (LSAP) to incorporate label semantics as well as input-label associations into the pre-training step (Figure 1). Our experiments show that LSAP yields higher performance with fewer fine-tuning examples in a variety of domains. Our contributions include the following: 1. A method to incorporate label semantics into generative models during pre-training. 2. A method for creating utterance-intent pairs for label semantic aware pre-training from unlabeled noisy data. Label semantics has been leveraged in many settings and tasks to improve performance and robustness, even before dense embedding representations became standard: Chang et al. (2008) achieve over 80% accuracy on binary text classification tasks without any labeled training examples by giving nave Bayes classifiers rich semantic representations (Gabrilovich and Markovitch, 2007) of labels. Song and Roth (2014) use a similar approach for hierarchical and multinomial classification, finding that dataless procedures approached the performance of (and sometimes outperformed) supervised approaches. More recently, label semantics based on dense embedding representations have become popular especially with the rise of contextualized word embeddings (Peters et al., 2018). One stream of work integrates label representations using label vectors and label attention: label-wise attention networks (LWAN; Mullenbach et al., 2018) are designed for datasets with very large structured label spaces, and where multiple labels can apply to a potentially very long document. Mullenbach et al. (2018) generate label feature vectors and use an LWAN based on convolutional neural networks (CNNs); Rios and Kavuluru (2018) extend the attention mechanism for zero-shot settings. Chalkidis et al. (2020) use BERT to embed the labels for an LWAN. 2 In contrast, our label space is flatter and much smaller, and the input texts are much shorter on average. Another label-semantic-aware approach for massively multi-label text classification is CLESS (Rethmeier and Augenstein, 2022). CLESS performs contrastive pre-training from scratch on a question-answering dataset with a large and sparse label space. Using contrasts between positive and negative answer embeddings, they obtain performance comparable to or better than RoBERTa using only in-domain data. However, CLESS pre-trains and classifies on the same data, and is meant 1 https://github.com/amazon-research/ label-aware-pretrain 2 Note that BERT-Base often performs similarly to or better than BERT-LWAN, and XLNet performs significantly better than BERT-Base in text classification settings that more closely resemble our setting (Yang et al., 2019). We therefore opt to compare to XLNet. for classification in one domain with a very large label space; our setting contains a larger mismatch between pre-training (where we have domain-general data and a large label space) and fine-tuning (where we have domain-specific data and a small label space). We want our approach to be more domain-general and to leverage the simplicity and effectiveness of token and span reconstruction objectives for example, masked language modeling (Devlin et al., 2019) and span denoising (Raffel et al., 2020). Thus, we opt to perform secondary pre-training on data from a variety of domains using an existing model. Other work integrates label embeddings into short-text intent and topic classification systems, more similarly to our task. Gaonkar et al. (2020) use label embeddings from BERT and a label attention mechanism to improve emotion classification accuracy. Generative approaches like those of Rongali et al. (2020); Athiwaratkun et al. (2020); Paolini et al. (2021) implicitly make use of label semantics for text and token classification tasks by generating the labels at prediction time. Rastogi et al. (2019) use embeddings of human-defined schema which guide a dialogue state tracking system. Few-shot text classification entails performing classification after training or tuning a model on only a few examples from the training split of an evaluation set. Recent approaches to this include (Ro)BERT(a)-based (Chen et al., 2020) and especially XLNet-based (Yang et al., 2019) classifiers, prototypical networks (Snell et al., 2017), dynamic memory induction networks (Geng et al., 2020), and generative classification (discussion follows). Text classification contains more specific subtasks. Here, we evaluate on topic classification (TC; labeling text with its general domain, e.g. world news) and intent classification (IC; labeling intentful text with what it is trying to accomplish, e.g. book flight). The scarcity of IC data in many domains prevents the use of many neural text classification methods (Krone et al., 2020). In IC, the input is a conversational utterance and the output is a label describing what the user intends to do; for example, given a set of intents { BookHotel , BookFlight , CheckAccount } and an input utterance I would like a flight to NYC next month, the model should classify the utterance as BookFlight . Recent approaches to few-shot IC include dual encoders (Cer et al., 2018; Henderson et al., 2020; 8319 Casanueva et al., 2020), combining prototypical networks with meta-learning (Krone et al., 2020), a nearest-neighbor discriminative method (Zhang et al., 2020a), and span-level contextualized embedding retrieval (Yu et al., 2021). Generative text classification has become more feasible with the advent of large pre-trained language models (Radford et al., 2019; Brown et al., 2020). Here, one tunes a model to generate a natural language label given an input sequence, which reduces the train-test gap and does not require architectural changes. Pattern-exploiting training (PET; Schick and Schutze, 2021a,b) entails formatting train and test examples as cloze-style prompts, where the label is generally one word. Here, a language model can see the inputs and the embeddings of the output classes during tuning and prediction, though unlike our approach, there is no domain-general training step. Prompt tuning (Lester et al., 2021; Gao et al., 2021) improves PET by automatically optimizing the prompt design. Our approach is even more widely applicable in that we give the model concatenated utterances and labels and allow it to transduce to variable-length label sequences, rather than reformatting the evaluation data into a cloze format where the label must be one word/token. Sequence-to-sequence (seq2seq) approaches based on pointer/copy mechanisms (Rongali et al., 2020) or T5 (Raffel et al., 2020) tend to be effective and more data-efficient for text and token classification in semantic parsing tasks, including slot labeling (Athiwaratkun et al., 2020) and entity-relation extraction (Paolini et al., 2021). In this method, one trains or fine-tunes a seq2seq model to transduce from an unlabeled text sequence to a sequence with labeled spans (or to just the label). Our approach is based on T5, but unlike prior work, we perform a label semantic aware secondary pre-training step on a variety of datasets before fine-tuning. Our approach, LSAP, performs a secondary pretraining step with T5 on a large scale collection of (pseudo-)labeled examples; see Figure 1 for an overview. In addition to training on existing labeled datasets (3.1), we (1) filter unlabeled data using a dialogue act classifier (3.1.1); (2) pseudo-label the utterances that pass the filter using an intent generator (3.1.1); and (3) perform secondary pretraining on the labeled and pseudo-labeled data using T5 (3.2). 3.1 details how we collect, select, and label pre-training data. 3.2 describes the pretraining formats we test in our experiments. Our pre-training data (Table 1) consists of utterances with intent labels. We use intentful utterances for pre-training because intent labels are often more specific, informative, and varied than sentiment or topic labels. For example, consider the intents BookFlight and ViewAirfare versus the topic AirTravel . Our pre-training corpus combines gold (human-labeled), silver (heuristically labeled), and bronze (pseudo-labeled) data. We first collect gold datasets: here, we use a set of non-public benchmark datasets as well as PolyAI Banking (Coope et al., 2020), which yields approximately 130K training examples containing over 1,200 unique intents. 3 To supplement this data, we add the silver WikiHow intents (Zhang et al., 2020b) dataset, which is heuristically labeled. Each WikiHow example consists of an utterance, the longest step in a WikiHow article, and an intent label (the article title with How To removed). Training on this dataset has been shown to improve few-shot IC performance in a variety of domains (Zhang et al., 2020b). Gold and silver conversational datasets are scarce and tend to focus on one or a few narrow domains. To obtain more data from a larger variety of domains, we propose a filtering and labeling pipeline for converting unlabeled conversational data into pseudo-labeled bronze pre-training data. See Figure 2 for an overview. We start by collecting conversational utterances. We first use the Customer Support on Twitter (CST-witter) dataset; 4 this consists of over 2.81M tweets to and from company customer support agents. We also use the Reddit PushShift dataset; 5 this dataset is large, so we only download most recent comment dumps by date until reaching 100M comments. Dialogue act classifier. In a conversational dataset, many utterances will not be intentful, and thus will not lend themselves to informative labels. For example, the statement Hiking is an 3 We ensure that there is no overlap between the gold data and our evaluation sets. 4 https://www.kaggle.com/thoughtvector/ customer-support-on-twitter 5 https://files.pushshift.io/reddit/comments/ 8320 Dataset Quality Description Training Examples Intents Internal benchmark data Gold Benchmark datasets. 15+ domains from English locales. 119,920 1,172 PolyAI Banking Gold Online banking queries. 10,016 77 WikiHow Intent Classification Silver Automatically labeled intent classification dataset. Intent is article title with How to removed, and utterance is the longest step in the article. outdoor activity does not express a clear goal out-of-context (i.e., it is non-intentful), whereas Buy a plane ticket to NYC for next month is a command with a clear goal. Applying an intent label to a non-intentful utterance may lead to supervision that is harmful to downstream performance. To filter for intentful utterances, we tune a RoBERTa-based binary classifier (intentful vs. non-intentful) using Multi-Domain Goal-Oriented Dialogue (Mul-tiDoGO; Peskov et al., 2019) and Schema-guided Dialogue (SGD; Rastogi et al., 2019; Kale and Rastogi, 2020). For MultiDoGO, we treat greet-ings/goodbyes, thank yous, and other generic intents as non-intentful/negative examples; we treat all other intents that are not out-of-domain as intent-ful/positive examples. For SGD, we treat any utterance tagged with INFORM intents as non-intentful; utterances with intents tagged as REQUEST are treated as intentful. When evaluating on a held-out set of MultiDoGO and SGD, the classifier achieves 98% precision. To evaluate the precision of the classifier on our newly filtered data, we randomly sample 150 utterances (per dataset) tagged as intentful by the classifier and calculate the proportion that are actually intentful, as judged by human evaluation. For CSTwitter, we obtain 91% precision; this high precision may be due to the dataset being composed primarily of intentful customer-service-focused queries. For Reddit, we initially obtained 54% precision. We qualitatively find that the probability assigned by the classifier to the positive label correlates well with intentfulness, so for Reddit, we exclude all examples to which our classifier assigned a positive-label probability lower than the median for all utterances tagged as positive examples. After probability thresholding, we obtain 76% precision on Reddit. Intent generator. To label the intentful utterances, we train a T5-based generative intent labeler. We fine-tune T5 on the gold and silver data to transduce from utterances to intents (e.g., intent classification: Find me a hotel in NYC  Book hotel), and then apply this tuned model to the filtered utterances. We find that this model gen-8321 Figure 3: Pre-training formats that we test in our experiments. erates intents not seen in the training set for 37% of the utterances. These novel intents often but do not always demonstrate lexical overlap with the utterance (e.g., i'm wondering if there's fireworks for sale there?  Fireworks for sale) or add specific descriptors to intents from the training set (e.g., Find a job appears in the training set, but our model generates Find a job in the UK which does not). We experiment with different pre-training formats, all but one of which are based on T5's span denoising objective. See Figure 3 for an overview of our formats and an example of each. First, we try random span denoising . In this approach, we concatenate each intent label (in natural language format) to its associated utterance. Then, we randomly noise 15% of the tokens 6 in the utterance-intent input sequence, reconstructing the contiguous noised spans in the output sequence. This is the same objective that T5 uses. 7 Our second approach is intent classification (IC) pre-training , where we supervise T5 with utterances and intents in the same format used for downstream supervised fine-tuning. Here, the input sequence is intent classification:  followed by the utterance. The output sequence is the intent. Finally, we implement label denoising , where we train on utterances and their respective intents as 6 15% is the proportion of tokens that are noised in the original pre-training setup (Raffel et al., 2020). 7 Unlike T5, we do not pack multiple sequences into single training examples. When we pack multiple utterance-intent pairs into training examples of length  512 , we find that performance drops sharply compared to training on individual utterance-intent pairs. before; however, instead of noising random spans, we deterministically noise the entire label sequence in the input sequence and reconstruct it in the output sequence. This is a way of framing the intent classification task as an unsupervised denoising task. This format is formally equivalent to IC pretraining in that we must transduce from utterances to intents, though we empirically show that the unsupervised denoising format matters a great deal for downstream IC performance. We search for non-case-sensitive intent matches with Snips and ATIS in our pre-training dataset, finding that exact intent matches (or intents with Snips/ATIS intents as substrings) are very rare in our data: 682 examples (0.005%) featured exact or substring overlap. We also search for the presence of any lexical overlap between the intents in ATIS/Snips and the intents in our pre-training examples (i.e., if any word in an ATIS or Snips intent appears in a pre-training example's intent, we count it): less than 8,000 examples (0.6%) in our pre-training dataset featured lexical overlap. Utterances tagged with exact intent matches were often not similar to the kinds of utterances seen in the evaluation sets; for example, PlayMusic is a label in Snips that typically refers to playing specific songs or artists, while the same label in our pre-training set often referred to requests to buy or play an instrument. To check whether there was semantic overlap more broadly between the utterances in our pretraining set and those in Snips/ATIS, we obtain sentence embeddings for 5 randomly sampled utterances from each intent in Snips and ATIS, and for each utterance in our pre-training dataset. Sentence embeddings are obtained using a sentence-BERT model (Reimers and Gurevych, 2019); we use all-MiniLM-L6-v2 , as it is both fast and performs best on semantic similarity tasks. We then calculate pairwise semantic similarity between the utterances in our pre-training set and the Snips/ATIS utterances by calculating the cosine similarity between sentence embedding pairs. When observing the most similar utterances, we do not often see much semantic overlap. For example, Play some Mf Doom from the sixties on pandora (in Snips) has the greatest sentence embeddings similarity to I love doom and won't let some launcher protest stop me from playing it (in our dataset); there is 8322 lexical overlap in the utterance, but the sentence in our dataset refers to playing a video game rather than a song. However, there were rare instances where semantic overlap was significant: for example, Add this tune to the Rock Save the Queen playlist (in Snips) was similar to Please add the Fresh Prince of Bel-Air song to this in our dataset. We aim to understand whether LSAP can improve text classification performance over a variety of state-of-the-art (SOTA) baselines. We evaluate LSAP on two text classification tasks: intent classification (IC) and topic classification (TC). IC is the most similar task to our pre-training objective. Our IC evaluation datasets include Snips (Coucke et al., 2018), ATIS (Price, 1990), and the low-resource reminder and weather domains provided in TOPv2 (Chen et al., 2020). Snips' intents are each from different domains. ATIS focuses on the flight domain; some intents appear only once or not at all in the training set, and some utterances are tagged with multiple intents (we count these as separate intents). For multi-class examples, we separate intents with the character # (e.g., book flight # airfare). TOPv2 ( reminder ) focuses on the creation and modification of personal reminders, while TOPv2 ( weather ) focuses on queries regarding the weather and time of sun-rise/sunset. For TC, we evaluate on Yahoo! Answers (YA) 8 and AG News. 9 YA consists of conversational user questions labeled with the general category of the 8 https://www.kaggle.com/soumikrakshit/ yahoo-answers-dataset 9 http://groups.di.unipi.it/  gulli/AG corpus of news articles.html question (e.g., Health, Sports). AG News consists of formal-register news articles labeled with their category (e.g., Business, World). When fine-tuning our T5-based models, the input sequence consists of an intent classification: pre-fix followed by the utterance. The output sequence is the intent in a natural language format. For example: intent classification: Find me a flight from NYC to Baltimore.  Book flight. 10 We create few-shot splits of size k for each evaluation set, where we sample  k examples for each class i from the set of labeled training examples T . More specifically,  k  { 1 , 2 , 4 , 8 , 16 , 32 } ,  i  T , if | T i |  k , we randomly sample k training examples from T i without replacement ; if | T i | < k , we simply use T i . We ensure that smaller splits are subsets of larger splits such that we may perform more principled comparisons across split sizes. In low-resource settings, the random seed can have a large impact on performance. We therefore average all accuracies over 5 random fine-tuning seeds. See Appendix G for hyperparameters. XLNet (Yang et al., 2019) is an autoregressive Transformer-based (Vaswani et al., 2017) language model which sees various word order permutations of the inputs during pre-training. We append a linear layer to the mean-pooled output of XLNet to obtain an XLNet-based classifier. This model has achieved SOTA performance on many text classification datasets, including AG News. SEQ2SEQ-PTR (Rongali et al., 2020) is based on sequence-to-sequence (Sutskever et al., 2014) and pointer generator networks (Vinyals et al., 2015; See et al., 2017); it achieves SOTA IC/SL performance on Snips, ATIS, and TOPv2. It transduces from an unlabeled utterance to a sequence with labeled spans (including the intent label, where the span is the entire sentence). LM-BFF (Gao et al., 2021) is a prompt-tuning-based model for few-shot text classification. It uses automatically generated and optimized prompts (from T5) to perform generative classification (with RoBERTa) by predicting a masked token after the 10 We also try label denoising fine-tuning to reduce the mismatch between pre-training and fine-tuning, but we find that this is not as effective as the traditional T5 fine-tuning setup. See Appendix F for scores. It achieves SOTA scores on few-shot bi-nary/trinary sentiment classification and NLI tasks. We employ the best-performing prompt-tuning with demonstrations approach, adapting it for IC by reducing the intent labels to single words. T5 is the closest baseline to LSAP: our model differs only in the addition of the label aware pretraining step. We thus compare LSAP against T5 to understand the contribution of our label aware pretraining approach to downstream performance. T5 is comparable to TANL (Paolini et al., 2021), which is based on vanilla T5; TANL's SOTA performance on a variety of structured prediction tasks indicates that T5 as-is can be a strong baseline. To test whether performance improvements with LSAP can be attributed to label semantics or domain adaptation on the utterances, we also present results for T5 adapted (with random span denoising) on only on the utterances and not the intents in our pre-training data. This is equivalent to LSAP with random span denoising, but without the intent labels. We refer to this as T5 (adapted). the best-performing pre-training format in few-shot settings across all evaluation sets . Differences in performance between LSAP formats decrease as we increase the fine-tuning set size; in full-resource settings, the difference between span denoising and label denoising is not significant. This suggests that explicitly demarcating intents from utterances during pre-training may help T5 better leverage the pre-training examples. 11 As label denoising performs best, we focus on that variant of LSAP from here on, though note that all 11 These results are stable across random samples of the few-shot splits. Table 4 displays IC accuracies across models. Compared to T5, the T5 (adapted) baseline achieves consistently higher IC accuracies. However, T5 (adapted) is vastly outperformed by LSAP (by up to 18% on Snips and 3% on ATIS in 1-shot settings), even though the utterances are the same across these settings; this indicates that much of the improvements may be attributed to label semantics , and that domain adaptation is only responsible for a small portion of the improvement. The XLNet classifier does not have access to label semantics during fine-tuning, instead observing class indices and utterances only; all other approaches hence have access to more information during fine-tuning and prediction. LM-BFF and SEQ2SEQ-PTR in very low-resource settings, though LM-BFF and SEQ2SEQ-PTR perform better than T5. Note that for ATIS, LM-BFF does not effectively handle the multi-class examples due to the single-word cloze format, nor the high number of similar labels due to the intended contrastive word use case; we thus do not compare to LM-BFF here. As we increase the size of the fine-tuning splits, performance increases and converges across methods. Thus, the primary contribution of LSAP is inducing quicker generalization across domains. Finally, we present topic classification accuracies for YA and AG News (Table 5). Our LSAP approach yields up to 35% improvements on AG News over the SOTA XLNet model in 1-shot settings, and over 100% on YA; we also maintain comparable performance to the SOTA XLNet baseline in full-resource settings. LSAP also improves up to 21% on AG News and 18% on YA over vanilla T5; this is evidence that our pre-training procedure is not just tuning T5 to recognize utterance-intent associations: it is also teaching T5 how to be a better text classifier in general. This also indicates that our procedure generalizes to labels that have not been seen during pre-training: topic labels are not present in the pre-training data, though some of these labels do appear as substrings of more specific intent labels (e.g., while Health does not appear as a label in our pre-training data, Get health information does appear). Using LSAP with label denoising, we ablate over each pre-training dataset and observe the effect on downstream IC performance (Table 6). The gold data improves few-shot IC performance over T5 by over 10%. Adding the silver data does not significantly change performance. Adding CSTwitter improves performance in few-shot settings, though it also increases variance across random seeds. The best accuracies and lowest variances are achieved after adding the Reddit data. Notably, we find that pre-training on only automatically filtered and labeled examples still improves performance over T5. To test whether this is due to CSTwitter and Reddit simply being well-suited to our evaluation sets, we try automatically filtering and labeling a less conversational dataset: Colossal Cleaned Common Crawl (C4; Raffel et al., 2020), the same dataset used for pre-training T5. We use the same number of training examples as in our combined CSTwitter+Reddit data for comparability (1,126,309 examples). We still observe performance improvements when pre-training on this dataset, and the difference between pre-training on this versus CSTwitter+Reddit is only significant at 2 examples per label. This is evidence that our method for creating new pre-training examples is effective with different types of data, including non-conversational data . Note that adding C4 to the gold+silver+CSTwitter+Reddit data does not significantly improve performance; we therefore do not include it in our final pre-training dataset. Including label sequences in the pre-training data results in large performance increases. Is this due solely to intent sequences being helpful indepen-dent information, or is the model learning useful associations between the inputs and labels? To investigate this, we experiment with misleading and random label semantics. For misleading label semantics, we remap the labels in an evaluation dataset by defining a bijective function from each intent to a randomly selected different intent, ensuring that identity mappings do not occur. In other words, if label i is replaced with label j , all instances of i are systematically converted to j in the training and test sets. We then fine-tune our models on these remapped-intent datasets and observe whether performance decreases, and by how much. Significant performance decreases indicate reliance on input-label associations. We also define a random label semantics variant of LSAP, where we randomly shuffle the intents with respect to the utterances in the pre-training data and then pre-train with label denoising on the shuffled data. Unlike the evaluation label remapping, this shuffling procedure preserves the number of instances of each intent such that our pre-training set is technically the samethe intents and utterances are simply mismatched. Performance with misleading and/or random label semantics (Table 7) 12 decreases considerably across datasets, and this is especially apparent in 12 Misleading label semantics results for TOPv2 are in Appendix D. the low-resource case. Label denoising seems the most sensitive to utterance-label associations. Performance drops decrease with increasing few-shot split sizes. This suggests that our models do rely on utterance-intent associations to achieve high IC performance, and that these associations are more important in lower-resource settings than higher-resource settings. We have proposed a pre-training approach for leveraging the semantic information inherent in labels. Our method improves few-shot text classification performance across domains while maintaining high performance in full-resource settings. This approach is fairly general and could potentially be extended to more structured semantic parsing tasks by annotating some of the pre-training examples, or perhaps by simply including diverse labeled examples from a variety of tasks in the pre-training step. Future work could investigate extending this method to pre-training from scratch, as well as tuning the utterance and label formats. One could also use demonstrations to achieve better performance with fewer gradient updates. We thank Giovanni Paolini, Ben Athiwaratkun, and the reviewers for their thoughtful feedback on earlier versions of this work.
abstain abstain abstain objective method objective method abstain abstain abstain abstain abstain abstain abstain abstain objective result objective objective objective other other other other other other objective other other other other method abstain abstain method abstain other other other other other other method other other other other other other abstain other abstain other other method method method method method method method abstain method method method abstain abstain abstain method abstain method method method abstain abstain other abstain abstain method method method abstain method method method method method abstain method method method abstain method method method method method method abstain method abstain abstain method method method method abstain method method method method method method method method method method method method method method abstain method method abstain method other abstain method abstain abstain method method method abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain method method method abstain method abstain method abstain result abstain result abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain objective result abstain abstain abstain other