Recent progress of abstractive text summarization largely relies on large pre-trained sequence-to-sequence Transformer models, which are computationally expensive. This paper aims to distill these large models into smaller ones for faster inference and with minimal performance loss. Pseudo-labeling based methods are popular in sequence-to-sequence model distillation. In this paper, we find simply manipulating attention temperatures in Transformers can make pseudo labels easier to learn for student models. Our experiments on three summarization datasets show our proposed method consistently improves vanilla pseudo-labeling based methods. Further empirical analysis shows that both pseudo labels and summaries produced by our students are shorter and more abstractive. Our code is available at https://github. com/Shengqiang-Zhang/plate . Automatic document summarization is the task of rewriting a long document into its shorter form while still retaining its most important content. In the literature, there are mainly two kinds of methods for summarization: extractive summarization and abstractive summarization (Nenkova and McK-eown, 2011). In this work, we focus on abstractive summarization, which is viewed as a sequence-to-sequence (Seq2Seq) learning problem, since recent abstractive models outperform their extractive counterparts and can produce more concise summaries (Raffel et al., 2020; Lewis et al., 2020; Zhang et al., 2020; Liu and Lapata, 2019). Recent progress of abstractive summarization largely relies on large pre-trained Transformer models (Raffel et al., 2020; Lewis et al., 2020; Zhang et al., 2020; Liu and Lapata, 2019; Bao et al., 2020). With these  Equal contribution. extremely large models, we can obtain state-of-the-art summarization results, but they are slow for online inference, which makes them difficult to be used in the production environment even with cutting-edge hardware. This paper aims to distill these large Transformer summarization models into smaller ones with minimal loss in performance. Knowledge distillation is a class of methods that leverage the output of a (large) teacher model to guide the training of a (small) student model. In classification tasks, it is typically done by minimizing the distance between the teacher and student predictions (Hinton et al., 2015). As to Seq2Seq models, an effective distillation method is called pseudo-labeling (Kim and Rush, 2016), where the teacher model generates pseudo summaries for all documents in the training set and the resulting document pseudo -summary pairs are used to train the student model. In this paper, we argue that attention distributions of a Seq2Seq teacher model might be too sharp. As a result, pseudo labels generated from it are sub-optimal for student models. In the summarization task, we observe that 1) pseudo summaries generated from our teacher model copy more continuous text spans from original documents than reference summaries (56% 4-grams in pseudo summaries and 15% 4-grams in reference summaries are copied from their original documents on CNN/DailyMail dataset); 2) pseudo summaries tend to summarize the leading part of a document (measured on CNN/DailyMail, 74% of sentences in pseudo summaries and 64% of sentences in reference summaries are from the leading 40% sentences in original documents). We obtain the two numbers above by matching each sentence in a summary with the sentence in its original document that can produce maximum ROUGE (Lin, 2004) score between them. We call the two biases above the copy bias and the leading bias . In order to have an intuitive feeling, we select a rep-127 resentative example 1 and visualize its cross attention weights 2 (see the left graph in Figure 1). We observe that attention weights form three lines, which indicates very time the decoder predicts the next word, its attention points to the next word in the input document. That may be the reason why multiple continuous spans of text are copied. Another phenomenon we observe is that all high-value attention weights (in deeper color) concentrate on the first 200 words in the input document, which reflects the leading bias. In either case, the attention distribution is too sharp (i.e., attention weights of the next word position or the leading part is much larger than other positions), which means our teacher model is over-confident. Based on the observations above, we propose a simple method called PLATE (as shorthand for P seudo-labeling with L arger A ttention TE mperature) to smooth attention distributions of teacher models. Specifically, we re-scale attention weights in all attention modules with a higher temperature, which leads to softer attention distributions. Figure 1 intuitively shows the effect of using higher attention temperatures. Compared with the left graph, the right graph with higher attention temperature has shorter lines (less copy bias) with high attention weights, and positions of high attention weights extend to the first 450 words (less leading bias). Less copy bias in pseudo summaries encourages student models to be more abstractive, while less leading bias in pseudo summaries encourages student models to take advantage of longer context in documents. Experiments on CNN/DailyMail, XSum, and New York Times datasets with student models of different sizes show PLATE consistently outperforms vanilla pseudo-labeling methods. Further empirical analysis shows that, with PLATE , both pseudo summaries generated by teacher models and summaries generated by student models are shorter and more abstractive, which matches the goal of abstractive summarization. Large pre-trained Seq2Seq Transformer models largely improve results of generation tasks including text summarization (Song et al., 2019; Lewis et al., 2020; Bao et al., 2020; Raffel et al., 2020; 1 See the detailed example in Appendix E. 2 We use cross attention because we can see how words Zhang et al., 2020). These models are pre-trained using unsupervised text-to-text objectives. For example, T5 (Raffel et al., 2020) is pre-trained by predicting corrupted text spans. BART (Lewis et al., 2020) employs denoising auto-encoding objectives such as text infilling and sentence permutation during its pre-training. The pre-training objective of PEGASUS (Zhang et al., 2020) is tailored for the summarization task, which predicts the most summary worthy sentences in a document. Our method aims to make these large models faster. In knowledge distillation, besides learning from gold labels in the training set, student models can learn from soft targets (Ba and Caruana, 2014; Hinton et al., 2015), intermediate hidden states (Romero et al., 2014), attentions (Zagoruyko and Komodakis, 2017; Wang et al., 2020), and target output derivatives (Czarnecki et al., 2017) of teacher models. Recent work for distillation of pre-trained Transformers (e.g., DistilBERT (Sanh et al., 2019), TinyBERT (Jiao et al., 2020), Mobile-BERT (Sun et al., 2020), BERT-of-Theseus (Xu et al., 2020a), MINILM (Wang et al., 2020)) focuses on natural language understanding tasks such as GLUE (Wang et al., 2018) or SQuAD (Rajpurkar et al., 2016) benchmarks. Most methods above are designed for classification models. In Seq2Seq learning tasks such as summarization, we can apply distillation methods above to each step of sequence model predictions. However, the sequence-level knowledge of teacher mod-128 els is not well utilized. Therefore, Kim and Rush (2016) introduce a sequence-level knowledge distillation method (i.e., pseudo-labeling ), where a student model is trained with pseudo labels generated by the teacher model using beam search decoding. Kim and Rush (2016) and later work (Kasai et al., 2020; Gu et al., 2017; Denkowski and Neubig, 2017) show pseudo-labeling achieves competitive performance for Seq2Seq tasks such as machine translation. Shleifer and Rush (2020) propose the shrink and fine-tune (SFT) approach for pre-trained summarization distillation, which re-finetunes a teacher model with some layers removed, and they show SFT outperforms pseudo-labeling and a mod-ification of direct knowledge distillation (Jiao et al., 2020) on one of their datasets, but not others. Our method, which builds on top of pseudo-labeling , is conceptually simple and improves pseudo-labeling across different summarization datasets. There is an interesting line of work called self-distillation or self-training (Furlanello et al., 2018; Xie et al., 2020; Deng et al., 2009; Liu et al., 2020; He et al., 2019), where the size of the student model is identical to the size of the teacher model. Our method can also be applied in self-distillation and can potentially be combined with the self-distillation methods above. 3 Summarization Distillation 3.1 Transformer based abstractive summarization Abstractive summarization aims to rewrite a document into its shorter form (i.e., summary), which is a typical Seq2Seq learning problem. We adopt the Seq2Seq Transformer (Vaswani et al., 2017) model. Given a document X = ( x 1 , x 2 , . . . , x | X | ) and its gold summary Y = ( y 1 , y 2 , . . . , y | Y | ) , we estimate the following conditional probability: p ( Y | X ;  ) = | Y | (cid:89) t =1 p ( y t | y <t , X ;  ) (1) where  is the model parameter and y <t stands for all tokens before position t (i.e., ( y 1 , y 2 , . . . , y t  1 ) ). The Seq2Seq Transformer model can be trained by minimizing the negative log-likelihood of gold document-summary pairs: LG (  ) =  1 | Y | log p ( Y | X ;  ) (2) where | Y | is the number of tokens in summary Y . Knowledge distillation refers to the task of transferring knowledge of a large teacher model (or a group of large teacher models) into a small student model. As to Seq2Seq learning tasks such as machine translation and summarization, pseudo-labeling based methods are usually used to imitate teacher predictions at the sequence level. Specifically, suppose we have a document X , and  Y = ( y 1 ,  y 2 , . . . ,  y |  Y | ) is a pseudo summary generated by a teacher model using beam search. The student can be trained by minimizing the negative log-likelihood of document-topseudo -summary pairs. Strictly, all possible pseudo summaries from X should be taken into account. Unfortunately, the computational cost is prohibitive. We therefore use a single sample  Y (which takes a large portion of probability mass from the teacher) instead as in Kim and Rush (2016). where Q , K , V are linear projections of hidden states of a layer and  is the temperature of the attention module which is usually  d ( d is the hidden dimension size of that attention head). Our distillation method PLATE works as follows. Assume we have a teacher model trained with  =  d . When the teacher generates pseudo labels with beam search, we use a higher attention temperature and set  =   d where  > 1 (  is the attention temperature coefficient). Note that we only change the teacher's attention temperature during inference time. When we train our student model with pseudo labels, we still use a normal temperature (i.e.,  =  d ). We find that adjusting the student's attention temperature does not work. Probably because the student can easily adapt to the scaled attention temperature during training. generate pseudo labels with more diversity, we further propose to use a random  for each input document (   U [ a, b ] ). Note that U [ a, b ] is a uniform distribution and we typically set a = 1 . 0 and b = 2 . 0 . We conduct our experiments on three popular document summarization datasets: CNN/DailyMail (Hermann et al., 2015), XSum (Narayan et al., 2018), and New York Times (Sandhaus, 2008). All datasets are tokenized with the GPT-2 tokenizer (Radford et al., 2019), which is based on UTF-8 BPE (Sennrich et al., 2016). CNNDM The CNN/DailyMail dataset (CNNDM; Hermann et al., 2015) contains online news articles from the CNN and DailyMail websites paired with their associated highlights as reference summaries. We follow the standard pre-processing steps described in See et al. (2017); Liu and Lapata (2019). 3 The resulting numbers of document-summary pairs for training, validation, and test are 287,227, 13,368, and 11,490, respectively. XSum The XSum dataset is collected by harvesting online articles from the BBC with single sentence summaries, which is professionally written. The summaries are extremely abstractive. We use the official splits of Narayan et al. (2018). There are 204,045 articles for training; 11,332 articles for validation; and 11,334 articles for test. NYT The New York Times dataset (NYT; Sand-haus, 2008) is composed of articles published by New York Times, and the summaries are written by library scientists. After applying the pre-processing procedures described in Durrett et al. (2016); Liu and Lapata (2019), we first obtain 110,540 articles with abstractive summaries. The test set is constructed by including the 9,076 articles published after January 1, 2007. The remaining 100,834 articles are further split into training and validation sets. After removing articles with summaries less than 50 words, we obtain the final dataset with 38,264 articles for training; 4,002 articles for validation; and 3,421 articles for test. 3 Scripts are available at https://github.com/ abisee/cnn-dailymail . Teacher/Student model settings We use BART Large (Lewis et al., 2020) as our teacher model, which has 12 layers in the encoder and decoder. The hidden size of each layer is 1024, and each layer contains 16 attention heads with a hidden size of 64. We have four kinds of student models. The first three student models are initialized from BART weights (therefore, their hidden sizes are the same as that of BART). All the three students have the 12 layers of BART encoder and differ in the number of decoder layers. They are denoted by BART 12-6 , BART 12-3 , and BART 12-12 with 6, 3, and 12 decoder layers, respectively. For BART 12-6 (or BART 12-3 ), the decoder is initialized from the first 6 (or 3) layers or the maximally spaced 6 (or 3) layers of BART decoder. The fourth student is the Transformer base model (Vaswani et al., 2017), which has 6 layers in each of the encoder and decoder. Each layer has a hidden size of 512 and 8 attention heads. This student is randomly initialized and denoted by Transformer . The latency statistics (Milliseconds) and numbers of parameters of above four models are in Table 1. Training and inference Hyper-parameters for BART , BART 12-6 , BART 12-3 , and BART 12-12 are similar. Specifically, all models are optimized using Adam (Kingma and Ba, 2014) with  1 = 0 . 9 ,  2 = 0 . 999 . Learning rates are tuned on validation sets (choose from 1e-5, 3e-5, 5e-5, 7e-5). We truncate all documents and summaries to 1024 sub-word tokens. We use a batch size of around 80 documents (we limit the max number of tokens on each GPU to 2048) and train our models for 20,000/15,000/6,000 steps with 500 warmup steps for CNNDM, XSum, and NYT, respectively. We also employ a weight decay of 0.01. For Transformer , the hyper-parameters of the Adam optimizer is a bit different, and we use  1 = 0 . 9 ,  2 = 0 . 98 . Learning rates are picked from 1e-4, 3e-4, 5e-4, 7e-4 accord-130 ing to validation sets. The weight decay is set to 0.0001. The warmup step we use is 4000. We train Transformer for 100 epochs and select the best model w.r.t. their ROUGE scores on validation sets. For all models above we apply a label smoothing of 0.1 to prevent overfitting (Pereyra et al., 2017). During inference, as common wisdom, we apply beam search. The beam size, length penalty, and minimal length are 4, 2.0, and 55 on CNNDM; 6, 0.1, and 1 on XSum; and 4, 0.7, and 80 on NYT, respectively. All our models are trained on 8 NVIDIA V100 GPUs. The training is fairly fast. Training on CNNDM with the teacher model (i.e., BART ) is most time-consuming. It takes about 45 minutes for one epoch, and we need 6 epochs in total. We evaluate the quality of different summarization systems using ROUGE. On CNNDM and XSum datasets, we report full-length F1 based ROUGE-1 (R1), ROUGE-2 (R2), and ROUGE-L (RL) scores. Following Durrett et al. (2016); Liu and Lapata (2019), we report limited-length recall based ROUGE-1, ROUGE-2, and ROUGE-L, where generated summaries are truncated to the lengths of gold summaries. All ROUGE scores are computed using the ROUGE-1.5.5.pl script 4 . Summaries generated by abstractive models may be ungrammatical or unfaithful to the original document. Additionally, we also measure the quality of generated summaries by eliciting human judgements. We randomly sample 50 documents from the test set of CNNDM. 12 annotators are invited (they are either native English speakers or graduate students with IELTS test score over 6.5). In the evaluation, participants are presented with a document and a list of outputs by different models. First, they are asked to evaluate the summaries on three dimensions: fluency (is the summary grammatically correct?), faithfulness (is the summary faithful to the original document?), and coverage (does the summary coverage important information of the document?). Then, they are asked to rank the summaries from best to worst as a way of determining the overall quality of summaries. Each document is ensured to be annotated by 3 different subjects. Our main results are shown in Table 2. The first block includes several recent abstractive summarization models based on large pre-trained Transformers. BERTSUM (Liu and Lapata, 2019) employs BERT (Devlin et al., 2019) as its encoder and uses randomly initialized decoder. T5 (Raffel et al., 2020), PEGASUS (Zhang et al., 2020) and BART (Lewis et al., 2020) are three popular large Seq2Seq Transformer models with different pretraining objectives. Our own fine-tuning version of BART (BART (ours)) is comparable or slightly better than the original reported BART results, and we use it as the teacher model on the three datasets. The second block presents results of student models. Shleifer and Rush (2020) compare pseudo-labeling (BART-PL), knowledge distillation using both output and intermediate layers (BART-KD) as well as shrink and fine-tuning (BART-SFT) methods. They also use BART as teacher models. Note their settings of student models are BART 12-6 on CNNDM and BART 12-3 on XSum. Results of our BART 12-3 and BART 12-6 student models are in the third and fourth block. We present results of students trained with gold labels (Gold) and regular pseudo labels (Regular) as well as pseudo labels with higher and random attention temperatures (PLATE B12-3  =1 . 5 , PLATE B12-3  =2 . 0 and PLATE B12-3rnd ). PLATE B12-3  =1 . 5 means that the student uses attention temperature coefficient  = 1 . 5 with architecture setting BART 12-3 . PLATE B12-3 rnd means that we use random attention temperature of   U [1 . 0 , 2 . 0] . We observe that using pseudo-labeling methods with higher attention temperatures consistently improves over its counterpart with normal attention temperatures (Regular) across all three datasets, and the differences between them are almost always significant measured with the ROUGE script 5 (see details in Table 2). Interestingly, our student models PLATE B12-3  =2 . 0 and PLATE B12-6  =2 . 0 outperform all models in comparison (including student models and even the teacher model) on CNNDM. Our best performing student model PLATE B12-3  =1 . 5 outperforms BART-PL, BART-SFT, and BART-KD on XSum. Meanwhile, our method is conceptually simpler and can further be combined with their methods with additional train-4 with -c 95 -r 1000 -n 2 -a -m arguments. In Section 3.3, we also propose a variant of our method, which employs random attention temperatures (PLATE rnd in Table 2). We can see that though random temperature based method is not as good as our best fixed-temperature method, it in general produces decent results. Therefore, we recommend using this method when the computing budget is limited. Note that we also tried more extreme  values as shown in Appendix B, and we find the value of 1.5 or 2.0 works better than others. In the fifth block, we additionally conduct self-distillation experiments, which is not the focus of this work. Our method improves the teacher model on CNNDM; ROUGE-2/L scores are improved on XSum; while on NYT, there are improvements on ROUGE-1/L. Results with the Transformer student (the sixth block) follow a similar trend, although the improvements are smaller. It may because the model-Ref Regular PLATE B12-6  =1 . ing power of Transformer without pre-training is not large enough to effectively model the differences in pseudo labels. It is also interesting to see that students distilled with pseudo-labeling do improve gold label based students using randomly initialized Transformer , but not with pre-trained models (i.e., BART 12-6 and BART 12-3 ), which may also be due to the strong modeling power of large pre-trained Transformers. We randomly sample 50 documents from the test set of CNNDM. We compare our best student model PLATE B12-6  =2 . 0 against the 132 Attention Setting R1 R2 RL  enc =  cross =  dec = 2 . regular pseudo-labeling model (Regular), another model PLATE B12-6  =1 . 5 and human reference (Ref). We ask human judges to rank the outputs of these models from best to worst. We convert the ranks to rank ratings (rank i to 5  i ) and further conduct student t -test on these ratings. As shown in Table 3, PLATE B12-6  =2 . 0 obtains the best ranking score and the difference between PLATE B12-6  =2 . 0 and the regular pseudo-labeling based method Regular is significant ( p < 0 . 05 ), which indicates our proposed method PLATE indeed produces better summaries. Ablation study In a Transformer, there are three types of attention modules (i.e., encoder self-attention, decoder self-attention and decoder cross-attention), and we can scale attention temperatures for all of them or some of them. Let  enc ,  cross , and  dec denote the attention temperature coefficient of the encoder self-attention module, the decoder cross-attention module, and the decoder self-attention module, respectively. As shown in Table 4, using large attention temperature coeffi-cients (2.0) for all three types of attention modules leads to the best result. When setting the coefficient of the cross attention module to  cross = 1 . 0 , the ROUGE scores drop most. Perhaps this is not surprising, since cross attentions are directly related to the selection of document contents for summarization. Besides, the attention temperature of the decoder self-attention is also crucial but not as important as the cross-attention (see the fourth row). Comparison with sampling and tuning output layer temperature Sampling based methods can produce more diverse and richer outputs than its beam search based counterpart and has been proven useful in back translation (Edunov et al., 2018). We implement the sampling method in Edunov et al. (2018) and Nucleus Sampling (Holtzman et al., 2019), a more advanced sampling method, to generate pseudo labels for distillation. We use the BART 12-6 as the student model, and the distillation results on CNNDM are in Table 5. As can be seen, both of the sampling based methods above perform worse than the regular beam search based pseudo-labeling method (Regular), let alone ours. Besides the attention temperatures, we can also tune the temperature T in the decoder output softmax layer. With a proper T (i.e., T = 0 . 5 ) during pseudo label generation, the resulting student model slightly outperforms the baseline student model with regular pseudo labeling method on ROUGE-2/L (see Table 5), but worse than PLATE  =2 . 0 . More results with different T s are in Appendix C. 4.5 Analysis Why does our distillation method work? To answer this question, we first try to analyze the reasons from both the external characteristics of the summaries generated by the teacher model and the internal characteristics of the teacher's attention mechanism. Then, we will give an in-depth explanation. Length and novel n -grams We first analyze the pseudo summaries generated by the teacher models. We calculate novel n -grams and lengths of generated summaries. Note that if an n -gram appears in the summary, but not in the original document, we call it a novel n -gram. Proportions of novel n -grams are used to measure the abstractiveness of summaries (See et al., 2017; Liu and Lapata, 2019). As shown in Table 6, when using a larger  , pseudo summaries are shorter 6 and contain a larger portion of novel n -grams. It indicates that the teachers can produce more concise and abstractive summaries, which matches the goal of abstractive summarization. Are these pseudo summaries of good quality? The performance of the teacher with different attention temperatures on CNNDM test 6 We also try changing the length penalty during teach-ers' inference to make pseudo summaries shorter, but we find this method does not help summarization distillation (see Appendix D for more details). set is shown in Table 7. Their results are all decent and close to each other (at least for ROUGE-1 and ROUGE-L). Interestingly, compared with  = 1 . 0 , the performance of the teacher with  = 2 . 0 is worse, but the resulting student is much better (see Table 2). Perhaps not surprisingly, the styles of summaries from students are similar with these from their teachers. Concise and abstractive teachers lead to concise and abstractive students (see Table 6). Conciseness and abstractiveness are good properties for summarization, which however may not be the case for other generation tasks such as machine translation. We apply PLATE to the WMT16 (Bojar et al., 2016) English-German translation task and use Transformer-big as the teacher and Transformer-base as the student. With  = 1 . 5 , we obtain a BLEU of 27.90, while the result of the regular pseudo-labeling is 27.79 (more details are in Appendix A). Attention We have shown earlier in Figure 1 that with higher attention temperature, cross-attention modules of a teacher can attend to later parts in documents. We observe that students behave similarly, and we put more cross attention visualization of students in Appendix F. To obtain corpus-level >@ >@ >@ >@ >@ 1RUPDOL]HGWRNHQSRVLWLRQWRLQWHUYDOVLQGRFXPHQWV         3 U RSR U WL RQ  R I  H Y L G H Q W  D WW H Q WL RQ  Z H L JK W V   statistics, we further calculate the evident cross-attention weight distributions of the teacher when generating pseudo labels on the training set of CNNDM. Note that an attention weight is evident if it is greater than 0.15, and these evident attention weights account for around 15% of all attention weights. Specifically, we normalize the token positions of each document to (0 . 0 , 1 . 0] and divide the normalized positions into five bins. The mean proportions of evident attentions for all bins are shown in Figure 2. Compared to the teacher with normal attention temperature (pink bar), teachers with higher attention temperatures (blue and green bars) attend less on the heading parts of documents while more on the tail parts of documents. pseudo summaries, which makes the teacher provide more summary-like pseudo labels to students. High-temperature teachers can alleviate the leading bias problems by providing pseudo labels with better coverage of source documents to students. More explanation According to the study of Xu et al. (2020b), the prediction entropy correlates strongly with whether the model is copying or generating, as well as where in the sentence the token is (content selection). The decoder tends to copy when the model has a low prediction entropy and generate novel bigrams when the model has a high prediction entropy. They also find that high entropy of attention distribution strongly correlates with the model's high prediction entropy. Our method with a higher attention temperature makes attention distributions of the teacher model smoother and leads to a higher entropy of attention distributions, which results in a higher prediction entropy. Therefore, the model with higher attention temperature tends to copy less and generate more novel tokens. The conclusion from Xu et al. (2020b) is in accordance with our observation in Table 6. In this work, we propose a simple but effective extension of pseudo-labeling method PLATE for summarization distillation. Experiments on three datasets demonstrate that our method can consistently outperform the vanilla pseudo-labeling method. Further empirical analysis shows that by using our method, teacher models can generate more concise and abstractive summaries. As a result, summaries produced by student models also become more concise and abstractive. In the future, we would like to explore our method to other generation tasks as well as self-training with unlabeled data. We are also interested in combining our method with other distillation methods and extending our method for better teacher model training.
abstain objective abstain result objective result other abstain abstain abstain result abstain abstain result objective abstain abstain abstain abstain abstain other result abstain result result abstain objective abstain objective abstain abstain abstain abstain abstain abstain other abstain other other other other other objective other other other method other other other other abstain other method other method method other abstain abstain method abstain abstain abstain method abstain method method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective result abstain objective objective