Transformer-based models have achieved state-of-the-art performance on short-input summarization. However, they still struggle with summarizing longer text. In this paper, we present DYLE , a novel dynamic latent extraction approach for abstractive long-input summarization. DYLE jointly trains an extractor and a generator and treats the extracted text snippets as the latent variable, allowing dynamic snippet-level attention weights during decoding. To provide adequate supervision, we propose simple yet effective heuristics for oracle extraction as well as a consistency loss term, which encourages the extractor to approximate the averaged dynamic weights predicted by the generator. We evaluate our method on different long-document and long-dialogue summarization tasks: GovReport, QMSum, and arXiv. Experiment results show that DYLE outperforms all existing methods on GovReport and QMSum, with gains up to 6.1 ROUGE, while yielding strong results on arXiv. Further analysis shows that the proposed dynamic weights provide interpretability of our generation process. 1 1 Introduction Transformer-based (Vaswani et al., 2017) pretrained language models (PLMs) such as BART (Lewis et al., 2020a) and T5 (Raffel et al., 2020), have achieved state-of-the-art performance on short text summarization. However, due to the high memory complexity of the full self-attention (Tay et al., 2020a), PLMs still struggle to handle long inputs (Rohde et al., 2021). Model efficiency and summary quality present a pair of challenges (Huang et al., 2021): models need to capture information scattered across the long input while maintaining a low computational cost. Prior models tackled long input summarization mostly in four ways. First, sparse attention (Child et al., 2019; Beltagy et al., 2020; Tay et al., 2020b) is used to reduce the memory complexity of the Transformers so that they can attend to more tokens. Second, extract-then-generate methods extract salient texts from the input and then summarize the extracted texts. Extractors are either independently trained with full supervision (Zhong et al., 2021b) or optimized using reinforcement learning (Williams, 1992; Chen and Bansal, 2018; Bae et al., 2019; Brainskas et al., 2021). Third, models are proposed to divide source text into sections (Gidiotis and Tsoumakas, 2020; Wu et al., 2021; Liu et al., 2021) which are individually summarized and combined to form a full summary. Fourth, hierarchical models (Rohde et al., 2021; Zhu et al., 2020) improve summarization by capturing sentence or discourse level dependencies. We elaborate on these four directions and their limitations in Section 2. We believe that the extract-then-generate approach mimics how a person would handle long-input summarization: first identify important pieces of information in the text and then summarize them (Kiyoumarsi, 2015; Sun et al., 2020). The extract-then-generate framework is based on the assumption that salient information useful for summarization only occupies a small portion of the input, 1687 which is a sensible assumption given the long input length. This approach shortens the source input to a pre-set length, which addresses the main challenge of the model not being able to handle longer input beyond a certain limit. However, previous separately-trained extract-then-generate approaches are limited as they suffer from cascaded errors from the extractor to the generator. Though various reinforcement learning techniques are introduced to bridge the two steps, they have noticeable drawbacks (discussed in Section 3.3), and we argue that the long input makes this approach suboptimal. In this paper, we propose a new approach for long-input summarization: Dynamic Latent Extraction for Abstractive Summarization (DYLE ). DYLE jointly trains the extractor and the generator and keeps the extracted text snippets latent. For an output token, DYLE compute its probability conditioned on each input snippet separately , and its generation probability is computed by marginal-izing over all the input snippets under a learned dynamic weights assigned by the generator conditioned on the previously generated tokens. We optimize the extractor with two surrogate losses. First, we compute the extractive oracle based on the reference summary with a greedy search over the best ROUGE scores. These oracle snippets are used as targets for the extractor learning signal. Moreover, we propose consistency loss to encourage the extractor to approximate its own predicted weights on the snippet to the averaged dynamic weights predicted by the generator. We conducted experiments on three long-input summarization datasets: GovReport (Huang et al., 2021) and arXiv (Cohan et al., 2018) for long-document summarization, and QMSum (Zhong et al., 2021b) for long-dialogue summarization. Our method DYLE largely outperforms existing methods on GovReport and QMSum, while achieving strong results on arXiv. Notably, DYLE yields gains of 4.2/6.1/4.0 of ROUGE-1/2/L points over the previous best method on GovReport. These experiments demonstrate the generalizability of DYLE to multiple long-input summarization tasks. We summarize our contributions as follows:  We introduce DYLE , a dynamic latent extraction approach for abstractive long-input summarization. DYLE better captures information in the long input and reduces computational cost;  We propose multiple auxiliary optimizations for the effective training of DYLE : 1) extractive oracle as a learning signal for the extractor; 2) consistency loss that bridges extraction and generation; 3) hybrid training methods that make the extraction more robust;  Experimental results show that DYLE largely outperforms the state-of-the-art on two long input summarization datasets. We also conducted a detailed analysis that shows dynamic weights improve model interpretability. Sparse attention mechanism The full attention mechanism has a quadratic memory cost. Prior research works have proposed different sparse attention mechanisms to reduce the memory cost. Longformer (Beltagy et al., 2020) uses a dilated sliding window of blocks and global attention patterns. BigBird (Zaheer et al., 2020) employs sliding windows and random blocks. Reformer (Kitaev et al., 2020) uses the locality-sensitive hashing. In addition to optimizing the encoder self-attention, Huang et al. (2021) proposes head-wise positional strides to reduce the cost of the encoder-decoder attention. However, sparse attention diminishes the benefits of pretraining and sacrifices parts of the receptive field. Extract-then-generate method This method extracts salient text snippets from the input, followed by generating an overall summary. Most of these approaches are trained separately (Zhang et al., 2019; Lebanoff et al., 2019; Xu and Durrett, 2019; Bajaj et al., 2021; Zhang et al., 2021b), which suffer from information loss as we pass the extracted snippets to the generator. Some approaches attempt to reduce that loss by bridging the two stages. Chen and Bansal (2018) adopts reinforcement learning (RL) with a sentence-level policy gradient. Bae et al. (2019) proposes summary-level policy gradient. Using RL suffers from various drawbacks on long input texts, which will be elaborated in Section 3.3. DYLE is different as we jointly train an extract-then-generate model for summarization using latent variables. conquer (Gidiotis and Tsoumakas, 2020; Grail et al., 2021; Zhang et al., 2021a). It breaks a long input into multiple parts, which are summarized separately and combined to produce a final summary. However, these models do not capture the contextual dependencies across parts and assume that the input has certain structure. Hierarchical models Various hierarchical models have been proposed to handle the longer inputs. Cohan et al. (2018) models the document discourse structure with a hierarchical encoder and a discourse-aware decoder. HAT-Bart (Rohde et al., 2021) proposes a new Hierarchical Attention Transformer-based architecture that attempts to capture sentence and paragraph-level information. HMNet (Zhu et al., 2020) builds a hierarchical structure that includes discourse-level information and speaker roles. However, these models focus mainly on model performance and not on reducing the memory and computational cost. An overview of our approach is shown in Figure 1. In Section 3.1, we formulate our task and the extractor-generator framework. In Section 3.2, we introduce our parameterization of the extractor for long inputs. In Section 3.3, we introduce generator formulation and the novel consistency loss. The extractor module is both optimized with the consistency loss and the oracle loss, which we elaborate on in Section 3.4. The overall training objective is summarized in Section 3.5. In the long-input summarization task, the input consists of L text snippets, X = ( x 1 , . . . , x L ) , and an optional query q if a query is paired with a summary. In long-input summarization, the number of text snippets, L , could be potentially large. The output is a summary y of length T . For the dialogue summarization task, dialogue utterances by each speaker are used as snippets. For documents, we tokenize the input into sentences and use each sentence as a snippet. The goal is to learn a model that generates a sequence of summary tokens y given the input snippets X and the previously generated tokens y <t : P  ( y | q, X ) = T (cid:89) t =1 P  ( y t | q, X, y <t ) RoBERTa RoBERTa query query Top-Extracted snippets Document Figure 2: Long-input extractor. The extractor takes the query and the source text as input and outputs a score s i = E  ( q, x i ) for each text snippet x i . Here  is the extractor parameters. We extract K snippets XK from the document X based on their scores: XK = topK ( E  ( q, x i ) , x i  X ) (1) After retrieving XK from X , the extractor-generator framework models the output probability by replacing X with XK , i.e., P  ( y | q, X ) = P  ( y | q, XK ) = T (cid:89) t =1 P  ( y t | q, XK , y <t ) (2) Note that the topK operation in Eq. (1) is nondifferentiable, and we do not propagate gradients through topK ; instead, we propose methods to optimize the extractor in Section 3.3 and Section 3.4. An interesting research question is how to design the extractor for long inputs. Limited by GPU memory, it is impractical to concatenate all snippets and encode them with a large pre-trained language model. As shown in Figure 2, we group consecutive snippets into chunks . We concatenate the query q with each chunk and compute the encoded vector for each snippet independently within the chunk it belongs to. We project the encoded vectors to scalar scores s i = E  ( q, x i ) using an MLP. The first challenge is that the extraction operation (topK in Eq. (1)) is non-differentiable. One approach is to adopt RL-based optimizations (Chen and Bansal, 2018; Bae et al., 2019), which has two drawbacks. First, reinforcement learning for large action spaces (i.e., extracting K out of L snippets when L is very large) has high variances. Second, current methods mostly use sentence-level ROUGE (Chen and Bansal, 2018) or summary-level ROUGE (Bae et al., 2019) as training rewards. Using sentence-level ROUGE could potentially select sentences with overlapping contents (Narayan et al., 2018), resulting in redundant final summaries. Using a summary-level ROUGE leads to the sparsity of the training signal, and longer input makes this approach harder to train. The second challenge is interpretability: one might want to know whether the generator is leveraging the extracted information at each decoding time step. To address these challenges, we propose a generator that dynamically assigns weights to every extracted snippet at each time step. Different from the extractor scores, which are independent of the decoding time step, the generator assigns different dynamic scores at different time steps. Dynamic weights make the decoding process interpretable and help denoise the extraction by down-weighting irrelevant snippets. It also provides training signals for the extractor using consistency loss . Generator formulation The overview of the generator is shown in Figure 3. For each extracted snippet x , the generator predicts the generation probability P  ( y t | q, x, y <t ) on this snippet and a dynamic weight P  ( x | q, XK , y <t ) for this snippet. The independent encoding of each extracted snippet saves memory because the snippets do not need to attend to each other. Without loss of generality, we assume that P  ( | q, x, y <t ) is computed by first mapping the input ( q, x, y <t ) to a contextualized representation vector h xt . For Transformers (Vaswani et al., 2017) and encoder-decoder with attention models (Bahdanau et al., 2015), h xt is usually the model's output before the final language model head. The generation probability P  ( y t | q, x, y <t ) is computed by feeding h xt into the language model head. For the dynamic weight P  ( x | q, XK , y <t ) , we adopt a separate MLP to map each h xt to a scalar logit l x , and P  ( | q, X, y <t ) is defined as softmax( { l x } x  X ) . We compute the generation probability by marginal-izing over all extracted snippets: P  ( y | q, XK ) = T (cid:89) t =1 (cid:88) x  XKP  ( y t | q, x, y <t ) P  ( x | q, XK , y <t ) (3) The dynamic weight P  ( x | q, XK , y <t ) at each decoding time step t allows us to interpret how the generator utilizes the extracted snippets. For example, a larger weight to a particular snippet indicates the larger importance of the snippet to the current decoding time step. The generation loss is defined as the NLL of the gold summary: L  gen =  log P  ( y | q, XK ) (4) where P  ( y | q, XK ) is defined in Eq. (2). Here we do not propagate gradients of L  gen to the extractor parameters since topK is non-differentiable. Instead, methods to optimize the extractor are described in Section 3.3 and Section 3.4. Consistency loss We also leverage the dynamic weights to provide a training signal for the extractor. Since the dynamic weight of a snippet can be interpreted as the importance of the snippet at a particular time step, we average the dynamic weights over all the decoding steps and view the averaged weight as the overall importance of the snippet. Based on this intuition, we propose what we term as consistency loss , which measures the distance between the averaged dynamic weights distribution and the extractor distribution. We want these two distributions to be close on an arbitrary subset of X . For simplicity, we take XK as the subset and define the consistency loss as L  consist = KL (cid:104) 1 TT (cid:88) t =1 P  ( | q, XK , y <t ) || softmax ( E  ( q, x i ) , x i  XK ) (cid:105) (5) 1690 Note that the consistency loss is superscripted with the extractor's parameters  , which means that we do not compute gradients for the generator's parameters  . Since we want the distributional distance to be small on an arbitrary subset of X , we do not propagate gradients through the topK operator. For long-input summarization, the extracted snippets XK used during training are important for stable optimization. Instead of using XK defined in Eq. (1), we propose to leverage extractive oracles during training. No extractive oracles are used during test time. Greedy search for extractive oracles Extractive oracles denote a set of selected text snippets whose concatenation maximizes the evaluation metric given the gold summary. We implement the extractive oracle using greedy search. Specifically, we start with an empty set, and we iteratively select a snippet from the input such that the concatenation of that snippet and the already selected snippets maximizes the average of ROUGE-1, ROUGE-2, and ROUGE-L scores given the gold summary. We denote the extractive oracles as X o . Hybrid training We leverage the extractive oracles to define XK used during training. If the number of oracles equals or exceeds K , we define XK as the first K oracle snippets. If the number of oracles is less than K , we define XK as the union of X o and the top snippets ranked by the extractor that is not appearing in X o . Such hybrid training has two benefits. First, compared with XK defined in Eq. (1), it provides higher-quality inputs to the generator. Second, it reduces the reliance on the oracle and improves the generalizability of our model beyond the training set, as other text snippets omitted in the greedy search might help the generation. Oracle loss The extractive oracles X o are used as a supervision signal for the extraction part of our model. The oracle loss L  oracle is computed from the cross-entropy loss between all chunks in the extractor selected set and the extractive oracle. Formally, the oracle loss is computed as L  oracle =  1 | X o | (cid:88) x  X o log e E  ( q,x ) (cid:80) x i  X e E  ( q,x i ) (6) Dataset Query Format Src. The overall training objective of our method is where  g ,  o , and  c are hyperparameters to bal-ance the loss components. Gradients are computed for the superscripted parameters. Specifically, the extractor is solely optimized with the consistency loss and the oracle loss, and the generator is solely optimized with the generation loss. QMSum (Zhong et al., 2021b) is a benchmark for query-based multi-domain meeting summarization. It consists of meetings from three domains: AMI (Carletta et al., 2005), ICSI (Janin et al., 2003), and committee meetings of the Welsh Parliament and Parliament of Canada; GovReport (Huang et al., 2021) is a large-scale long document summarization dataset, consisting of about 19.5k U.S. government reports with expert-written abstractive summaries; GovReport is a good benchmark as it contains significantly longer documents (average 9.4k words) and summaries (553 words) than other long document datasets, such as ArXiv, PubMed (Cohan et al., 2018), Bill-Sum (Kornilova and Eidelman, 2019), and Big-Patent (Sharma et al., 2019); arXiv (Cohan et al., 2018) is a dataset of scien-tific articles from arXiv. Abstracts of the articles are used as the target summary. ArXiv is chosen over PubMed (Cohan et al., 2018) as arXiv contains longer articles compared to PubMed. Baselines for Comparisons We compare DYLE with the previous state-of-the-art methods on the aforementioned three datasets. More specifically: 1) For GovReport, we report the performance from the original paper, which uses various encoder self-attention and the proposed HEPOS encoder-decoder attention; 2) For QMSum, we compare with Zhong et al. (2021a), the current SoTA and other baselines mentioned in that work; 3) For arXiv, we include the results from the best performing models in previous works, including ExtSum-LG (Xiao and Carenini, 2019), PEGASUS (Zhang et al., 2020), DANCER (Gidiotis and Tsoumakas, 2020), BigBird (Zaheer et al., 2020), HEPOS + LSH (Huang et al., 2021), HAT-BART (Rohde et al., 2021), Longformer (Beltagy et al., 2020), and SSN-DM (Cui and Hu, 2021). Note that those baselines spans over different strategies to handle long input, such as sparse-attention (HEPOS, BigBird, Longformer), hierarchical attention (HAT-BART), extract-then-generate (Locator + different generators). Pretrained-LM The extractor is initialized with RoBERTa-base (Liu et al., 2019) weights. The generator is initialized with BART-large (Lewis et al., 2020a) weights. We use the Adam optimizer and set the extractor learning rate to 5e-5 and the generator learning rate to 5e-6. Hyperparameters  g ,  o , and  c are the coeffi-cients for the generation loss, oracle loss, and the consistency loss respectively. For  g and  o , we did a 2-step binary search between 0 and 2. For  c , we did a 3-step binary search between 0 and 10. For the QMSum dataset, we used  g = 1 ,  o = 1 ,  c = 1 . For the GovReport dataset, we used  g = 0 . 5 ,  o = 1 ,  c = 1 . For the ArXiv dataset, we used  g = 0 . 5 ,  o = 1 ,  c = 5 . Hardware We apply gradient checkpointing (Chen et al., 2016) to save the GPU memory. Each experiment is run on one NVIDIA Quadro RTX 8000 GPU. The effective batch size is set to 8. The evaluation results are summarized in Table 2, Table 3, and Table 4. For GovReport, DYLE yields gains of 4.15/6.21/4.00 of ROUGE-1/2/L scores compared to the previous best method. Experiments on GovReport show that DYLE is performant over prior sparse attention approaches. On QMSum, DYLE yields the new state-of-the-art ROUGE-1/2/L scores of 34.42/9.71/30.10, outperforms UniLM with DialogLM pretraining. Comparing DYLE with locator-based models on the QMSum dataset shows that DYLE outperforms prior extract-then-generate approaches where the locator is independently trained with intermediate annotated text spans. This shows the effectiveness of DYLE's joint training approach. These results show that DYLE can be applied to both the long document summarization and long dialogue summarization tasks. DYLE 's better performance can be attributed to lowered information loss between the extraction and the generation steps and its ability to handle input of a much longer length. We notice that while DYLE largely outperforms the LSH baseline (Huang et al., 2021) on the GovReport dataset, it underperforms the LSH baseline on arXiv. We posit two reasons. First, the input of the GovReport is much longer than that of arXiv. Most, if not all, of the sentences in the arXiv input article can be processed by the LSH model. Second, the summaries of the arXiv dataset are more abstractive than those of GovReport. It is possible that individually extracted text snippet is not the best linguistic unit for generating output tokens. It is our future work to explore the optimal input unit for an extract-then-generate approach. Nevertheless, DYLE outperforms other extraction-based approaches ( e.g., SSN-DM (Cui and Hu, 2021)) and divide-and-conquer approaches ( e.g., DANCER (Gidiotis and Tsoumakas, 2020)). We conduct ablation studies to investigate the effectiveness of the auxiliary optimizations we introduced. Specifically, we report the full model's performance after removing 1) hybrid training, 2) consistency loss, 3) extractive oracle loss. In our default model, the consistency loss is computed on the combination of the extracted snippets and oracle snippets; in the w/o hybrid experiment, the consistency loss is only computed on the set of oracle snippets; in w/o consistency experiment, the consistency loss is not computed. The results are summarized in Table 5. Note that without the hybrid training optimization, only the extractive oracles will be used to train the generator. When the consistency loss is not calculated, the extractor and the generator can be viewed as being trained independently with the extractive oracles. We see that excluding either of the hybrid training, consistency loss, or oracle loss optimization leads to a performance drop. Training the model without the supervision of the oracle leads to the greatest decrease in model performance, showing the importance of good supervision for the extractor. Removing the consistency loss also decreases the model performance. This shows that the consistency loss allows the extractor to better learn to select salient snippets from the input text and enables DYLE to generalize better to the test set. Analysis of extracted snippets We are interested in the amount of salient information passed to the generator. To investigate this, we report the decomposed precision and recall of ROUGE scores in Table 6. We observe that the extracted snippets have much higher recall than the generated summaries, while the generated summaries have higher precision. This suggests that to improve the overall performance, we can increase the information coverage (i.e., recall) of the extractor and improve the accuracy of the generator in identifying the salient snippets (i.e., precision). approach is more interpretable than sparse attention and two-step extraction-generation pipeline meth-1693 Specifically, dynamic weights in the generator shows how the information is used throughout the decoding process. In Figure 4, we visualize the dynamic weights for the extracted snippets assigned by the generator during decoding. In each subfig-ure, we visualize the dynamic weight matrices of the generated summary and a random summary from other samples in the validation set. The x axis and y -axis represent the index of the extracted topK snippets and the decoding time step, respectively. Darker squares denote higher weights. For each generated summary, we observe multiple consecutive high-weight areas, indicating alignments between the extracted snippets and the generated summary. By contrast, weights are uniformly distributed for random summaries. Interestingly, we observe that, on QMSum, fewer sentences are considered when generating the summaries. Our explanation for this observation is that QMSum is a query-based dataset, where the queried information is more concentrated in a few snippets. By contrast, we find that a larger number of snippets are used on the GovReport dataset as seen in Figure 4, as GovReport is a general summarization dataset. Effect of number of extracted snippets To evaluate the effect of number of extracted snippets on model performance, we vary the value of K of topK in Eq. (1) and test it on both the GovReport and QMSum datasets. We observe that the model performance generally increases as the value of K increases. This is expected as more extracted snippets provide the generator with more information to form a final summary. The results are summa-1694 R-1 R-2 R-L GovReport K =25 61.01 28.83 57.82 K =20 59.25 27.46 55.74 K =15 58.55 26.95 54.89 K =10 54.98 24.10 51.25 QMSum K =25 34.42 9.71 30.10 K =20 33.10 8.69 29.62 K =15 31.78 8.36 28.31 K =10 33.30 9.18 29.53 Table 7: Comparing model performance with different values of K on the GovReport and QMSum dataset R-1 R-2 R-L GovReportExtractor Output 61.01 28.83 57.82 Oracle 68.02 39.16 65.29 QMSumExtractor Output 34.42 9.71 30.10 Oracle 39.80 14.74 36.06 Table 8: Feeding extractive oracles to generator. rized in Table 7. Due to the limit of GPU memory, the largest K value we tried is 25. Effect of consistency loss We evaluate the effect of consistency loss on extractor performance. Note that removing the consistency loss means that the extractor and the generator are independently trained. The results are presented in Table 5 as part of the ablation study. Removing the consistency loss leads to worse model performance. We observe that the consistency loss helps the model better learn the importance of the selected text snippets useful for the generation. Extractor performance compared with extractive oracles We feed the extractive oracles to the generator. The results are summarized in Table 8. We observe that extractive oracles contain more salient information than the text snippets extracted by the extractor. Feeding the extractive oracle to the generator indicates the upper bound of the extractor performance. However, we observe that the gap between the performance of using the extractive oracle and using the extractor output is relatively small. Comparison with RAG The generator of our method is related to but differs significantly from Retrieval-Augmented Generation (RAG) (Lewis et al., 2020b). The similarity only lies in the idea of marginalization over a set of text snippets, which is shown to be useful in question answering as well (Ni et al., 2021b). However, unlike our dynamic weights, the weights in RAG remains static during decoding. In our notations, RAG's generation probability can be formulated as: P  ( y | q, XK ) = T (cid:89) t =1 P  ( y t | q, XK , y <t ) = T (cid:89) t =1 (cid:88) x  XKP  ( y t | q, x, y <t ) P  ( x | q, XK ) (8) The static weight P  ( x | q, XK ) in Eq. 8 is computed based on q and XK , while our dynamic weight P  ( x | q, XK , y <t ) is additionally conditioned on the already generated tokens. Limitations and future directions We acknowledge that joint training of the extractor and the generator cannot eliminate information loss, which might be addressed by combining DYLE and sparse attention to encode longer snippets. Though formulated for long-input summarization, DYLE can be applied to general long-input generation tasks where information is scattered across the input, e.g., open-domain question answering and multi-turn dialogue systems with long dialogue history. In this paper, we propose the first framework that jointly trains an extract-then-generate model with latent extraction. The first-step extraction picks out salient information from the long input, thereby extending the input length that the model can handle. Our novel joint training method addresses the challenge of information loss associated with the prior extract-then-generate approaches. Our model largely outperforms the current state-of-the-art on GovReport and QMSum, while achieving strong results on arXiv. Lastly, DYLE has the advantages of being able to process arbitrarily long input with a lower memory cost and interpretable generator weights. The authors would like to thank Yixin Liu and Ming Zhong for the discussions. We also would like to thank the anonymous reviewers for their helpful comments. This work is supported in part by a grant from Microsoft Research.
abstain abstain objective abstain objective other other objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain method objective abstain objective other other abstain abstain objective other result other other other other other other other other method other other other other method abstain abstain abstain abstain abstain abstain abstain abstain method method method method abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain method abstain method method method method method method abstain abstain method abstain abstain method method method method method method abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain other abstain other method other abstain abstain abstain method abstain method method method abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain other other other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method abstain result abstain abstain method method abstain abstain method abstain method method method method abstain method abstain other method method abstain abstain abstain method method abstain method abstain method method abstain method method method method abstain objective abstain objective other abstain other other other