Sentence fusion is the task of joining several independent sentences into a single coherent text. Current datasets for sentence fusion are small and insufficient for training modern neural models. In this paper, we propose a method for automatically-generating fusion examples from raw text and present DISCOFUSE , a large scale dataset for discourse-based sentence fusion. We author a set of rules for identifying a diverse set of discourse phenomena in raw text, and decomposing the text into two independent sentences. We apply our approach on two document collections: Wikipedia and Sports articles, yielding 60 million fusion examples annotated with discourse information required to reconstruct the fused text. We develop a sequence-to-sequence model on DISCOFUSE and thoroughly analyze its strengths and weaknesses with respect to the various discourse phenomena, using both automatic as well as human evaluation. Finally, we conduct transfer learning experiments with WEBSPLIT , a recent dataset for text simplification. We show that pretraining on DISCOFUSE substantially improves performance on WEBSPLIT when viewed as a sentence fusion task. Sentence fusion is the task of combining several independent sentences into a single coherent text (Barzilay and McKeown, 2005). Sentence fusion is important in many NLP applications, including retrieval-based dialogue (Song et al., 2018; Yan and Zhao, 2018), text summarization (Barzilay and McKeown, 2005; Bing et al., 2015) and question answering (Li et al., 2018; Marsi and Krahmer, 2005). Such systems retrieve multiple sentences from different sources, documents or paragraphs, and use them to construct a coherent text. Sentence fusion is challenging because it requires understanding the discourse semantics between the input sentences. Consider the example in Figure 1: a coherent fusion of the sentences requires understanding that the second sentence contrasts the first one, in order to insert the discourse connective However . In addition, the gender and syntactic role of the entity Zeitler needs to be inferred to insert the pronoun he . Prior work on sentence fusion (Barzilay and McKeown, 2005; Turner and Charniak, 2005; Filippova, 2010; Elsner and Santhanam, 2011; Thadani and McKeown, 2013; Bing et al., 2015; Chali et al., 2017) utilized very small amounts of labeled data, which are insufficient to train modern neural models. In this work, we propose a method for automatically generating sentence fusion examples at scale from raw text corpora. To this end, we go over sentences and contiguous pairs of sentences in a corpus, and apply a set of manually-constructed rules, which identify the occurrence of prevalent fusion operations. The rules specify how to modify the sentences such that they are unfused into two independent sentences. E.g., in Figure 1 one rule will delete the discourse connective However , and another will replace the pronoun he with the named entity Zeitler . In the generated examples, the original fused text becomes the target, and the unfused sentences (generated by rules) are the input. Importantly, sentence fusion models trained on our data cannot simply learn to invert rule application, because information is lost and can be recovered only by understanding the text semantics . As mentioned, learning to insert However in Figure 1 requires inferring that the sentences contrast. We cover a wide range of fusion phenomena such as inserting discourse connectives in various positions of the sentences, anaphora and cataphora identification, and sentence merging through coordination, relative clauses and apposition. We applied our method on two large document collections, Wikipedia and sports articles from the Web, resulting in two datasets of 16 million and 44 million examples respectively. We call the combined dataset DISCOFUSE . We extensively analyze the quality of our dataset with crowdsourcing, and find that workers understand the text after splitting in 85% of the cases, and the other 15% are due to either the original text being unclear or errors in rule application. We trained a state-of-the-art sequence-to-sequence model (Vaswani et al., 2017) and analyzed the fusion phenomena in which the model struggles. We found that the model succeeds in fusing sentences through structural constructions such as apposition or relative clauses, but performs badly when fusion involves inserting a particular discourse connective, or selecting pronominals. Last, we performed transfer learning by training on DISCOFUSE and then fine-tuning on a smaller dataset from a different distribution. To this end, we utilize WEBSPLIT , a recent dataset for sentence splitting (Narayan et al., 2017; Aharoni and Goldberg, 2018), viewing WEBSPLIT as a sentence fusion task. We found that pre-training on DISCOFUSE substantially improves the performance of a fusion model in this setup. To conclude, our contributions are: 1. DISCOFUSE : a dataset of 60 million sentence fusion examples from two different corpora. 2. A method for automatically generating sentence fusion examples from raw text. 3. Automatic and human evaluation of the Transformer model on the fusion task. 4. A transfer learning setting in which model performance improves when pre-trained with DISCOFUSE . Existing fusion datasets are small, which is perhaps why only few works have explored the application of supervised models to sentence fusion (Elsner and Santhanam, 2011; Thadani and McKeown, 2013). McKeown et al. (2010) introduced a human-generated corpus of 3,000 examples. Elsner and Santhanam (2011) extracted around 300 fusion examples from preand post-editing news articles. Thadani and McKeown (2013) constructed 1,858 examples from summarization tasks. Such datasets are too small to train modern data-hungry neural models. Related to sentence fusion is its inverse task of sentence splitting. Collados (2013) automatically constructed a Spanish simplification dataset by splitting single sentences into several simpler ones. Recently, two larger datasets for text splitting were released (Botha et al., 2018; Narayan et al., 2017; Aharoni and Goldberg, 2018). However, using these datasets for the mirror task of sentence fusion is problematic. First, sentence splitting often involves removing content from the original sentence for simplification, and this content is impossible to recover in the fusion direction. Second, these datasets do not focus on discourse and thus prominent discourse phenomena may be missed. Last, our new dataset is more than an order of magnitude larger than the above sentence splitting datasets. Another related line of recent work focused on predicting discourse connectives between sentences and automatically generating examples from raw text (Liu et al., 2016; Malmi et al., 2018). We substantially expand over those works by handling more diverse linguistic phenomena, such as connectives in single sentences, generating anaphora and cataphora constructions, relative clauses, coordination and more, which are all represented in a single dataset. Moreover, our dataset is 20x larger compared to prior work, allowing us to examine in depth long-tail scenarios. We next describe our process for building DISCOFUSE , which contains 60 million sentence fusion examples from two different document collections: Wikipedia and Web articles about sports. DISCOFUSE contains union-fusion examples, i.e. fusing sentences without loss of content (Marsi and Krahmer, 2005). To automatically extract examples, we manually crafted a list of text splitting rules. Our rule-set covers 9 fusion phenomena, including handling discourse connectives, coordination and relative clauses, and entity resolution for anaphora and cataphora constructions. For entity resolution, both anaphoric pronouns ( she , they , his ) and anaphoric nominals ( the team , the man ) are considered, based on the output of a coreference system. The covered phenomena are summarized in Table 1 and a detailed description is given in Appendix A.1. Given a text t consisting of one or two consecutive sentences, each of our rules addresses a specific discourse phenomenon and has two parts: (a) conditions for matching the phenomenon in t , and (b) operations over a dependency tree annotated with coreference resolution. Applying the operations generates a fusion example ( x = ( s 1 , s 2 ) , t ) , in which ( s 1 , s 2 ) are two independent sentences originating from t , but stripped from the discourse phenomenon that tied them in t . Figure 2 gives an example of a rule for the apposition structure. The rule is applied to the sentence The Jacksonville Jazz Piano Competition, a 30 year tradition, takes place at the Florida The-atre . First, the input is matched to the rule's condition. In this case, the condition is a single clause surrounded by two commas, which has a determiner as its first token and includes an apposition with incoming edge from a preceding token to the clause. Once matched, an example is generated. For this rule, the first sentence is created by removing the apposition clause, and the second sentence by removing the part after the clause and inserting the appropriate be verb ( is ). Generation examples for all 9 rule types are provided in Table 1. As explained in Section 1, solving sentence fusion involves more than just reverse-engineering the generation rules. The model needs to decide whether to insert a discourse connective with the right semantics, whether to merge the input sentences, and what syntactic construction (relative clause, coordination, apposition) is most appropriate in the given context. Last, often several discourse phenomena occur in a single text t . Thus, we allow combining anaphora rules with one of the following rule types: discourse connective, inner connective and sentence coordination, which cover frequent combinations in our texts. To create DISCOFUSE we retrieved the latest Wikipedia release and crawled the Web for several million sports articles. Documents were annotated with dependency trees and coreference resolution using Google Cloud Natural Language. 1 We considered each sentence and pair of consecutive sentences in each document as candidates, applying the example generation process described in Section 3.1. Additionally, we added as examples sentence pairs from the original corpus that did not match any rule, that is ( s 1 , s 2 ) = t , so that a trained model would also learn when not to change the input. We filtered out examples with sentence length  6 tokens, and examples with non-ASCII characters. This process resulted in 44 , 177 , 443 sports examples and 16 , 642 , 323 Wikipedia examples. We randomly split these examples into 98% train, 1% dev, and 1% test sets, 1 https://cloud.google.com/natural-language/ Phenomenon Example Discourse (A) Hebden Bridge is a popular place to live . Like prior work (Malmi et al., 2018), we observed a skewed distribution of discourse phenomena in the data. Specifically, examples with anaphora or the connectives and and but constitute 99.7% of Sports and 59% of Wikipedia examples. Such a skewed distribution is likely to bias models and will fail to elucidate the ability of models to capture a wide range of linguistic phenomena. Therefore, we constructed a version of DISCOFUSE by down-sampling examples containing and and but or anaphora. The down-sampled dataset contains 12,080,513 Sports examples and 4,581,352 Wikipedia examples. The resulting distributions of discourse types and most common connectives in the two parts of DISCOFUSE are provided in Appendix A.2. We will release both the original and the down-sampled versions of DISCOFUSE . To assess the quality of the generated fusion examples in DISCOFUSE , we randomly selected 500 examples from each of the development sets of the Wikipedia and the Sports parts. We then conducted a crowdsourcing experiment in which each example was rated by 5 proficient English speakers, limiting each rater to at most 6 items. Each rater was presented with the two independent sentences in the example and was asked to indicate whether the text is understandable. If the rater answered yes, she was then asked to characterize the relation between the sentences and how she would fuse them. We next detail the results. Reason Example Originaltext (A) UPDATE: Peat falls because footwork and quickness. unclear (a) UPDATE: Peat falls. (b) Footwork and quickness. Missingcontext (A) We were right on the heels of Spurs, although Everton were closing in. (a) We were right on the heels of Spurs. (b) Everton were closing in. Bad rule genera-tion (A) He told reporters after the game his reaction was because he missed a wide-open Randall Cobb in the end zone. (a) He told reporters after the game his reaction was.(b)He missed a wide-open Randall Cobb in the end zone. Raters were asked whether they can understand the text after the example is split. Table 2 summarizes this evaluation. Most examples were marked as understandable by the raters (yes)  86% of Wikipedia examples and 83.4% of Sports examples. The rest either had no majority of rater votes or were marked as not understandable. To shed light on the possible reasons for obscurity, we analyzed 70 random examples that were not marked as understandable by the majority of raters. In 29 examples (41%) the original text was unclear and for 17 examples a broader context was needed (24%). In the remaining 24 examples (34%), our rules generated sentences with grammatical or semantic errors. Examples for these cases are in Table 3. Additionally, we analyzed 100 random examples for grammatical errors, and found that our rules did not introduce any errors in 79 examples. For 15 examples, the errors did not modify the meaning of the text nor caused semantic errors. The detected grammatical errors include extra commas, missing determiners and bad pronoun replacements, and are demonstrated in Table 4. 4.2 Fusion Evaluation Next, we evaluated agreement on the fusion task for the 847 examples marked as understandable in Section 4.1. Because there are many ways in which sentences can be fused, one cannot expect raters to produce the original text t verbatim. Instead, we analyzed three central decisions and estimated whether people agree on those: (a) whether to merge the two sentences into a single one or keep them separate; (b) whether there are entities in the text that should be replaced with nominal or pronominal anaphors or cataphors; and (c) which discourse connective to add (if any). For the last question, we presented raters with one connective from each of the four coarse-grained senses for discourse connectives defined by the PDTB (Prasad et al., 2008): comparison , expansion , contingency and temporal , as well as a no-connective option. If the original text in the example includes a connective, we provided it as one of the options. We observed a strong bias among raters towards refraining from performing any changes. E.g., while only 38% of the examples did not contain a connective in t , the raters chose not to add a connective in 69.2% of the cases. Similarly, only in 29.1% of the examples the two sentences were not merged into a single one, while the raters chose not to merge in 53.1% of the examples. Similar behavior was also observed by Malmi et al. (2018) and Rohde et al. (2016). We further looked at the agreement between the rater majority and the gold' fusion decision. This analysis is shown in Table 5. Agreement on merging the input sentences into one is almost random (52%), since usually both options are valid. Consensus on whether to add an anaphor is higher, but not very high (63%), especially in sentences when the anaphor in t is a nominal rather than a pronoun. Finally, there is higher agreement on selecting the connective category (57%), for which the random baseline is 20%. As mentioned, raters tend to keep the sentences unchanged. But in cases where raters agree to add a connective, agreement figures increase substantially. Specifically, when it is clear that a connective is needed, there is also high agreement for picking the right one (76%), for deciding whether Decision All Examples (%) Examples with Type connectives (%) Single / pair 52.0 70.1 Anaphor 63.4 70.1 Connective 57.0 76.4 category # Examples 847 271 Table 5: Average agreement for each fusion decision between the gold annotation and rater majority on examples marked as understandable by the raters. Using DISCOFUSE , we trained a Transformer seq2seq model (Vaswani et al., 2017) that reads the input sentence pair and generates a fused text. We report model performance on the test-set using automatic metrics as well as human evaluation. We also provide detailed analysis of the different phenomena captured by this model. We tokenized all texts using byte-pair-encoding (Sennrich et al., 2015) and compared the following three Transformer models : DFSPORTtrained on the sports portion of DISCOFUSE after down-sampling. DFWIKItrained on the Wikipedia portion of DISCOFUSE after down-sampling. DF S+W trained on a 50%-50% mixture of the sports and Wikipedia portions of DISCOFUSE after down-sampling. All models share the same network architecture, based on the best practices discussed by Popel and Bojar (2018). We tuned parameters to select the best learning and dropout rates for each model with respect to the Exact Match objective (de-scribed in Section 5.2). Network architecture and hyper-parameters are in Appendix A.3. As a baseline, we also tested a model called COPY , which simply concatenates the two input sentences. We evaluated model performance using two automatic metrics. The first is Exact Match ( Exact ) to see how often the model generates the exact same SPORTS Full Sampled SARI Exact SARI Exact DFSPORT 81.9 42.3 % 83.9 50.6% DFWIKI 77.8 31.7% 80.1 40.1% DF S+W 80.7 38.3% 82.9 47.0% COPY 40.0 1.1% 40.4 3.8% WIKIPEDIA Full Sampled SARI Exact SARI Exact DFSPORT 80.0 41.5% 80.0 41.9% DFWIKI 83.1 47.6% 84.5 51.1% DF S+W 82.8 46.7% 83.7 49.2% COPY 40.3 1.0% 39.6 2.1% Table 6: Exact and SARI scores of DFSPORT , DFWIKI , DF S+W and COPY , on the test sets of DISCOFUSE before ( Full ) and after down-sampling ( Sampled ). text as the gold fusion. The second is SARI (Xu et al., 2016), which computes the set of added, removed, and kept n-grams in the model output, comparing the output both with the gold text and the input text. Then it computes the F 1 scores for these three sets and averages the scores. We compute SARI on up to 4-grams, as in Xu et al. (2016). We refrained from using metrics like BLEU because in fusion there is large overlap between the input sentences and their fused version, and such metrics do not capture well fine-grained differences of only a single word. We note that our definition of SARI 2 slightly differs from the one given by Xu et al. (2016) in 2 Our SARI implementation is available at: https: //github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/sari_hook.py two aspects: ( i ) We define 00 = 1 when computing precision and recall, otherwise SARI could be less than 1 even if the output matches the gold text exactly. ( ii ) Instead of considering only the precision of deleted n-grams, we use F 1 for all three sets. Otherwise, SARI will give high scores to models that merely copy everything in the input, without even trying to infer what to delete. Table 6 summarizes the results. When training and testing on the same domain, either Sports or Wikipedia, SARI score is a little above 80 points for the full dataset. Yet Exact is not high, around 42% for Sports and 47% for Wikipedia, showing that in the majority of the examples the model's fusion differs from the gold. Tested on the down-sampled test-set, performance increases significantly for Exact , especially on Sports, where discourse phenomena is more skewed. We next turn to cross-domain evaluation. When applying a model trained on one domain to the other domain performance drops. This shows that the discourse phenomena distribution differs between the domains, indicating that transfer learning is not trivial even with these large datasets. This is especially evident when applying DFWIKI to Sports, where Exact falls from 42% to 32% on the full dataset and from 50% to 40% on the down-sampled one. Interestingly, when learning on the mixed training set, performance on both domains is close to in-domain performance, showing that the model has the capacity to handle both domains. Finally, we take advantage of the provided annotation of the different discourse phenomena within each example in DISCOFUSE . We conducted a detailed analysis of in-domain model performance by discourse type, presented in Table 7. Results show that structural discourse types, such as apposition and relative clause, are easier to learn with both high exact match and SARI scores. While differences with respect to SARI scores are not large between phenomena, exact match varies more. Anaphora and verb phrase coordination are more challenging, but still require matching of the same noun (the named entity or the subject). On the other hand, discourse types that involve connective prediction, such as sentence coordination and discourse connective, require semantic understanding, and performance is significantly lower. In addition, when two discourse types are required for fusion, performance drops dramatically. As our second experiment we employed crowdsourcing to test how distinguishable the fusion model outputs are from the gold fused texts. Concretely, we present raters an independent sentence pair from DISCOFUSE and two fused versions the gold version and one generated by a model. Raters were asked to detect the gold version. For each example, we took the majority of 5 raters as the final choice. This experiment mitigates the dif-ficulties of automatic text generation evaluation, where many outputs are valid for a single input. We sampled 1000 random examples from each development set of the two domains and applied the in-domain model to both. The raters were presented only with examples where the model output was different from the gold fusion, and assumed 50% detection accuracy otherwise. Table 8 depicts the results. Out of cases when model output differed from the gold, raters were able to identify the human version in 65% of Sports examples and 61% of Wikipedia examples. Looking at the entire set, humans were able to identify the human version in 57% (Sports) and 55% (Wikipedia) of the cases. This shows that our Transformer model, applied over a dataset of millions of examples, is able to learn good fusions in general. Nevertheless, models are still far from perfect  human accuracy is clearly better than random and this improvement is statistically significant at a level of p < 10  5 for Sports and p < 10  3 for Wikipedia. We next present an analysis of the types of errors our models produce. To this end, we sampled 40K examples of DFSPORT and DFWIKI outputs on Sports and Wikipedia development sets. We then automatically aligned predicted sequences to gold Connective DFSPORTDFWIKI Top 3 connectives accuracy accuracy and 50.9 53.7 and, but, (cid:104) other (cid:105) but 42.8 43.7 but, (cid:104) other (cid:105) , and because 61.5 60.7 because, (cid:104) other (cid:105) , but although 35.1 33.2 although, (cid:104) other (cid:105) , but so 50.6 50.2 so, but, and or 70.5 72.1 or, and, (cid:104) other (cid:105) however 28.3 26.7 (cid:104) other (cid:105) , however, but while 70.1 70.6 while, (cid:104) other (cid:105) , but so that 64.3 63.0 so that, (cid:104) other (cid:105) , because unless 68.9 67.0 unless, because, (cid:104) other (cid:105) for example 26.9 28.1 (cid:104) other (cid:105) , for example, however Table 9: Alignment-based connective prediction accuracy for the most common connectives. sequences and looked at the differences between aligned words. The trained models successfully learned to copy most of the input text, and thus errors due to alignment problems are rare. We start by considering the semantic relation between the input sentences. Table 9 displays model accuracy in predicting the most common connectives in DISCOFUSE , as well as the top connectives predicted in this slot. We observe that when the model predicts a wrong connective, that connective is often reasonable, e.g., predicting but instead of and or however . Moreover, a second source of error is not adding a connective at all. It is also clear that some connectives, like however , although and for example , are harder to learn. We also analyzed the models' ability to correctly infer pronoun anaphors including gender, possessive and plurality. Figure 3 shows the pronoun confusion matrix for DFWIKI , 3 where lines refer to gold pronouns and columns to the generated pronoun in the same position. The clear diagonal shows that in most cases, the model successfully outputs the correct pronoun. However, the (cid:104) other (cid:105) column indicates that occasionally the model tends not to replace the entity in the input with a pronoun anaphor. In addition, the model seems to struggle with possession and plural 3rd person ( it , its , they , their , theirs ). With the DISCOFUSE approach we can collect a large amount of examples automatically. Still, these examples only reflect the manual rules that identify discourse phenomena. We wanted to see if DISCOFUSE covers enough cases such that a trained model would be helpful for testing on fusion datasets generated by different approaches. In this experiment, we looked at the recently released WEBSPLIT dataset 1.0 (Narayan et al., 2017). It consists of examples ( t, { s i } ni =1 ) , where t is a sentence that verbalizes the same set of RDF triples as { s i } ni =1 . We note that WEBSPLIT was originally developed for sentence splitting, from t to { s i } n i =1 , but here we view its examples for the reverse fusion task: from { s i } ni =1 to t . We only considered examples where { s i } ni =1 corresponds to exactly two simpler sentences ( n = 2 ). This leaves us with 135K training, 8K validation, and 8K test samples. We tokenized the data using byte-pair-encoding (Sennrich et al., 2015) and compared three models: ( i ) The COPY baseline that concatenates the two input sentences, ( ii ) a model trained on WEBSPLIT alone, and ( iii ) a model pre-trained on DFWIKI and fine-tuned on WEBSPLIT . For the last two models, we use the CopyNet architecture (Gu et al., 2016), which is similar Training data SARI Keep Add Delete COPY 18.1 52.9 0.5 0.9 WEBSPLIT 40.5 44.6 7.8 69.3 DFWIKI + WEBSPLIT 44.2 54.8 10.4 67.5 Table 10: Fusion results on WEBSPLIT , measured by SARI and the F1 scores that compose it. to state-of-the-art models for the splitting task on WEBSPLIT (Narayan et al., 2017; Botha et al., 2018). While the Transformer outperformed this model on our main experiments, here it overfit on the small training set of WEBSPLIT . The training details are provided in Appendix A.3. Table 10 shows the results of the experiment. Similarly to Section 5, we measured the model performance using SARI . Pre-training with DFWIKI improves SARI score by 9% compared to using WEBSPLIT alone. In particular, the F1 of the kept' and added' n-grams is significantly higher, by 23% and 33% respectively. Specifi-cally, added' tokens refer also to correctly choosing discourse connectives, to which the large-scale examples in DISCOFUSE were likely helpful. We note that even with pre-training, the SARI add' score is only 10.4. This is probably due to the large amount of paraphrasing done in WEBSPLIT , which makes it problematic for fusion evaluation (see also Section 2). For example: Sentence 1: Bolt , a comic character AKA Larry Bolatinsky , was created by Paris Cullins and Ernie Colon . Sentence 2: Paris Cullins is a United States national . Gold: Larry Bolatinsky is the alternative name for the comic book character Bolt , which was created by Ernie Colon and the American Paris Cullins . Correctly inferring the added terms (shown in red) requires paraphrasing knowledge that is outside the scope of DISCOFUSE . We presented DISCOFUSE , a large-scale dataset for sentence fusion that was generated by applying a rule-based method. It contains millions of examples from two domains, annotated with multiple discourse phenomena. We used DISCOFUSE to build supervised neural models for sentence fusion and conducted fine-grained analyses of the results. Currently, our models fuse only two sentences together. We would like to expand them to more input sentences in future work. We also demonstrated DISCOFUSE 's usefulness in a transfer learning setup on a different fusion test-set, hoping it would facilitate research on text fusion in data-scarce domains. We thank Marta Recasens, Jan Botha, and Sascha Rothe from Google AI for helpful discussions. This research was supported by the Yandex Initiative in Machine Learning and by The Israel Science Foundation grant 942/16. This work was completed in partial fulfillment for the Ph.D degree of the first author.
abstain abstain objective method method objective method result abstain abstain abstain abstain abstain abstain abstain objective method abstain abstain abstain method abstain abstain result abstain result method result method method result objective objective objective objective other other other other other other other other other other other objective other method method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method method method abstain other abstain abstain abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain method method method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain result abstain abstain objective other other other