We introduce SCDE, a dataset to evaluate the performance of computational models through sentence prediction. SCDE is a human-created sentence cloze dataset, collected from public school English examinations. Our task requires a model to fill up multiple blanks in a passage from a shared candidate set with distractors designed by English teachers. Experimental results demonstrate that this task requires the use of non-local , discourse-level context beyond the immediate sentence neighborhood. The blanks require joint solving and significantly impair each other's context. Furthermore, through ablations, we show that the distractors are of high quality and make the task more challenging. Our experiments show that there is a significant performance gap between advanced models (72%) and humans (87%), encouraging future models to bridge this gap. 1 2 1 Introduction Cloze questions were first proposed by Taylor (1953) as a readability test, motivated by Gestalt psychology. They become an efficient way of testing reading for public exams, overtaking the dominant paradigm of subjective questions (Fo-tos, 1991; Jonz, 1991). Cloze datasets (Zweig and Burges, 2011; Hermann et al., 2015; Hill et al., 2015; Paperno et al., 2016; Onishi et al., 2016; Xie et al., 2018) became prevalent as question-answering (QA) benchmarks since they are convenient either to be generated automatically or by annotators. These datasets could be split into two clear types: 1. Where the context is a complete text, and there is an explicit question posed which is a statement with a cloze gap. The answer is either generated freely or is a span  Equal Contribution 1 Data: vgtomahawk.github.io/sced.html 2 Code: https://github.com/shawnkx/SCDE from the context, e.g. Children's Books Test (CBT) (Hill et al., 2015). 2. Where the context itself comes with cloze gaps. There is no explicit question. The answer is generated freely or chosen from a set of candidates, e.g. CLOTH (Xie et al., 2018). Herein, we focus on the 2nd category. A common property of these datasets is that they have gaps at the level of words, entities or short syntactic spans. The entity and span-based clozes may sometimes be multi-token, but they do not extend beyond a few tokens. Nevertheless, none of these datasets have cloze gaps at the level of full sentences . Since many syntactic and semantic cues are present in the same sentence, this makes the gap easier to fill compared to the sentence level cloze case where models would have to rely on discourse cues beyond the same sentence. Besides lack of intra-sentence cues, sentence-level cloze may require comparing candidates of very different lengths. For instance, the example in Table 1 has a standard deviation of 7 . 6 with candidate lengths between 3 to 25 . A model that only represents words well may not get comparable probabilities at sentence level for very different sentence lengths. Therefore, robust sentence representation models are also required to solve this question. In this paper, we present SCDE, a dataset of sentence-level cloze questions sourced from public school examinations. Each dataset example consists of a passage with multiple sentence-level blanks and a shared set of candidates. Besides the right answer to each cloze in the passage, the candidate set also contains ones which don't answer any cloze, a.k.a., distractors . Both cloze positions and distractors are authored by teachers who design the public school examinations carefully. 3.2 explains our data collection. A representative example from SCDE is shown in Table 1. Passage: A student's life is never easy. And it is even more difficult if you will have to complete your study in a foreign land. 1 The following are some basic things you need to do before even seizing that passport and boarding on the plane. Knowing the country. You shouldn't bother researching the country's hottest tourist spots or historical places. You won't go there as a tourist, but as a student. 2 In addition, read about their laws. You surely don't want to face legal problems, especially if you're away from home. 3 Don't expect that you can graduate abroad without knowing even the basics of the language. Before leaving your home country, take online lessons to at least master some of their words and sentences. This will be useful in living and studying there. Doing this will also prepare you in communicating with those who can't speak English. Preparing for other needs. Check the conversion of your money to their local currency. 4 . The Internet of your intended school will be very helpful in findings an apartment and helping you understand local currency. Remember, you're not only carrying your own reputation but your country's reputation as well. If you act foolishly, people there might think that all of your countrymen are foolish as well. 5 Candidates: A. Studying their language. Discussion: Blank 3 is the easiest to solve, since Studying their language is a near-paraphrase of Knowing even the basics of language. Blank 2 needs to be reasoned out by Inference specifically E can be inferred from the previous sentence. Note however that C is also a possible inference from the previous sentence it is only after reading the entire context, which seems to be about learning various aspects of a country, that E seems to fit better. Blank 1 needs Summary  it requires understanding several later sentences and abstracting out that they all refer to lots of preparations . Finally, Blank 5 can be mapped to B by inferring that people thinking all your countrymen are foolish is bad , while Blank 4 is a easy WordMatch on apartment to D. The other distractor G, although topically related to preparation for going abroad, does not directly fit into any of the blank contexts Table 1: A Representative Example from SCDE. Another salient aspect of our dataset is that more than 40% of blanks belong to the reasoning category Inference (more on this in  3.3 and Table 4) which require models to compare plausibility of competing hypotheses given a premise (whether the previous or last sentence(s), or even a combination of information from the two). Filling these blanks requires the model to reason by using commonsense knowledge, factual knowledge, time gaps, etc. Some of these can be thought of as simple entailment, but more generally, many of these can be seen as requiring abductive reasoning, which is of recent interest (Bhagavatula et al., 2019; Sap et al., 2019a,b) to the NLP community. In summary, our contributions are as follows 1. We introduce the task of sentence level cloze completion with multiple sentence blanks and a shared candidate set with distractors. 2. We release SCDE, a sentence level cloze dataset of  6 k passages and  30 k blanks. 3. We estimate human performance on SCDE, and benchmark several models, including state-of-the-art contextual embeddings (Ta-ble 5). We find a significant gap of > 15% for future models to close in order to match human performance. 4. Through several ablations described in  5.6, we show that distractors designed by English teachers are of high quality and make the task more challenging. 5. We show that extra sentence level cloze questions generated automatically from an external corpus can be used to further improve model performance through data augmentation (See  5.7). Several cloze test datasets are collected to measure reading comprehension ability of machines. CNN/DailyMail (Hermann et al., 2015), an early dataset of current QA research, constructs cloze questions from article summaries, with article spans as answers. Their cloze gaps are entities and hence one or few tokens long at best. The LAMBADA dataset (Paperno et al., 2016) constructs a corpus of word level cloze gaps, such that each gap is in the last passage sentence. CBT (Hill and Simha, 2016) creates word level cloze questions by removing a word in the last sentence of every consecutive 21 sentences, with the first 20 sentences being the context. Onishi et al. (2016) curate a dataset of who-did-what type sentences with Dataset SL MB Distractors Candidates Position (cid:107) Context (cid:107) w SCDE (cid:51) (cid:51) Human Shared Anywhere 319 ROCSTORIES (2016) (cid:51)  Human End 25 CLOTH (2018)  (cid:51) Human Separated Anywhere 243 LAMBADA (2016)   Exhaustive End 76 CBT (2015)   Automatic End 465 MRSCC (2011)   Human Anywhere 20 Table 2: Comparing SCDE with previous cloze datasets. word level blanks. The CLOTH (Xie et al., 2018) dataset collects word level cloze questions from English exams designed by teachers. MRSCC (Zweig and Burges, 2011) consists of 1,040 word level cloze questions created by human annotators. Among recent cloze datasets, ROCStories (Mostafazadeh et al., 2016) is the closest we could find to a sentence level cloze dataset. In this task, the first 4 sentences of a 5-sentence story are provided, and the task is to choose the correct ending from a pair of candidate ending sentences. However, there are several key differences between SCDE and ROCStories. Firstly, there are multi-blanks in SCDE which are not in a fixed position and require learning cues from bidirectional contexts of varying lengths. Secondly, the endings in ROCStories have been found to contain anno-tation artifacts (Gururangan et al., 2018) which makes a large fraction of them predictable independent of context. In contrast, SCDE is by design independent of artifacts, since a) given a blank, only some of our candidates are distractors, the rest being answers for other blanks. Even if one were to learn a classifier to distinguish distractors without context, the non-distractor candidates would be unresolvable without context. b) we further check how distinguishable our distractors are from non-distractors without context by training a strong classifier in this setting, as described in  5.6. The classifier obtains a reasonably low F1 score of 0 . 38 . Public school examinations have been used as a data source by many earlier QA works, two prominent examples being the CLEF QA tracks (Penas et al., 2014; Rodrigo et al., 2015) and RACE (Lai et al., 2017). In this task, each question consists of a passage, S multiple sentence level blanks B , and a shared set of candidates C with distractors D , where D  C Problem Complexity 3 For our case, given the typical value of | C | and | B | being 7 and 5 respectively, the size of the answer space, | A | is 2520. Thus, the chance of guessing all blanks correctly at random is only 0.04%. Moreover, there is a 48 . 2 % probability of being entirely wrong with randomly guessing. Finally, given an answer list chosen uniformly at random, the expectation of number of distractors in the answer list is 1.4, i.e. on average, roughly one and half answers are distractors. Raw sentence cloze problems are crawled from public websites 4 which curate middle and high school English exams designed by teachers. In total, 14,062 raw passages and 68,515 blank questions are crawled from these websites and the following steps are used to clean them. Firstly, duplicate passages are removed. Secondly, when the of-ficial answer to the problems are images, two OCR toolkits 5 are employed to convert these images to text and the questions with different results from these two programs will be discarded. Finally, we remove examples which have 1) answers pointing to non-existent candidates, 2) missing or null candidates, 3) number of blanks > number of candidates, 4) missing answers. After cleaning, we obtain our SCDE dataset with 5,959 passages and 29,731 blanks. They are 3 We defer the derivation to Appendix  1 4 http://www.21cnjy.com/; http://5utk.ks5u.com/; http://zujuan.xkw.com/; https://www.gzenxx.com/Html/rw/. 5 tesseract; ABBYY FineReader Statistic Value Total Passages 5,959 Total Blanks 29,731 Blanks Per Passage 4.99 # Candidates Per Passage 6.79 Avg Candidates Per Blank 1.35 % Consecutive Blanks 1.28 # Words Per Passage 319.64 Vocabulary Size 48.6k Var(Candidate Length) 19.54 Table 3: SCDE Statistics. randomly split into training, validation and test sets with 4790, 511 and 658 passages respectively. The detailed statistics are presented in Table 3. We find that candidates have very different lengths and passages have long context. In order to evaluate students' mastery of a language, teachers usually design tests in a way that questions cover different aspects of a language. Reasoning Types As illustrated with examples in Table 4, we set a four-fold categorization for the reasoning which leads to a ground truth candidate being assigned to a blank. Our reasoning type taxonomy is motivated by categorization of question types in earlier works in QA such as (Chen et al., 2016; Trischler et al., 2017) 6 . Strictly speaking, these reasoning types could co-exist. But for simplicity, we classify each blank into only one of the four. WORDMATCH : If the candidate has word overlap, especially of non-stopwords or infrequent phrases, with context around the blank. PARAPHRASE : If the candidate doesn't have an explicit word overlap with the context, but nevertheless contains words or phrases which are paraphrases of those in the context. INFERENCE : If the candidate is a valid hypothesis conditioned on the left context [as premise], or a necessary pre-condition/premise based on the right context. Note that the candidate in this case doesn't contain word overlap/paraphrases which would obviate need for inferential reasoning. The reasoning required needs not 6 See Section 4.2 from both respective papers. be just strict entailment (Bowman et al., 2015; Marelli et al., 2014) but could also involve abductive reasoning (Bhagavatula et al., 2019), where the candidate is just one of many likely hypothesis (premise) given the left (right) context as premise (hypothesis). SUMMARY : If the candidate is a summary, introduction, or conclusion of multiple sentences before or after it. In this type, unlike INFERENCE , there is no requirement to deduce and reason about new hypothe-ses/possibilities not present in the premise only consolidation and rearranging of information is required. A sample of 100 passages containing 500 blanks are manually categorized into these four categories. Examples and statistics of these four types are listed in Table 4. More than 40% blanks need inference to be solved, denoting the high difficulty of our dataset. We experiment with giving our models different amounts of context. Through this, we can explore how context length affects model performance. 1. P(N): Immediate previous (next) sentence 2. P+N: Immediate previous and next sentence 3. AP(AN): All previous (next) sentences 4. AP+AN: All previous and next sentences AP+AN is the unablated setting, where all passage sentences are available to the model. Before exploring deep representational approaches, we would like to find how well symbolic ones perform at this task. Starting with works such as Iyyer et al. (2015) and Arora et al. (2017), it has become convention to first benchmark simple baselines of this kind. PMI merely encodes how likely it is for a word pair to occur in consecutive sentences. It does not consider the internal sentence structures, or the relative position of the words in their respective sentence. Intuitively, it can be called a surface-level approach. A high performance by PMI would indicate that candidates can be matched to blanks by simple ngram statistics, without requiring sentence representation, which would make SCDE uninteresting. Note that our PMI definition diverges from typical PMI since its asymmetric between w s and w c . Since S and C are the sets of non-terminating and non-starting sentences respectively, they overlap but aren't identical. For a pair of sentences, we find aggregate PMI ( S, C ) as: PMI ( S, C ) = 1 | C || S | (cid:88) w c  C (cid:88) w s  SPMI ( w s , w c ) This definition can be extended to all n-grams upto a certain n . We denote this by PMI n . We notice that PMI n performance saturates after n = 2 . Hence, in our experiments, we use PMI 2 . One intuitive way to solve this task is to generate the blank sentence given the context by advanced pre-trained language models (LM). Formally, suppose the blank is the i th sentence, s i , and s 1 , . . . , s i  1 , s i +1 , . . . , s n are the context. Our goal is to choose c k from the candidate set which could maximize the joint probability p ( s 1 , . . . , s i  1 , c k , s i +1 , . . . , s n ) . Due to limited number of passages available to train a robust LM, Transformer-XL (TR.XL) Base (Dai et al., 2019), trained on WikiText-103, is employed to address this task. In order to make decoding time tractable, context length is limited to three sentences before and after the blank. Coherence models assign a continuous score to a sentence sequence indicative of its coherence. This score is usually unnormalized and not needed to be a probability [unlike language models]. We use the local coherence approaches implemented by the COHERE 7 framework (Smith et al., 2016). Roughly, this model works on the intuition that successive sentences exhibit regularities in syntactic patterns. Specifically, it uses ngram patterns on linearized syntactic parses (e.g. S NP VP . . . ) of consecutive sentences. Once 7 github.com/karins/CoherenceFramework trained, this model can return a coherence score for any sentence sequence. The COHERE model is first trained on all ground-truth passages from our training set, with the ground truth answers filled into the blanks. At test-time, we score each possible answer permutation using the trained COHERE model and pick the highest scoring one. Note that decoding for COHERE is by definition exhaustive, and doesn't make any assumptions by answering the blanks in a particular order. Conneau et al. (2017) use textual inference supervision as a signal to train a shared sentence encoder for premises and hypotheses, which can later be used as a sentence representor. We refer to this approach as INFST. Context features of a given blank and one candidate feed to two encoders in INFST respectively and classify whether this candidate is suitable to this blank. The maximum tokens of context features is set as 256. Bi-directional LSTMs with the max pooling operation are employed as our encoders. We follow the training procedure described in Conneau et al. (2017). Input Representations Let c k denotes the k th candidate. s  i and s + i denote the i th sentence before and after the blank respectively and | P | and | N | represent total number of sentences before and after the current blank respectively. Following the input convention in Devlin et al. (2018), the input sequence given various context lengths and c k is: 1. P : [ CLS ] s  1 [ SEP ] c k 2. N : [ CLS ] c k [ SEP ] s +1 3. AP : [ CLS ] s  | P | . . . s  1 [ SEP ] c k 4. AN : [ CLS ] c k [ SEP ] s +1 . . . s + | N | To retain sentence sequentiality, the order between the context and the candidate follows that in the original passage. Furthermore, for (A)P+(A)N, we create and score one input sample for each of the context directions during prediction. The average of these two scores is taken as the final score. The maximum tokens of input is set as 256 in our experiments and only the context is truncated to meet this requirement. understanding the relationship between two sentences, which is highly correlated with our task. Therefore, we use the pre-trained BERT-Large-uncasedd with its NSP layer to predict the most appropriate candidate for each blank given its context. Specifically, BERT is employed to predict the probability of the context and the candidate being consecutive. Finetuning BERT A wide range of NLP tasks have greatly benefited from the pre-trained BERT model. Therefore, we also finetune the pre-trained BERT-Large model on our task through sequence pair classification schema. Specifically, for each blank, its correct candidate will be labelled as 0 and the label of all other wrong candidates is 1. Batch size and number of epochs for all models are 32 and 3. We employ Adam (Kingma and Ba, 2014) as the optimizer with three different learning rates { 1 e  5 , 2 e  5 , 3 e  5 } . Best model selection is based on validation performance. All BERT finetuning experiments including ablation study follow this training strategy. 1. INC : Answering each blank from left to right in order. Once a blank is answered with a candidate, this candidate is unavailable for later blanks. 2. EXH : Exhaustively scoring all permutations of candidates to answer the blanks. The score of a permutation is simply the sum of each its Type Model P N AP AN P+N AP+AN UNSUP BERT+I NC 33.0/2.1 34.7/4.1 29.8/2.1 15.7/0.3 34.7/2.3 27.3/1.4 +E XH 34.2/3.2 40.2/4.7 31.5/2.6 14.7/0.0 40.2/4.7 36.9/3.5 FT BERT+I NC 44.3/6.8 48.0/9.6 50.4/9.9 56.9/16.1 61.0/20.4 66.6/25.1 +E XH 47.2/8.5 54.2/11.2 60.0/17.5 60.0/17.5 66.5/25.2 71.7 / 29.9 SUP PMI 2 +I NC 23.4/1.2 24.4/1.5 16.2/0.3 17.5/0.1 26.2/1.7 17.1/0.0 +E XH 24.7/1.5 28.2/1.5 20.6/0.9 13.3/0.0 29.7/2.6 25.2/0.6 Table 6: Test BA/PA of various model types unsupervised (UNSUP), finetuned (FT) and supervised (SUP) across varying context levels, with INC or EXH decoding. We design two metrics to evaluate models. Both of these metrics are reported as percentage. Passage Accuracy (PA): PA is 1 iff the model gets all blanks in a passage correct, and 0 otherwise. The average of PA over all passages is reported. We hire annotators from AMT to both answer and label difficulty for 144 randomly chosen test examples. Annotators are restricted to be from USA/UK and have Master designation on AMT 8 , along with > 90 % HIT approval rate. On average, each annotator spends 624 seconds to answer one example. Difficulty level is chosen from { VeryHard , Hard , Moderate , Easy , VeryEasy } . 3 . 5 % of annotators find the task VeryHard , while 8 . 3% find it VeryEasy . The largest fraction of 38 . 2% find it to be Moderate . We note that SCDE contains a larger proportion of non-easy 8 Marked by AMT based on approval %, no. approved etc. questions (61.0%). Human performance is reported in Table 5. Annotators achieve BA of 87% which we take as the ceiling performance for models to match. All models are trained with AP+AN context and decoded by EXH 9 . Results are shown in Table 5. Finetuning BERT achieves the best performance among other models, though it still lags behind human performance significantly. Unsupervised models could only solve one third of all blanks. Surprisingly, PMI 2 and COHERE performs worse than the unsupervised models. We conjecture that it is difficult for COHERE, using syntactic regularities alone, to distinguish between the ground truth answer for a particular blank and another candidate which is a ground truth answer for another nearby blank. As noted, PMI 2 suffers due to inability of incorporating larger context. To explore effects of various context length and decoding strategies, models are trained with different context lengths and inferred by both decoding methods. Results are shown in Table 6. INC vs EXHEXH is better than INC for most approaches, indicating that human created blanks are interdependent and need joint answering. Context Length Increasing the context length, such as (P vs. AP), could significantly improve model performance, showing that this task needs discourse-level context to be answered. Furthermore, models with bidirectional context, such as (P+N), perform better than single-direction context, e.g., P, indicating that this task needs global context. Lastly, we observe that PMI-based approaches which do not explicitly encode sentences 9 Unless stated otherwise, models decode with EXH and are trained with full context i.e AP+AN Figure 1: Test blank accuracy of BERT-ft and Human on each reasoning type category introduced in  3.3. are unable to incorporate larger context levels, showing best performance with P+N. BERT after finetuning (BERT-ft) can perform reasonably well (72%) but there is still a gap comparing with human performance (87%). In this section, we would like to analyze the strength and weakness of BERT-ft compared with HUMAN . Therefore, we analyze their performance across different reasoning categories on test set. From Figure 1, inference questions are the most difficult for both HUMAN and BERT-ft and questions needing WordMatch are relatively easy. Compared with human performance, BERT-ft could achieve comparable BA on WordMatch and paraphrasing problems. However, BERT-ft performs much worse on questions needing inference and summary. We also refer to some examples from Table 4. In Example 4 , BERT-ft prefers A but the answer is C. The reason why BERT-ft chooses A may be that enjoy life happens in the context, but summarizing the next sentence is necessary to achieve the correct answer. Therefore, it is necessary to improve the ability of BERT to represent meaning at the sentence level beyond representing individual words in context. We also explore how the system performance corresponds to the human judgement of difficulty. Since evaluates rate the problems into 5 difficulty levels, we report the system BA/PA for each level in Table 8. For BA (blank-level accuracy), we see that, overall, the system accuracy decreases as difficulty increases from VeryEasy (0.75) to VeryHard (0.68). However, the decrease is not exactly monotonic (there is a small increase from VeryEasy to Easy, as also from Moderate to Hard). We conjecture that non-monotonicity could be due to two reasons:  Our difficulty annotations are at passage level rather than blank level. There might be some hard blanks in a passage marked overall Easy. Conversely, there might be easy blanks in a passage marked overall Hard. Since we've more examples marked with certain difficulty levels e.g 30.5% examples are Easy while only 8.3% are VeryEasy. This might make system accuracy average for levels with more examples more stable (lower sample variance), leading to some non-monotonicity (e.g for Easy and VeryEasy) For PA (passage-level accuracy, i.e., getting all questions correct) also, we see a clear decrease as difficulty increases from VeryEasy (0.63) to Very-Hard(0.2). The decrease here is sharper than BA , with only one violation of monotonicity (increase from 0.29 to 0.35 on Moderate to Hard). The sharper trend for PA supports our first point above. An attractive aspect of this task is distractors designed by English teachers. We verify distractor quality through the following experiments. Model Performance w/o Distractors All distractors in the test set are removed and models are evaluated on this non-distracted test set. Results are shown in Table 7. It is clear to see that after removing these distracting candidates, models can get better scores, showing that models find it hard to exclude distractors during prediction. Randomly Sampled Distractors After removing human-created distractors, we further randomly sample sentences from other passages as new distractors. To mitigate sampling variance, we run this experiment with 8 seeds and report the Model Uni. averaged score in Table 7. Comparing with distractors designed by teachers, models could discern these distractors more easily. Annotation artifacts of distractors Annotation artifacts (Gururangan et al., 2018) occurs in many datasets created by human annotators. A potential artifact type for our task is whether we could detect distractors without passages. Therefore, we finetune BERT-Large as a binary classifier, the input of which is just distractors and other correct candidates. With this model, we could only obtain 38% F1 score on the test set, showing that it is difficult to filter distractors out without any context. Distractor Error (DE) We define DE as the number of predicted answers per passage which are actually distractors. Through DE, we measure a model's ability to exclude distractors during prediction. Results are shown in Table 9. HUMAN has the lowest DE and BERT-ft could discern distractors to some extent. However, DE of PMI 2 is more than 1, meaning that on average, there is atleast one distractor in the predicted answer list. In summary, distractors created by teachers are high quality and increase task difficulty. To explore automatic generation of examples for the task, we construct sentence cloze questions by randomly choosing five sentences in a passage as blanks. We defer automatically generating distractors to future work since non-trivial distractor generation is a hard problem in itself. Specifically, we extract all passages from RACE (Lai et al., 2017) (which is also from exams) and filter out passages which have less than 10 sentences or more than 30 sentences. While choosing blank positions, we prevent three or more blanks consecutive to each other in generated questions. Finally, 16,706 examples are obtained automatically. Here, questions generated automatically and collected from examinations are called QA and QH respectively. We leverage QA in three ways: 1). train models only on QA , 2) first train models on QA and finetune models on QH , i.e., QA ; QH , 3) train models on the concatenation of QA and QH , i.e., QA + QH . BERT-Large is finetuned through these ways and results are shown in Table 10. The model trained only on QA has worst performance and we attribute this to the difficulty of distinguishing distractors without seeing them during training. Therefore, this model has the highest DE. However, models trained on QH and QA could achieve better performance. We conjecture this is because QA assists the model to have better generalization. We introduce SCDE, a sentence cloze dataset with high quality distractors carefully designed by English teachers. SCDE requires use of discourse-level context and different reasoning types. More importantly, the high quality distractors make this task more challenging. Human performance is found to exceed advanced contextual embedding and language models by a significant margin. Through SCDE, we aim to encourage the development of more advanced language understanding models. We thank Qizhe Xie, Hiroaki Hayashi and the anonymous reviewers for valuable comments.
method abstain method abstain abstain result result abstain abstain abstain abstain other objective objective objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain objective abstain abstain result result result other other other other other other other other other abstain other other other other other abstain other method other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain method abstain method method method method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain method abstain abstain method method abstain abstain other method abstain method method method abstain abstain method method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method abstain abstain abstain objective other