We aim to comprehensively identify all the event causal relations in a document, both within a sentence and across sentences, which is important for reconstructing pivotal event structures. The challenges we identified are two: 1) event causal relations are sparse among all possible event pairs in a document, in addition, 2) few causal relations are explicitly stated. Both challenges are especially true for identifying causal relations between events across sentences. To address these challenges, we model rich aspects of document-level causal structures for achieving comprehensive causal relation identification. The causal structures include heavy involvements of document-level main events in causal relations as well as several types of fine-grained constraints that capture implications from certain sentential syntactic relations and discourse relations as well as interactions between event causal relations and event coreference relations. Our experimental results show that modeling the global and fine-grained aspects of causal structures using Integer Linear Programming (ILP) greatly improves the performance of causal relation identification, especially in identifying cross-sentence causal relations. Understanding causal relations between events in a document is an important step in text understanding and is beneficial to various NLP applications, such as information extraction, question answering and text summarization. Causal relations can occur between any two events in a document, both between events within a sentence and between events across sentences. In this paper, we aim to identify all the event causal relations in a document. The main challenges for achieving comprehensive causal relation identification are that event Figure 1: An Example of Main Event Causal Structure causal relations are sparse among all the event pairs in a document and few event causal relations are explicitly stated. The challenges are especially true for identifying cross-sentence event causal relations and most of them have no clear causal indicators. To address these challenges, we model rich aspects of document-level causal structures, i.e., structural distributions of causal relations within a document, for achieving comprehensive causal relation identification in news articles. Our key observation for improving causal relation identification is that causal relations, especially cross-sentence causal relations, tend to involve one or two main events of a document. The main events are the focus of a story, which are usually mentioned in the title of an article and have repeated mentions throughout the document. Intuitively, causal relations in a document are often used to explain why the main events happened as well as consequences of the main events. For example, as shown in figure 1, killing is the main event. The events crossfire, spraying, richo-cheted, struck are its preconditions, and accuse, trial are its consequences. Indeed, many causal relations are related to the main event. In addition to the global causal structures related to main events of a document, we model three types of fine-grained causal structures in order to accurately identify each individual causal relation. First, specific sentential syntactic relations may evoke causal relations between event pairs. For instance, adverbial clause modifier of a verb phrase explains its consequence, condition or purpose. Second, we model implications of a discourse relation between two text units (e.g., the contingency discourse relation) towards causal relations between events in the two text units. Third, we model interactions between event causal relations and event coreference relations. For example, coreferent event mentions should have the same causal relations; a causal relation and an identity relation should not co-exist between any two events. We use Integer Linear Programming (ILP) to model these rich causal structures within a document by designing constraints and modifying the objective function to encourage causal relations akin to the observed causal structures and discourage the opposite. Our experimental results on the dataset EventStoryLine (Caselli and Vossen, 2017) show that modeling the global and fine-grained aspects of causal structures within a document greatly improves the performance of causal relation identification, especially in identifying cross-sentence causal relations. In the last decade or so, both unsupervised and supervised causal relation identification approaches have been proposed including linguistic patterns, statistical measures and supervised classifiers, primarily with the goal of acquiring event causality knowledge from a text corpus. The proposed approaches mainly rely on explicit contextual patterns (Girju; Hashimoto et al., 2014) or other causality cues (Riaz and Girju, 2010; Do et al., 2011), statistical associations between events (Beamer and Girju, 2009; Hu et al., 2017; Hu and Walker, 2017; Do et al., 2011; Hashimoto et al., 2014), and lexical semantics of events (Riaz and Girju, 2013, 2014b,a; Hashimoto et al., 2014). An increasing amount of recent works focused on recognizing event causal relations within a document, but mostly limited to identifying intra-sentence causal relations with explicit causal indicators. Mirza et al. (2014) annotated event causal relations in the TempEval-3 corpus and created CausalTimeBank. Mirza and Tonelli (2014) stated that incorporating temporal information improved the performance of a causal relation classifier. Mirza and Tonelli (2016) built both a rule-based multi-sieve approach and a feature based classifier to recognize causal relations in CausalTime-Bank. However, causal relations in CausalTime-Bank are few and only explicitly stated intra-sentence causal relations were annotated. In addition, Mostafazadeh et al. (2016) annotated both temporal and causal relations in 320 short stories (five sentences in each story) taken from the ROC-Stories Corpus and indicated strong correlations between causal relations and temporal relations. Lately, Caselli and Vossen (2017) created a corpus called EventStoryLine, which contains 258 documents and more than 5,000 causal relations. The EventStoryLine corpus is the largest dataset for causal relation identification till now with comprehensive event causal relations annotated, both intra-sentence and cross-sentence, which presents unique challenges for causal relation identification. Caselli and Vossen (2017) showed that only 117 annotated causal relations in this dataset are indicated by explicit causal cue phrases while the others are implicit. We conduct experiments on the EventStoryLine dataset. Distinguished from most of the previous approaches that identify one causal relation each time, we model coarse-grained and fine-grained document-level event causal structures and infer all the causal relations in a document. Integer linear programming (ILP) approaches have been applied to predict a set of temporal relations or an event timeline in a document (Do et al., 2012; Teng et al., 2016; Ning et al., 2017). ILP has been used to improve causal relation identification (Do et al., 2011), but only with fine-grained constraints considering discourse relations between two text units. Our approach innovates on modeling other aspects of document-level causal structures, especially heavy involvements of main events in causal relations, that facilitate resolving multiple causal relations. Table 1 shows the statistics of the corpus EventStoryLine v0.9 1 (Caselli and Vossen, 2017). Causal relations annotated in EventStoryLine are between two event mentions. Different causal relations are annotated in EventStoryLine, called rising action and falling action, which indicate the directions of causal relations and intuitively correspond to precondition and conse-quence relations. Note that in this paper, we focus on identifying all the pairs of events in a document that are causally related, but not on classifying the direction of a causal relation though; specifically, we aim to recognize if there exists a causal relation between any two events A and B in a document, but we do not further distinguish if A causes B vs. B causes A . On average, there is 1.2 event mentions in each sentence. There are 7,805 intra-sentence and 46,521 cross-sentence event mention pairs in total in the corpus, around 22% (1,770) and 8% (3,855) of them were annotated with a causal relation respectively. Out of the annotated causal links, only 117 Caselli and Vossen (2017) causal relations are indicated by explicit causal cue phrases while the others are implicit. In our experiments, we use the gold event mentions in EventStoryLine and exclude aspectual, causative, perception and reporting event mentions 2 , most of which were not annotated with any causal relation according to Caselli and Vossen (2017). Intraand cross-sentence causal relations are different by nature. For instance, dependency rela-1 Statistics are calculated based on latest release https://github.com/tommasoc80/EventStoryLine 2 639 event mentions were excluded in this way. tions between words in a sentence may be more useful for detecting intra-sentence causal relations, than when used for detecting cross-sentence causal relations. Therefore, we train two separate logistic regression classifiers, one for intra-sentence causal link detection and the other for cross-sentence causal link detection. We consider all event mention pairs within a sentence as training instances for the intra-sentence causal relation classifier. Then we pair event mentions from two sentences with one event mention from each sentence, which are used as training instances for the cross-sentence classifier. Note that training instances for both classifiers are unbalanced, with a POS:NEG ration of around 1:3 and 1:10 for intraand cross-sentence cases respectively. We applied the balanced class weight option 3 in logistic regression classifiers to deal with the class imbalance problem. Lexical Features : We implement rich lexical features to capture event word forms and similarities between two events, event modifiers and event arguments. First, we encode word and lemma for each token in two event phrases as features. Then we created various similarity features between two events. Similarities Based on Event Word Form Match. Three binary features indicating whether the lowercases of two event head words, two event head lemmas and two complete event phrases are exactly the same. Similarities Based on Wordnet. We first identify synsets for each event head word in Wordnet. Then for each pair of synsets, with one synset for each event head word, we calculate the Wup similarity (Wu and Palmer, 1994). We create numerical features using the average, minimal and maximal Wup similarities. Similarities Based on Word Embeddings. We apply l2 normalization on event head word embeddings, and then we calculate the Euclidean distance and Cosine distance between 3 http://scikit-learn.org/stable/modules/generated/ sklearn.linear model.LogisticRegression.html two word embeddings and use them as features. We use Glove Vectors (Pennington et al., 2014) for word embeddings. Similarities Based on Event Modifiers. We run the dependency parsing tool from the Stanford CoreNLP (Manning et al., 2014) and identify event modifiers as words that have a certain dependency relation 4 with an event head word. We measure the similarity between two events using the number of common modifiers and the number of common dependency relations that connect a modifier with an event head word. Similarities Based on Event Arguments. We consider entities that have a direct dependency relation with an event head word as its event arguments. We use the Stanford CoreNLP to identify entities and their types. We measure the similarity between two events using the number of common event arguments and the number of common entity types. Causal Potential Features : As inspired by the causal potential metric proposed by (Beamer and Girju, 2009), we encode features based on the point-wise mutual information (PMI) score and the relative textual order between two events. We calculate the PMI score of two event words in EventStoryLine by using co-occurrences of two events in one sentence, and we use the score as a numerical feature. Syntactic Features : We use dependency relations on the dependency path between two events. We use the basic dependencies extracted from Stan-fordCoreNLP (Manning et al., 2014). For cross-sentence event pairs, we consider the dependency path from each event to the root node in its own sentence in extracting dependency relations, following Cheng and Miyao (2017). In addition, we use Part Of Speech tags of two event head words as features. We observed that the cross-sentence causal relation classifier is usually not as capable as the intra-sentence classifier, probably due to less contextual evidence to rely on. Therefore, for cross-sentence event mention pairs that can be converted 4 Specifically, we consider 'nmod', 'amod', 'advmod', 'mark', 'aux', 'auxpass', 'expl', 'cc', 'cop', 'punct' to be modifiers. to intra-sentence cases through event coreference links, we use a heuristic method to improve causal relation prediction performance and replace the predictions from the cross-sentence classifier with the predictions from the intra-sentence classifier 5 , by using system predicted event coreference links. Note that two events may have more than one pair of mentions, one mention for each event, co-occur within one sentence, we will use the highest score produced by the intra-sentence classifier over all the event mention pairs. In addition, the score replacement procedure may change prediction scores of some intra-sentence event mention pairs as well. For instance, if one event mention has a coreferent mention within the same sentence that is closer to and is more clearly in a causal relation with the other event mention according to the intra-sentence classifier, and when paired up, the new event pair has received a higher score, then we will replace the score of the original event pair with the higher score. We implemented the within-document neural network based event coreference classifier as described in (Choubey and Huang, 2017a) and used the system to obtain event coreference links. Our Integer Linear Programming (ILP) system performs document level global inference for resolving all the intra-sentence and inter-sentence event causal relations in a document. Let p ij denotes confidence score from the corresponding local pairwise classifier for assigning a causal relation to the event pair ( i, j ) . Let  refer to the set of event mentions in a document, we formulate our basic ILP objective function with equation 1. We then augment the objective function with new objectives (equation 2) and add constraints to induce causal structures, including heavy involvements of main events (  M and  F ) in causal relations throughout the document, as well as fine-grained interactions of event causal relations with discourse relations (  D ), and event coreference 5 Note that we only conduct the score replacement when a score produced by the intra-sentence classifier is higher than the score produced by the cross-sentence classifier, which indicates that the intra-sentence classifier is more confident. relations(  C ) as well as syntactic structure constraints (  S ) for identifying causal relations. Main Event: Main events are central to the story in a document and tend to participate in multiple causal links. Similar to Choubey et al. (2018), we recognize main events based on characteristics of event coreference chains within a document. Specifically, we rank events based on the number of event mentions referring to an event, and choose the top two events as main events 6 . Then we add a new objective function (equation 3) and additional constraints to encourage causal links in event mention pairs containing a main event (equation 4) and discourage causal links in the remaining mention pairs (equation 5). M = max (cid:104) (cid:88) i   [ k m 1 m 1 ( i ) + k m 2 m 2 ( i )]  (cid:88) i     [ k n 1 n 1 ( i ) + k n 2 n 2 ( i )] (cid:105) (3)  i   , (cid:88) j  ,d i = d j x ij  m 1 ( i )  i   , (cid:88) j  ,d i (cid:54) = d j x ij  m 2 ( i ) (4)  i     , (cid:88) j     ,d i = d j x ij  n 1 ( i )  i     , (cid:88) j     ,d i (cid:54) = d j x ij  n 2 ( i ) (5) In the above equations,  denotes the set of main event mentions, and d i denotes the sentence number for event i . The independent variables m 1 ( i ) and m 2 ( i ) indicate the minimum number of intraand cross-sentence causal relations that main events participate in. By maximizing m 1 ( i ) and m 2 ( i ) in the objective function  M , our model favors main events to have more causal relations. Similarly, variables n 1 ( i ) and n 2 ( i ) in equation 5 are separately defined to set upper thresholds on the maximum number of intra-6 If there is a tie between two event clusters with the same number of coreferential event mentions, we use the sum of confidence scores for pairs of coreferential event mentions in a cluster to break the tie. The confidence scores were assigned by the local pairwise coreference relation classifier. and cross-sentence causal relations without a main event. Unlike m 1 ( i ) and m 2 ( i ) , we aim to minimize the variables n 1 ( i ) and n 2 ( i ) to restrict non-main events from participating in causal relations. Notice that we apply the constraints separately to intraand cross-sentence mention pairs. This is primarily because main events are likely to participate in many more cross-sentence causal relations compared to intra-sentence cases. Furthermore, we observe that a main event may trigger several consequent events which themselves are causally related. However, causal relations involving only non-main events are less likely to show transitivity. Therefore, we add the constraint 6 to ensure non-transitivity among causal relations with no main event. Locality Constraints: Main events may not always have the largest coreference chain size, and the position of an event mention provides another strong heuristics for identifying the main event (Upadhyay and Roth, 2016). In addition, the first sentence often summarizes the central context of story and are likely to describe foreground events (Grimes, 1975) that have causal relation with multiple other events. Therefore, we add an objective function (equation 7) and additional constraints (equation 8) to encourage causal relations that contain an event from the first sentence. S = max (cid:88) i  S k f b 1 ( i )  (cid:88) i   (cid:88) j   k f l ij  | d i  d j | (7)  i  F, (cid:88) j   x ij  b 1 ( i ) (8a)  < i, j >  M, x ij  l ij + 1 i/ { F }  1 j/ { F } (8b) where, F represents all the events in first sentence, independent variable b 1 ( i ) indicates the minimum number of causal relations that an event in F participates in, M represents the set of event mention pairs that can be mapped to the same sentence and l ij is a leakage variable that allows distant event mentions in F receiving a very high confidence value p ij to have a causal relation. Particularly, we encourage causal links between two event mentions that are in nearby sentences or can be mapped to the same sentence using coreference links 7 . By maximizing the variables b 1 ( i ) and minimizing the term l ij  | d i  d j | , we encourage event mentions in F complying with certain constraints to have more causal relations. Syntactic Relations: Specific sentential syntactic relations may evoke causal relations between event pairs. First, adverbial clause modifier of a verb phrase explains its consequence, condition or purpose; Second, nominal events mentioned as subject in the main clause presents an assertional structure that delivers foreground (Grimes, 1975) information which may have causal associations with other events; Third, non-finite verb events that share arguments and complement the main event of a sentence are likely to have causal associations with the main event. Therefore, we add an objective function (equa-tion 9) and additional constraints (equation 10) to encourage causal relations that contain a nominal event as subject or verb event that modifies its parent with advcl or xcomp dependency relations. Here, S represents event mentions that possess one of the above syntactic structures, independent variable b 2 ( i ) indicates the minimum number of causal relations that an event in S participates in. Note that equation 10(b) was modified from 8(b) and allows discounted optimization (with l ij ) for events in S that are mappable to the same or nearby sentences. S = max (cid:88) i  S k s b 2 ( i ) (9)  i  S, (cid:88) j   x ij  b 2 ( i ) (10a)  < i, j >  M, x ij  l ij + 1 i/ { F,S }  1 j/ { F,S } (10b) Discourse Relations: Note that the implications of discourse relations between two text units towards causal relations between events in the two text units have been discussed in the previous work (Do et al., 2011). In this work, we consider three types of discourse relations 8 . First, 7 Two event mentions are mappable if their respective coreferential event mentions co-occur in at least one sentence. 8 We use PDTB parser (Lin et al., 2014) to identify three discourse relations. two subtypes of the contingency discourse relation, namely cause and condition , strongly suggest that causal links exist between events in the two discourse units. On the contrary, the comparison discourse relation highlights semantic in-dependence between two discourse units, thus inhibits causal relations between events described in them. Third, all causal relations are inherently temporal. An event that causes another event must necessarily occur before or temporally overlap with the latter. Thus, clauses having one of these temporal discourse relations may also favor causal relations between events in them. We model the above three dependencies between discourse relations and causation through constraints 11 and the objective function 12. r = Contingency, (cid:88) i  arg 1 (cid:88) j  arg 2 x ij  1  r = Comparison, (cid:88) i  arg 1 (cid:88) j  arg 2 x ij  0  r = Temporal, (cid:88) i  arg 1 (cid:88) j  arg 2 x ij  T ( r ) (11)  D = max (cid:88) r = Temporal k t T ( r ) (12) Specifically, we enforce events in clauses with the contingency discourse relation to have at least one event pair with causal relation. Similarly, we inhibit a causal relation between any event pair in clauses with the comparison discourse relation. For events in clauses with a temporal discourse relation, we aim to maximize the number of causal relations without grounding it to any hard lower bound. Here, r denotes the discourse relation between two discourse arguments, arg 1 and arg 2 , and Temporal refers to the set of temporal discourse relations. We use the pre-trained PDTB discourse parser (Lin et al., 2014) to obtain discourse relations in a document. Event Coreference Relations: We model interactions between event causal relations and event coreference relations by adding constraints 13 and 14 and an objective function 15. C = max (cid:88) i   (cid:88) j   (cid:2) (cid:88) k    k c ( c 1 ( i, j, k ) + c 2 ( i, j, k )) (cid:3)  (1  k c )( c 3 ( i, j )) Here  represents the identity (coreference) relation. The constraint 13 ensures that causal relation and coreference relation are mutually exclusive, allowing some violations when p i,j is high. The constraints 14 along with the objective function 15 encourage coreferent event mentions to have a causal relation with the same other event. While this relation between causal and coreference relations is strictly true for gold standard data, we observed that these constraints make the system very sensitive to noise when using system predicted coreference links. Therefore, we added binary leakage variables c 1 ( i, j, k ) , c 2 ( i, j, k ) and c 3 ( i, j ) to relax these constraints.By maximizing the negative of leakage variables, we allow our model to overcome this instability. There are 22 topics in the EventStoryLine corpus. We put them in order based on their topic IDs and use documents in the last two topics as the development set. We trained the ILP system using the rest 20 topics and tuned parameters based on the system performance on the development set. We report experimental results by conducting 5-fold cross validation on the rest 20 topics. For event causal relation identification, we report precision, recall, and F1-score. The weighting parameters for constraints, including k m 1 , k m 2 , k n 1 , k n 2 , k f , k t , k c and k s , were first pre-set to be a small number 0 . 1 . We then conducted grid search and searched for the best value for each parameter over the range from 0 . 1 to 0 . 5 with a step size of 0 . 1 . The best values for the parameters are 0 . 2 , 0 . 1 , 0 . 1 , 0 . 5 , 0 . 2 , 0 . 1 , 0 . 1 , 0 . 2 respectively. Cheng and Miyao (2017): a dependency path based sequential neural network model that extensively models compositional meanings of the context between two event mentions for causal relation identification. This model was first used for identifying event temporal relations and has been shown effective in identifying both intraand cross-sentence temporal relations. Choubey and Huang (2017b): another dependency path based sequential neural network model that was first developed for identifying temporal relations between event mentions within a sentence. We make this model also work for cross sentence cases by merging the root nodes of two dependency trees associated with two separate sentences and extracting a dependency path connecting events across sentences. So far, there is no well recognized effective approach for causal relation identification within a document. We applied the above two models for causal relation identification considering that causal relations are closely related with certain temporal relations and a causal event must occur before or overlap with the consequence event. LR (Lexical): the same logistic regression classifier as our local pairwise classifier but using the lexical features only. LR (Full): our local pairwise classifier using the full set of features. + Score Replacement: our local pairwise classifier using the full set of features, with the heuristic score replacement procedure applied. The first section of table 2 shows the performance of baseline models on intraand cross-sentence causal relation identification. The model OP labels each event mention pair as causal and suffers from low precisions 9 , especially on identifying cross-sentence causal relations. The two dependency path based neural network model (Cheng and Miyao, 2017; Choubey and Huang, 2017b) do not perform effectively on identifying causal relations. The performance is especially poor on cross-sentence cases. The model LR (Lexical) improved the precision of causal relation identification but suf-9 The reason it did not achieve the 100% recall is that we did not consider reporting, causative, perception or aspectual events. fers from low recall. In contrast, the model LR (Causal Potential) improved the recall but suffers from low precision. The model LR (full) with rich lexical, semantic and syntactic features achieved the best trade-off between precision and recall. + Score Replacement significantly improves the recall and F1-score on identifying cross-sentence causal relations, which also slightly improves the recall of intra-sentence cases. But the precision of causal relation identification remains low, especially on cross-sentence cases. The second section of table 2 shows the performance of our ILP model after gradually adding each type of constraints. +Main Event Constraints shows the performance of the ILP system with constraints encouraging causal relations involving a main event. By modeling this aspect of document-level causal structures, the precision of cross-sentence causal relation identification was clearly improved by around 6.3%. With a small loss on recall, the F1-score was improved by 4.1%. Modeling this document-level causal structure also improves both precision and recall on identifying intra-sentence causal relations, but with a relatively small margin. Compared to the local pairwise model + Score Replacement , the overall F1-score improvement from using global main event constraints is statistically significant with p < 0.05 (Dietterich, 1998). +Locality Constraints strengthens the effects of modeling main events and further improved the performance of both crossand intra-sentence causal relation identification. Next, adding sentential syntactic structure based constraints ( +Syntactic Constraints ) recovered additional intra-sentence causal relations and cross-sentence causal relations as well due to score replacement, and improved their recall by 4.4% and 2.8% respectively with little or no drop on precision. Then, after adding discourse constraints ( +Discourse Constraints ), both precision and recall on intra-sentence causal relation identification were slightly improved while the performance on cross-sentence causal relation identification remains roughly the same, this is mainly due to the fact that few cross-sentence discourse relations were identified by the discourse parser we used. Finally, after adding conference constraints ( +Coreference Constraints ), the precision of cross-sentence causal relation identification was increased by 2.9%, with a small loss on recall, the F1-score was improved by 1.8%. Unsurprisingly, the overall performance on intra-sentence causal relation identification was not affected much by coreference constraints since event coreference relations often involve events across sentences. Compared to the model considering global constraints only (the line + Locality Constraints ), the overall F1-score improvement from using fine-grained causal structure constraints is statistically significant with p < 0.01. To sum up, by modeling the global and fine-grained aspects of causal structures, the performance of both intraand cross-sentence causal reFigure 2: F1-scores on documents with different lengths. The x-axis indicates the number of sentences a document has. The y-axis indicates the macro average F1-score of causal relation identification. lation identification was greatly improved by 3.9% and 7.5% in F1-score respectively. Compared to the local pairwise model + Score Replacement , the overall F1-score improvement from using both global main event constraints and fine-grained causal structure constraints is statistically significant with p < 0.002. Impact of Document Lengths Figure 2 shows performance comparisons of three models on documents with different lengths. The first impression is that causal relation identification becomes harder when documents are longer. If we look into the figure, the score replacement heuristic improves the performance of causal relation identification on medium-sized documents, but not on short ( < 4 sentences) or long ( > 10 sentences) documents. This may either due to little event coreference information for use in short documents or event coreference information becoming too noisy in long documents. Compared to the mixed effects of the score replacement heuristic, the ILP system improved the performance of causal relation identification consistently in documents of any length, through modeling rich document-level causal structures. We have presented an ILP system that collectively identifies all the causal relations within a document, both intraand cross-sentence causal relations, by modeling the global and fine-grained aspects of causal structures. In the future, we will continue to enrich document-level causal structures, e.g., by considering segment-wise topic layouts and rhetorical discourse structures. This work was partially supported by the National Science Foundation via NSF Award IIS-1755943. Disclaimer: the views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of NSF or the U.S. Government.
objective method abstain abstain abstain result abstain result abstain abstain objective abstain abstain result result abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain objective result other other other other other other other other other other other method method other other abstain other other other objective other other other method other other other method method method other method method method method abstain abstain abstain method method method abstain other method abstain method method abstain method method method method method method method method method abstain method abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain other abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain other method method abstain abstain abstain abstain abstain abstain abstain other method abstain method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method method abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain method method other other