The Emotion Cause Extraction (ECE) task aims to identify clauses which contain emotion-evoking information for a particular emotion expressed in text. We observe that a widely-used ECE dataset exhibits a bias that the majority of annotated cause clauses are either directly before their associated emotion clauses or are the emotion clauses themselves. Existing models for ECE tend to explore such relative position information and suffer from the dataset bias. To investigate the degree of reliance of existing ECE models on clause relative positions, we propose a novel strategy to generate adversarial examples in which the relative position information is no longer the indicative feature of cause clauses. We test the performance of existing models on such adversarial examples and observe a significant performance drop. To address the dataset bias, we propose a novel graph-based method to explicitly model the emotion triggering paths by leveraging the commonsense knowledge to enhance the semantic dependencies between a candidate clause and an emotion clause. Experimental results show that our proposed approach performs on par with the existing state-of-the-art methods on the original ECE dataset, and is more robust against adversarial attacks compared to existing models. 1 1 Introduction Instead of detecting sentiment polarity from text, recent years have seen a surge of research activities that identify the cause of emotions expressed in text (Gui et al., 2017; Cheng et al., 2017a; Rashkin et al., 2018; Xia and Ding, 2019; Kim and Klinger, 2018; Oberlander and Klinger, 2020). In a typical dataset for Emotion Cause Extract (ECE) (Gui 1 Our code can be accessed at https://github.com /hanqi-qi/Position-Bias-Mitigation-in-Emotion-Cause-Analysis et al., 2017), a document consists of multiple clauses, one of which is the emotion clause annotated with a pre-defined emotion class label. In addition, one or more clauses are annotated as the cause clause(s) which expresses triggering factors leading to the emotion expressed in the emotion clause. An emotion extraction model trained on the dataset is expected to classify a given clause as a cause clause or not, given the emotion clause. However, due to the difficulty in data collection, the ECE datasets were typically constructed by using emotion words as queries to retrieve relevant contexts as candidates for emotion cause annotation, which might lead to a strong positional bias (Ding and Kejriwal, 2020). Figure 1 depicts the distribution of positions of cause clauses relative to the emotion clause in the ECE dataset (Gui et al., 2016). Most cause clauses are either immediately preceding their corresponding emotion clauses or are the emotion clauses themselves. Existing ECE models tend to exploit such relative position information and have achieved good results on emotion cause detection. For example, The Relative Position Augmented with Dynamic Global Labels (PAE-DGL) (Ding et al., 2019), RNN-Transformer Hierarchical Network (RTHN) (Xia et al., 2019) and Multi-Attention-based Neural Network (MANN) (Li et al., 2019) all concatenate the relative position embeddings with clause semantic embeddings as the clause representations. We argue that models utilising clause relative positions would inherently suffer from the dataset bias, and therefore may not generalise well to unseen data when the cause clause is not in proximity to the emotion clause. For example, in a recently released emotion cause dataset, only 25-27% cause clauses are located immediately before the emotion clause (Poria et al., 2020). To investigate the degree of reliance of existing ECE models on clause relative positions, we propose a novel strategy to generate adversarial examples in which the relative position information is no longer the indicative feature of cause clauses. We test the performance of existing models on such adversarial examples and observe a significant performance drop. To alleviate the position bias problem, we propose to leverage the commonsense knowledge to enhance the semantic dependencies between a candidate clause and the emotion clause. More concretely, we build a clause graph, whose node features are initialised by the clause representations, and has two types of edges i.e., Sequence-Edge ( SEdge ) and Knowledge-Edge ( K-Edge ). A S-Edge links two consecutive clauses to capture the clause neighbourhood information, while a K-Edge links a candidate clause with the emotion clause if there exists a knowledge path extracted from the ConceptNet (Speer et al., 2017) between them. We extend Relation-GCNs (Schlichtkrull et al., 2018) to update the graph nodes by gathering information encoded in the two types of edges. Finally, the cause clause is detected by performing node (i.e., clause) classification on the clause graph. In summary, our contributions are three-fold:  We investigate the bias in the Emotion Cause Extraction (ECE) dataset and propose a novel strategy to generate adversarial examples in which the position of a candidate clause relative to the emotion clause is no longer the indicative feature for cause extraction. We develop a new emotion cause extraction approach built on clause graphs in which nodes are clauses and edges linking two nodes capture the neighbourhood information as well as the implicit reasoning paths extracted from a commonsense knowledge base between clauses. Node representations are updated using the extended Relation-GCN. Experimental results show that our proposed approach performs on par with the existing state-of-the-art methods on the original ECE dataset, and is more robust when evaluating on the adversarial examples. Position-insensitive Models. A more traditional line of research exploited structural representations of textual units relying on rule-based systems (Lee et al., 2010) or incorporated commonsense knowledge bases (Gao et al., 2015) for emotion cause extraction. Machine learning methods leveraged text features (Gui et al., 2017) and combined them with multi-kernel Support Vector Machine (SVM) (Xu et al., 2017). More recent works developed neural architectures to generate effective semantic features. Cheng et al. (2017b) employed LSTM models, Gui et al. (2017) made use of memory networks, while Li et al. (2018) devised a Convolutional Neural Network (CNN) with a co-attention mechanism. (Chen et al., 2018) used the emotion classification task to enhance cause extraction results. Position-aware Models. More recent methodologies have started to explicitly leverage the positions of cause clauses with respect to the emotion clause. A common strategy is to concatenate the clause relative position embedding with the candidate clause representation (Ding et al., 2019; Xia et al., 2019; Li et al., 2019). The Relative Position Augmented with Dynamic Global Labels (PAE-DGL) (Ding et al., 2019) reordered clauses based on their distances from the target emotion clause, and propagated the information of surrounding clauses to the others. Xu et al. (2019) used emotion dependent and independent features to rank clauses and identify the cause. The RNN-Transformer Hierarchical Network (RTHN) (Xia et al., 2019) argued there exist relations between clauses in a document and proposed to classify multiple clauses simultaneously. Li et al. (2019) proposed a Multi-Attention-based Neural Network (MANN) to model the interactions between a candidate clause and the emotion clause. The generated representations are fed to a CNN layer for emotion cause extraction. The Hierarchical Neural Network (Fan et al., 2019) aimed at narrowing the gap between the prediction distribution p and the true distribution of the cause clause relative positions. We first define the Emotion Cause Extraction (ECE) task here. A document D contains N clauses D = { C i } Ni =1 , one of which is annotated as an emotion clause CE with a pre-defined emotion class label, E w . The ECE task is to identify one or more cause clauses, C t , 1  t  N , that trigger the emotion expressed in CE . Note that the emotion clause itself can be a cause clause. We propose a Knowledge-Aware Graph (KAG) model as shown in Figure 2, which incorporates knowledge paths extracted from ConceptNet for emotion cause extraction. More concretely, for each document, a graph is first constructed by representing each clause in the document as a node. The edge linking two nodes captures the sequential relation between neighbouring clauses (called the Sequence Edge or S-Edge ). In addition, to better capture the semantic relation between a candidate clause and the emotion clause, we identify keywords in the candidate clause which can reach the annotated emotion class label by following the knowledge paths in the ConceptNet. The extracted knowledge paths from ConceptNet are used to enrich the relationship between the candidate clause and the emotion clause and are inserted into the clause graph as the Knowledge Edge or K-Edge . We argue that by adding the K-Edges , we can better model the semantic relations between a candidate clause and the emotion clause, regardless of their relative positional distance. In what follows, we will first describe how to extract knowledge paths from ConceptNet, then present the incorporation of the knowledge paths into context modelling, and finally discuss the use of Graphical Convolutional Network (GCN) for learning node (or clause) representations and the prediction of the cause clause based on the learned node representations. ConceptNet is a commonsense knowledge graph, which represents entities as nodes and relationship between them as edges. To explore the causal re-Bai Jinyue, an ordinary worker in XingTai Steel factory in HeBei province and the department leader replied to my mail when I found that my advice had been adopted I realized that I had made contributions to the country's development talked to the journalist with exicitment different departments, like the public security, have accepted his advice with a Thank You letter in his hands C2) C4) C3) C6 ) C7) C8) Since 27 years ago acceptance culture diffusion spread C1) C5) make better world happiness ConceptNet Figure 3: A document consisting of 8 clauses in the ECE dataset with extracted knowledge paths from the ConceptNet. lation between a candidate clause and the emotion clause, we propose to extract cause-related paths linking a word in the candidate clause with the annotated emotion word or the emotion class label, E w , in the emotion clause. More concretely, for a candidate clause, we first perform word segmentation using the Chinese segmentation tool, Jieba 2 , and then extract the top three keywords ranked by Text-Rank 3 . Based on the findings in (Fan et al., 2019) that sentiment descriptions can be relevant to the emotion cause, we also include adjectives in the keywords set. We regard each keyword in a candidate clause as a head entity , e h , and the emotion word or the emotion class label in the emotion clause as the tail entity , e t . Similar to (Lin et al., 2019), we apply networkx 4 to perform a depth-first search on the ConceptNet to identify the paths which start from e h and end at e t , and only keep the paths which contain less than two intermediate entities. This is because shorter paths are more likely to offer reliable reasoning evidence (Xiong et al., 2017). Since not all relations in ConceptNet are related to or indicative of causal relations, we further remove the paths which contain any of these four relations:  antonym ',  distinct from ',  not desires ', and  not capable of '. Finally, we order paths by their lengths in an ascending order and choose the top K paths as the result for each candidate-emotion clause pair 5 . An example is shown in Figure 3. The 5-th 2 https://github.com/fxsjy/jieba 3 We have also experimented with other keyword extraction strategies, such as extracting words with higher TFIDF values or keeping all words after removing the stop words. But we did not observe improved emotion cause detection results. 4 http://networkx.github.io/ 5 We set K to 15, which is the median of the number of paths between all the candidate-emotion clause pairs in our dataset. clause is annotated as the emotion clause and the emotion class label is  happiness '. For the keyword,  adopted ', in the first clause, we show two example paths extracted from ConceptNet, each of which links the word  adopted ' with  happiness '. One such a path is  adopted  related to  acceptance  has subevent  make better world  causes  happiness . As shown in Figure 2, there are four components in our model: a document encoding module, a context-aware path representation learning module, a GCN-based graph representation updating module, and finally a softmax layer for cause clause classification. Initial Clause/Document Representation Learning For each clause C i , we derive its representation, C i , by using a Bi-LSTM operating on its constituent word vectors, where each word vector w i  R d is obtained via an embedding layer. To capture the sequential relationship ( S-Edges ) between neighbouring clauses in a document, we feed the clause sequence into a transformer architecture. Similar to the original transformer incorporating the position embedding with the word embedding, we utilise the clause position information to enrich the clause representation. Here, the position embedding o i of each clause is concatenated with its representation C i generated by Bi-LSTM. We consider different ways for encoding position embeddings using either relative or absolute clause positions and explore their differences in the experiments section. In addition, we will also show the results without using position embeddings at all. Since the aim of our task is to identify the cause clause given an emotion clause, we capture the dependencies between each candidate clause and the emotion clause. Therefore, in the document context modelling, we consider the emotion clause  CE , generated in a similar way as  C i , as the query vector, and the candidate clause representation  C i as both the key and value vectors, in order to derive the document representation, D  R d . Context-Aware Path Representation In Section 3.1, we have chosen a maximum of K paths { p t } Kt =1 linking each candidate C i with the emotion clause. However, not every path correlates equally to the document context. Taking the document shown in Figure 3 as an example, the purple knowledge path is more closely related to the document context compared to the green path. As such, we should assign a higher weight to the purple path than the green one. We propose to use the document-level representation D obtained above as the query vector, and a knowledge path as both key and value vectors, in order to calculate the similarity between the knowledge path and the document context. For each pair of a candidate clause C i and the emotion clause, we then aggregate the K knowledge paths to derive the context-aware path representation s i  R d below: s i = K (cid:88) t =1  t p t  t = softmax ( DT p t (cid:80) Kj =1 DT p j ) (2) where D is the document representation, p t is the path representation obtained from Bi-LSTM on a path expressed as an entity-relation word sequence. Update of Clause Representations by GCN After constructing a clause graph such as the one shown in Figure 2(c), we update the clause/node representations via S-Edges and K-Edges . Only clauses with valid knowledge paths to the emotion clause are connected with the emotion clause node. After initialising the node (or clause) in the clause graph with  C i and the extracted knowledge path with s i , we update clause representation using an extended version of GCN, i.e. Relation-GCNs (aka. R-GCNs) (Schlichtkrull et al., 2018), which is designed for information aggregation over multiple different edges: h (cid:96) +1 i =  ( (cid:88) r R Ni (cid:88) j  N i 1 c i,r W (cid:96)r h (cid:96)j + W (cid:96) 0 h (cid:96)i ) (3) where W (cid:96)r h (cid:96)j is the linear transformed information from the neighbouring node j with relation r at the (cid:96) -th layer, W (cid:96)r  R d  d is relation-specific, N i is the set of neighbouring nodes of the i -th node, RN j is the set of distinct edges linking the current node and its neighbouring nodes. When aggregating the neighbouring nodes information along the K-Edge , we leverage the path representation s i to measure the node importance. This idea is inspired by the translation-based models in graph embedding methods (Bordes et al., 2013). Here, if a clause pair contains a possible reasoning process described by the K-Edge , then  h E   h i + s i holds. Otherwise,  h i + s i should be far away from the emotion clause representation  h E . 6 Therefore, we measure the importance of graph nodes according to the similarity between ( h i + s i ) and h E . Here, we use the scaled Dot-Attention to calculate the similarity e iE and obtain the updated node representation z i . where e E is { e iE } N  1 i =1 . d is the dimension of graph node representations, and N r k is a set of neighbours by the K-Edge . Then, we combine the information encoded in SEdge with z i as in Eq. 3, and perform a non-linear transformation to update the graph node representation h (cid:96) +1 i : h (cid:96) +1 i =  (cid:0) z (cid:96)i + (cid:88) j  N rsi ( W j h j ) (cid:1) (5) where N r s i is a set of i -th neighbours connected by the S-Edges . Cause Clause Detection Finally, we concatenate the candidate clause node h i and the emotion node representation h e generated by the graph, and apply a softmax function to yield the predictive class distribution  y i . We conduct a thorough experimental assessment of the proposed approach against several state-of-the-art models 7 . 6 Here, we do not consider the cases when the candidate clause is the emotion clause (i.e.,  h i =  h E ), as the similarity between  h E + s i and  h E will be much larger than the other pairs. 7 Training and hyper-parameter details can be found in Appendix A. Methods P (%) R (%) F1 (%) W/O Pos RB 67.47 42.87 52.43 EMOCause 26.72 71.30 38.87 Ngrams+SVM 42.00 43.75 42.85 Multi-Kernel 65.88 69.27 67.52 CNN 62.15 59.44 60.76 CANN 77.21 68.91 72.66 Memnet 70.76 68.38 69.55 W. Pos HCS 73.88 71.54 72.69 MANN 78.43 75.87 77.06 LambdaMART 77.20 74.99 76.08 PAE-DGL 76.19 69.08 72.42 RTHN 76.97 76.62 76.77 Our KAG 79.12 75.81 77.43 : w/o R-GCNs 73.68 72.76 73.14 : w/o K-Edge 75.67 72.63 74.12 : w/o S-Edge 76.34 75.46 75.88 Table 1: Results of different models on the ECE dataset. Dataset and Evaluation Metrics The evaluation dataset (Gui et al., 2016) consists of 2,105 documents from SINA city news. As the dataset size is not large, we perform 10-fold cross-validation and report results on three standard metrics, i.e. Precision (P), Recall (R), and F1-Measure, all evaluated at the clause level. Baselines We compare our model with the position-insensitive and position-aware baselines: RB (Lee et al., 2010) and EMOCause (Russo et al., 2011) are rules-based methods. Multi-Kernel (Gui et al., 2016) and Ngrams+SVM (Xu et al., 2017) leverage S upport V ector M achines via different textual feature to train emotion cause classifiers. CNN (Kim, 2014) and CANN (Li et al., 2018) are vanilla or attention-enhanced approaches. Memnet (Gui et al., 2017) uses a deep memory network to re-frame ECE as a question-answering task. Position-aware models use the relative position embedding to enhance the semantic features. HCS (Yu et al., 2019) uses separate hierarchical and attention module to obtain context and information. Besides that, PAE-DGL (Ding et al., 2019) and RTHN (Xia et al., 2019) use similar G lobal P rediction E mbedding ( GPE ) to twist the clauses' first-round predictions. MANN (Li et al., 2019) performs multi-head attention in CNN to jointly encode the emotion and candidate clauses. LambdaMART (Xu et al., 2019) uses the relative position, word-embedding similarity and topic similarity as emotion-related feature to extract cause. Table 1 shows the cause clause classification results on the ECE dataset. Two rule-based methods have poor performances, possibly due to their pre-defined rules. Multi-Kernel performs better than the vanilla SVM, being able to leverage more contextual information. Across the other three groups, the precision scores are higher than recall scores, and it is probably due to the unbalanced number of cause clauses (18.36%) and non-cause clauses (81.64%), leading the models to predict a clause as non-cause more often. Models in the position-aware group perform better than those in the other groups, indicating the importance of position information. Our proposed model outperforms all the other models except RHNN in which its recall score is slightly lower. We have also performed ablation studies by removing either K-Edge or S-Edge , or both of them (w/o R-GCNs ). The results show that removing the R-GCNs leads to a drop of nearly 4.3% in F1. Also, both the K-Edge and S-Edge contributes to emotion cause extraction. As contextual modelling has considered the position information, the removal of S-Edge leads to a smaller drop compared to the removal of K-Edge . In order to examine the impact of using the clause position information in different models, we replace the relative position information of the candidate clause with absolute positions. In the extreme case, we remove the position information from the models. The results are shown in Figure 4. It can be observed that the best results are achieved using relative positions for all models. Replacing relative positions using either absolution positions or no position at all results in a significant performance drop. In particular, MANN and PAE-DGL have over 50-54% drop in F1. The performance degradation is less significant for RTHN, partly due to its use of the Transformer architecture for context modeling. Nevertheless, we have observed a decrease in F1 score in the range of 20-35%. Our proposed model is less sensitive to the relative positions of candidate clauses. Its robust performance partly attributes to the use of (1) hierarchical contextual modeling via the Transformer structure, and (2) the K-Egde which helps explore causal links via commonsense knowledge regardless of a clause's 65.49 72.42 76.77 77.08 15.31 18.39 56.94 69.43 15.09 17.9 41.45 68.29 15 25 35 45 55 65 75 85 MANN PAE-DGL RTHN OURS F1(%) relative position absolute position no position Figure 4: Emotion cause extraction when using relative, absolute or no clause positional information. In recent years, there have been growing interests in understanding vulnerabilities of NLP systems (Goodfellow et al., 2015; Ebrahimi et al., 2017; Wallace et al., 2019; Jin et al., 2020). Adversarial examples explore regions where the model performs poorly, which could help understanding and improving the model. Our purpose here is to evaluate if KAG is vulnerable as existing ECE models when the cause clauses are not in proximity to the emotion clause.Therefore, we propose a principled way to generate adversarial samples such that the relative position is no longer an indicative feature for the ECE task. Generation of adversarial examples We generate adversarial examples to trick ECE models, which relies on swapping two clauses C r 1 and C r 2 , where r 1 denotes the position of the most likely cause clause, while r 2 denotes the position of the least likely cause clause. We identify r 1 by locating the most likely cause clause based on its relative position with respect to the emotion clause in a document. As illustrated in Figure 1, over half of the cause clauses are immediately before the emotion clause in the dataset. We assume that the position of a cause clause can be modelled by a Gaussian distribution and estimate the mean and variance directly from the data, which are, { ,  2 } = { 1 , 0 . 5445 } . The position index r 1 can then be sampled from the Gaussian distribution. As the sampled value is continuous, we round the value to its nearest integer: r 1  (cid:98) g (cid:101) , g (cid:118) Gaussian ( ,  2 ) . To locate the least likely cause clause, we propose to choose the value for r 2 according to the attention score between a candidate clause and the emotion clause. Our intuition is that if the emotion clause has a lower score attended to a candidate clause, then it is less likely to be the cause clause. We use an existing emotion cause extraction model to generate contextual representations and use the Dot-Attention (Luong et al., 2015) to measure the similarity between each candidate clause and the emotion clause. We then select the index i which gives the lowest attention score and assign it to r 2 : r 2 = arg min i {  i } Ni =1 ,  i = Dot-Att. where  C i is the representation of the i -th candidate clause,  CE is the representation of the emotion clause, and N denotes a total of N clauses in a document. Here, we use existing ECE models as different discriminators to generate different adversarial samples. 8 The desirable adversarial samples will fool the discriminator to predict the inverse label. We use leave-one-model-out to evaluate the performance of ECE models. In particular, one model is used as a Discriminator for generating adversarial samples which are subsequently used to evaluate the performance of other models. Results The results are shown in Table 2. The attacked ECE models are merely trained on the original dataset. The generated adversarial examples are used as the test set only. We can observe a significant performance drop of 23-32% for the existing ECE models, some of which even perform worse than the earlier rule-based methods, showing their sensitivity to the positional bias in the dataset. We also observe the performance degradation of our proposed KAG. But its performance drop is less significant compared to other models. The results verify the effectiveness of capturing the semantic dependencies between a candidate clause and the emotion clause via contextual and commonsense knowledge encoding. To understand how KAG aggregate information based on different paths, we randomly choose two examples to visualise the attention distributions (Eq. 4) on different graph nodes (i.e., clauses) in Figure 5. 9 These attention weights show the  distance ' between a candidate clause and the emotion clause during the reasoning process. The cause clauses are underlined, and keywords are in bold. C i in brackets indicate the relative clause position to the emotion clause (which is denoted as C 0 ). Ex.1 The crime that ten people were killed shocked the whole country ( C  4 ) . This was due to personal grievances ( C  3 ). Qiu had arguments with the management staff ( C  2 ), and thought the Taoist temple host had molested his wife ( C  1 ). He became angry ( C 0 ), and killed the host and destroyed the temple ( C 1 ). In Ex.1, the emotion word is  angry ', the knowledge path identified by our model from ConceptNet is,  arguments  fight  angry  for Clause C  2 , and  molest  irritate  exasperate  angry  for Clause C  1 . Our model assigns the same attention weight to the clauses C  2 , C  1 and the emotion clause, as shown in Figure 5. This shows that both paths are equally weighted by our model. Due to the K-Edge attention weights, our model can correctly identify both C  2 and C  1 clauses as the cause clauses. Ex.2 The LongBao Primary school locates between the two villages ( C  2 ). Some unemployed people always cut through the school to take a shortcut ( C  1 ). Liu Yurong worried that it would affect children's study ( C 0 ). When he did not have teaching duties ( C 1 ), he stood guard outside the school gate ( C 2 ). 9 More cases can be found in the Appendix. been assigned the largest attention weight as shown in Figure 5. Note that the path identified is spurious since the emotion of  worried ' is triggered by  unemployment ' in the ConceptNet, while in the original text,  worried ' is caused by the event,  Unemployed people cut through the school '. This shows that simply using keywords or entities searching for knowledge paths from commonsense knowledge bases may lead to spurious knowledge extracted. We will leave the extraction of event-driven commonsense knowledge as future work. In this paper, we examine the positional bias in the annotated ECE dataset and investigate the degree of reliance of the clause position information in existing ECE models. We design a novel approach for generating adversarial samples. Moreover, we propose a graph-based model to enhance the semantic dependencies between a candidate clause and a given emotion clause by extracting relevant knowledge paths from ConceptNet. The experimental results show that our proposed method achieves comparative performance to the state-of-the-art methods, and is more robust against adversarial attacks. Our current model extracts knowledge paths linking two keywords identified in two separate clauses. In the future, we will exploit how to incorporate the event-level commonsense knowledge to improve the performance of emotion cause extraction. This work was funded by the EPSRC (grant no. EP/T017112/1, EP/V048597/1). HY receives the PhD scholarship funded jointly by the University of Warwick and the Chinese Scholarship Council. YH is supported by a Turing AI Fellowship funded by the UK Research and Innovation (grant no. EP/V020579/1). We thank Yizhen Jia and Daoye Zhu for their valuable work on earlier code framework of this paper. We also thank the anonymous reviewers for their valuable comments.
abstain result abstain objective result objective objective abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result objective method abstain abstain abstain objective objective objective objective other other other other other other other other other other other other other other other objective other other other objective other other method other method objective other other objective objective abstain abstain abstain other method abstain other abstain other other objective other abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain abstain other abstain method abstain method method abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective objective method result other other other other other