Understanding causality has vital importance for various Natural Language Processing (NLP) applications. Beyond the labeled instances, conceptual explanations of the causality can provide deep understanding of the causal facts to facilitate the causal reasoning process. However, such explanation information still remains absent in existing causal reasoning resources. In this paper, we fill this gap by presenting a human-annotated explainable CAusal REasoning dataset (e-CARE), which contains over 21K causal reasoning questions, together with natural language formed explanations of the causal questions. Experimental results show that generating valid explanations for causal facts still remains especially challenging for the state-of-the-art models, and the explanation information can be helpful for promoting the accuracy and stability of causal reasoning models. Causal reasoning is one of the most central cognitive abilities of human beings (Waldmann and Hagmayer, 2013; Jonassen et al., 2008), which enables one to understand the observed facts and predict the future. However, although recent causal reasoning models have achieved impressive performances on certain hand-crafted datasets, there still remains a considerable gap compared to human performances, as they cannot achieve stable performances across different datasets and are susceptible to adversarial attacks (McCoy et al., 2019; Poliak et al., 2018; Gururangan et al., 2018). One key factor leading to such drastic contrast is that, present causal reasoning models only learn to induce empirical causal patterns that are predictive to the label, while human beings seek for deep and conceptual understanding of the causality to explain the observed causal facts. The conceptual  Corresponding author Cause Effect Causal Relationship Conceptual Explanation Observed Causal Fact C1: Adding rock into hydrochloric acid E1: Rock dissolved Acid is corrosive Figure 1: Conceptual explanations of observed causality can be helpful for understanding the unseen causal facts. explanations can not only serve as a touchstone to examine whether the underlying causal mechanism has been thoroughly understood, but it can also in turn support the causal reasoning process. As illustrated in Figure 1, observing the causal fact C 1 : adding rock into hydrochloric acid causes E 1 : rock dissolved , one may further ask why such a causal relationship exists and reach the plausible conceptual explanation that Acid is corrosive , which goes beyond the isolated facts and reaches the conceptual nature to reveal the principle of the causal mechanism. However, despite the critical importance of conceptual explanations in causal reasoning, there is still a lack of such an explainable causal reasoning dataset. To fill this gap, we contribute an explainable CAusal REasoning dataset (e-CARE),together with a new causal explanation generation task, and a novel Causal Explanation Quality (CEQ) evaluation metric. The e-CARE dataset is constructed by crowd-sourcing and contains over 21K multiple-choice causal reasoning questions, which makes e-CARE the largest human-annotated commonsense causal reasoning dataset to the best of our knowledge. In addition to the causal reasoning question itself, e-CARE also provides a free-text-formed conceptual explanation for each causal question to explain why the causation exists. On this basis, we propose a new causal explanation generation task that requires models not only to choose the correct causal fact but also to generate the ex-432 planation for the choice. In addition, to directly measure the quality of generated explanations, we propose a novel causal explanation quality evaluation metric (namely, CEQ score). Compared to conventional text generation evaluation metrics such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) which mainly evaluate the textual or semantic similarity between generated explanations with golden annotations, CEQ score focuses on evaluating how much promotion an explanation can bring to understanding the causal mechanism. The dataset is publicly available at https: //github.com/Waste-Wood/e-CARE/ . Experimental results demonstrate that the causal questions of e-CARE are still challenging for the state-of-the-art (SOTA) pretrained language models, indicating the effectiveness of the e-CARE dataset in evaluating the causal learning ability of models. In addition, the explanation signal received in the training process can enhance the performance and the stability of the reasoning model, while the SOTA baselines still have trouble in explaining the causal facts at a conceptual level. These analyses highlight the importance of the conceptual explanations in causal reasoning, and suggest an avenue for future researches. Existing commonsense causal reasoning corpora differ in their annotation guidelines and how they are constructed: (1) whether the corpus is automatically constructed or built by human annotation; (2) whether the annotation unit of the corpus is word-level, phrase-level, or sentence-level. To obtain abundant causal knowledge, a natural way is extracting causal knowledge using heuristic rules from large-scale open-domain web text corpora (Luo et al., 2016; Li et al., 2020; Sap et al., 2019). However, the reporting bias may challenge both the coverage and quality of the extracted causal knowledge. Different from automatic construction, human annotation can endow datasets with higher precision. A line of work focuses on providing word-level causality knowledge (Girju et al., 2007; Mostafazadeh et al., 2016; Do et al., 2011; Hen-drickx et al., 2019). However, a word is not a complete semantic unit, which may limit the integrity of causal expressions and lead to ambi-Dataset Anno. guity. To address this issue, other datasets are constructed to provide phrase-level (Caselli and Vossen, 2017; Bethard and Martin, 2008; Mirza et al., 2014; Dunietz et al., 2017) and sentence-level (Ning et al., 2019; Roemmele et al., 2011) causal knowledge. Among these datasets, COPA (Roemmele et al., 2011) has become a widely adopted benchmark. Nevertheless, the size of COPA is rather limited, which may result in over-fitting and arouse concerns about the confidence of the results. In this paper, we introduce an e xplainable CAusal REasoning dataset (e-CARE). As shown in Table 1, to the best of our knowledge, e-CARE is the largest human-annotated causal reasoning dataset. With more than 21,000 instances, the e-CARE dataset can serve as a more reliable benchmark. Furthermore, compared to previous work, e-CARE can provide additional explanation information, which plays a critical role in learning the underlying mechanism of causal knowledge. Recently, an increasing amount of datasets have been proposed to address the explainability of textual inference tasks, such as textual entailment inference (Camburu et al., 2018), question-answering (QA) (DeYoung et al., 2019; Perez et al., 2019) and multi-hop QA (Ye et al., 2020). The form and content of the explanations vary with the nature of specific tasks. The QA task requires a model to answer the question based on evidences within given texts. Therefore, the explanation for this task should de-433 Number Train Dev Test Total Causal Questions 14,928 2,132 4,264 21,324 Uniq. scribe where and how an answer can be found (Wiegreffe and Marasovic, 2021). The explanations can have various forms, including answer-bearing sentences (Perez et al., 2019), structured information connecting the question and answer (Hancock et al., 2018; Ye et al., 2020), or even human-annotated free-formed sentences (Cam-buru et al., 2018; Rajani et al., 2019). In contrast, the multi-hop QA task requires the model to infer the correct answer through multiple reasoning steps. Hence, the explanation of this task needs to provide the specific reasoning paths (Wiegreffe and Marasovic, 2021; Jhamtani and Clark, 2020). Our work is quite different from previous work. We notice that all of these previous work only offer explanations that explain a specific question. Whereas we aim at providing a conceptual understanding of the causality, which has the potential to explain a set of related causal observations , rather than only explain a specific causal fact. e-CARE contains a total of 21,324 instances, corresponding to 13,048 unique explanations. This also makes e-CARE the largest human-annotated commonsense causal reasoning benchmark. The corpus-level statistics of the e-CARE dataset are shown in Table 2. As shown in Table 3, each instance of the e-CARE dataset is constituted by two components: (1) a multiple-choice causal reasoning question, composed of a premise and two hypotheses, and one of the hypotheses can form a valid causal fact with the premise; (2) a conceptual explanation about the essential condition that enables the existence of the causal fact. For example, as Table 3 shows, the explanation points out the nature of copper that Copper is a good thermal conductor , so that holding copper on fire will make fingers feel burnt immediately. The appendix provides more discussion about the explanations within e-CARE. On this basis, we introduce two tasks: Causal Reasoning Task We formulate the causal Premise : Tom holds a copper block by hand and heats it on fire. reasoning task as a multiple-choice task: given a premise event, one needs to choose a more plausible hypothesis from two candidates, so that the premise and the correct hypothesis can form into a valid causal fact. Explanation Generation Task It requires the model to generate a free-text-formed explanation for a given causal fact (composed of a premise and the corresponding correct hypothesis). To construct the e-CARE dataset, we start by collecting statements that describe conceptual understandings of world knowledge. Then given a statement, we ask different annotators to generate causal facts that can be explained by the statement, and build causal questions based on these causal facts. This is because we hope to provide conceptual explanations with more generality, that can explain a set of correlated causal facts, instead of only applicable to a certain isolated causal fact. Moreover, the statements can serve as clues to help the annotators to come up with causal facts. Collecting Potential Explanations Two key issues remain in collecting statements as potential explanations: (1) what kind of statements can be potential conceptual explanations of the causal facts; (2) where to find the appropriate statements. For the first question, Jonassen et al. (2008) concluded that, in general, the explanation of causality mainly describes three categories of information: (1) the nature or attributes of the objectives involved in the causal facts; (2) forces or actions that cause changes and drive transient motions; (3) the goals, intentions, motives or purposes of the causal agents. In addition, to be the conceptual explanation of a causal fact, the statement should be able to involve with a category of objects or people, but not only focus on a specific object or person (Sembugamoorthy and Chandrasekaran, 1986). Following these principles, we notice that there are already several available knowledge bases containing statements about such generic world knowledge, including ConceptNet (Speer 434 and Havasi, 2013), WordNet (Fellbaum, 2010), Atomic (Sap et al., 2019) and GenericsKB (Bhak-thavatsalam et al., 2020). However, ConceptNet and WordNet are structured knowledge graphs, containing only triplet-structured statements with a limited number of predicates. The scope of Atomic is limited in the activities of human beings. Compared to these knowledge bases, GenericsKB is an open-domain, large-scale knowledge base, containing rich generic world knowledge described in free-form text. Therefore, we collect the statements from GenericsKB to ensure the coverage and diversity of the potential explanations. Specifically, we filter out the statements in GenericsKB with low reliability, and the statements that may disobey the above-mentioned three principles. More details are provided in the Appendix. Thereafter, a total of 19,746 statements are left to form into a potential explanation set, which is further provided to the annotators to generate the causal questions. Annotating Causal Reasoning Questions Given the potential explanation set, annotators were recruited to generate corresponding causal questions. Specifically, a causal question is generated by two steps: First, an annotator was presented with a statement as a potential explanation, and was instructed to write a causal fact (composed of a cause and an effect), so that the causal fact can be interpreted by the given statement. In this step, a key issue is controlling the quality of generated causal facts. Thus we demonstrated illustrative examples to guide the annotators to avoid the following mistakes: (1) The created cause and effect are not in a valid causal relationship; (2) The created causal fact cannot be explained by the provided statement; (3) There are factual errors or imaginary contents in the created causal facts. In the causal fact generation process, each statement is randomly distributed to 1-3 annotators, so that we can find some statements that could explain multiple causal facts. Note that, in this process, we do not assume all statements are necessary to be a valid explanation. In other words, we do not require that the annotators must generate a causal fact for each given statement. Instead, we leave it to the judgment of annotators. In this way, the unreliable statements can be further excluded to promote the quality of our dataset. After the generation of causal facts, an ask-for indicator a  [cause, effect] was randomly generated, where a = cause (effect) means that the cause (effect) event is the hypothesis, and the effect (cause) event is the premise of the causal question, respectively. Then given the ask-for indicator, in order to control the grammar and writing style consistency, the same annotator was prompted to write a distract cause (effect) as the implausible hypothesis according to the ask-for indicator. In this process, the annotators were instructed to create the implausible hypothesis as close as possible to the true hypothesis, meanwhile prevent creating uninformative distractors (such as simply adding a not into the true hypothesis). A significant challenge in dataset construction is avoiding introducing superficial cues into the dataset (Gururangan et al., 2018; Poliak et al., 2018), which refers to the unintentional features that leak the label information. To address this issue, following Bhagavatula et al. (2019) and Sakaguchi et al. (2020), we employ an adversarial filtering algorithm to replace the implausible hypotheses that can easily be distinguished with the correct hypotheses using the superficial clues. More details about the adversarial filtering are provided in the Appendix. As Table 4 shows, after the adversarial filtering, without the existence of the premise, the SOTA pretrained language models can hardly distinguish two candidate hypotheses, which indicates that to predict the correct label, a model must understand the causal relationship between the premise and hypothesis, rather than only depend on the superficial cues within the two hypotheses. After the refinement, we evaluate the quality of the annotated causal questions and collected explanations through crowdsourcing. We assess the quality of causal questions by testing if there is agreement among human raters on the answer of causal questions. Specifically, we randomly sampled 200 causal questions from e-CARE, and en-435 listed 10 annotators to answer the causal questions. In this process, each causal question was evaluated by three annotators. When answering the causal questions, the raters were allowed to choose an additional option None of the above if neither hypothesis was deemed plausible. The human annotators achieve a 92% accuracy with a high agreement (Cohen's  = 0.935) (Cohen, 1960). To validate the quality of explanations, we enlisted volunteers to determine whether or not the explanations can explain corresponding causal facts. In total 200 causal facts with corresponding explanations were sampled and distributed to 10 volunteers, and each explanation was evaluated by three volunteers. After the evaluation, on average 89.5% of the explanations were deemed as valid (Cohen's  = 0.832), showcasing the quality of the explanations in e-CARE. A number of automatic scores have been proposed to evaluate the quality of generated explanations, such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004). However, these metrics evaluate the quality of the generated explanations only through comparing the textual or semantic similarity between the generated explanations and the golden annotation. Alternatively, an ideal causal explanation quality evaluation metric should directly measure if the causal fact is appropriately explained by the explanation. Hence, we propose a novel causal explanation quality evaluation metric (namely, CEQ score) as a step towards directly measuring the quality of generated explanations. We devise the CEQ score based on the consideration that a better explanation should provide more information for understanding the causality, so that the prediction model can more accurately estimate the reasonableness of the causal fact. Previous literature characterized such reasonableness as the causal strength of the given causal fact (Roemmele et al., 2011; Luo et al., 2016), where the causal strength is a score in [0 , 1] . Hence, in theory, for a valid causal fact, its causal strength should be equal to 1. Given a valid causal fact, an explanation should help to increase its estimated causal strength to the ground-truth value 1. generated explanation by measuring the increase of causal strength brought by the explanation. Specifically, let C , E , and X denote the cause, the effect and the generated explanation, respectively. Formally, the CEQ score is defined as: CEQ =  cs = cs( C, E | X )  cs( C, E ) , (1) where cs( C, E ) is the original causal strength between C and E ; cs( C, E | X ) is the causal strength after involvement of the additional explanation information. The explanation enhanced causal strength cs( C, E | X ) is defined as: cs( C, E | X ) = max [cs( C + X, E ) , cs( C, E + X )] , (2) where + denotes the string concatenate operation. Therefore, the CEQ score is positively related to the increase of causal strength between C and E after the involvement of the explanation X . In this paper, we employ a widely-adopted model-agnostic method proposed by Luo et al. (2016) to calculate the causal strength. The model-agnostic nature enable us to avoid reliance on certain models and keep the fairness of evaluation. Specifically, the phrase-level causal strength is derived through synthesizing the word-level causality. where ( CA , EB ) is an arbitrary causal fact; NCA and NEB are the number of words within CA and EB , respectively; cs( w i , w j ) is the causal strength between word w i and w j , which is estimated from a large corpus as: We examine the performance of state-of-the-art pretrained language models on the causal reasoning task and the explanation generation task. Furthermore, we investigate the specific role of explanations in causal reasoning by: (1) a predict-and-generate experiment, which requires models to conduct the causal reasoning task and generate corresponding explanations simultaneously; (2) a stability analysis using adversarial attacks. Settings We cast the causal reasoning task as a prediction problem: The input of the model is a candidate causal fact composed of a premise and one of the corresponding candidate hypotheses. The output is a score measuring the reasonableness of the candidate causal fact. We evaluate the causal reasoning ability of several SOTA pretrained language models, including discriminative pretrained language models BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019), and ALBERT (Lan et al., 2019); as well as autoregressive generative pretrained language models GPT2 (Radford et al., 2019) and BART (Lewis et al., 2020), which can also be adapted to the predictive causal reasoning task. In this section and the following parts, all experiments are conducted using the base-sized version of the pretrained language models. Additional details about experimental settings are provided in the Appendix. Results As shown in Table 5, ALBERT achieves the highest accuracy of 73.86% on the causal reasoning task of e-CARE. However, ALBERT can achieve an accuracy of 86.0% on the widely adopted causal reasoning benchmark COPA by our implementation. This is mainly because, on one hand, previous causal reasoning datasets are too small to evaluate the genuine reasoning ability of the model. On the other hand, previous datasets may provide some superficial cues for the reasoning models to achieve superb performances. In contrast, e-CARE is the largest causal reasoning dataset that can provide enough test instances to evaluate the actual ability of the model. More-Corr. over, in the annotating process of e-CARE, we introduced an adversarial filtering process to avoid the influence of superficial cues on the performances of reasoning models. Hence, we believe that e-CARE dataset can serve as a new benchmark for effectively evaluating models' causal reasoning ability. We also notice that human beings can achieve an accuracy of 92.00% on the e-CARE dataset. The large gap between the human performance and the pretrained language models suggests that the causal reasoning questions provided in our dataset still remain challenging, and calls for more powerful causal reasoning models. We investigate whether the model can generate correct explanations for given valid causal facts by training a GRU-based Seq2Seq model (Chung et al., 2014), and finetuning a generative pretrained language model GPT2 (Radford et al., 2019) on the e-CARE dataset. Both models take the concatenation of the cause and effect as input. Please refer to the Appendix for more details. Evaluation Metrics We automatically evaluate the quality of generated explanations using average-BLEU (n=4) (Papineni et al., 2002), ROUGE-l (Lin, 2004), Perplexity (Horgan, 1995), together with our proposed CEQ score. Human Evaluation We also assess the quality of model-generated explanations through human evaluation. Specifically, we sampled 200 explanations generated by each method. Then three workers were shown with the generated explanations, together with corresponding causal facts, and were asked to label whether the generated explanation can explain the corresponding causal fact. Quantitative Results As shown in Table 6, 89.5% of human-written explanations are found to be valid, while the generative pretrained language 437 Causal Facts (Generated) Explanation Human Annotation CEQ Cause: He was infected with gram-positive bacteria. model GPT2 only achieves a correctness of 20.0%. The last row of Table 6 reports the score of held-out human-written explanations, which serves as a ceiling for model performance. The significant gap indicates that, although GPT2 can achieve impressive performance on various natural language generation tasks, it still remains especially challenging for GPT2 to deeply understand the causal facts and then generate explanations like human beings. This may be one of the main obstacles hindering the further improvement of present causal reasoning models. Moreover, we measure the similarity between the automatic scores with the results of human evaluation using the Spearman correlation coefficient. As Table 7 shows, ROUGH-l and average-BLEU barely have a correlation with the results of human evaluation. This is because average-BLEU and ROUGH-l only implicitly evaluate the quality of generated explanations by measuring the textual similarity with the golden annotations. Compared to average-BLEU and ROUGH-l, the CEQ score has a significant positive relationship with the human evaluation results. This indicates the efficiency of the CEQ score in evaluating the quality of generated explanations. Qualitative Analysis In Table 8, we provide examples of explanations generated by GPT2. We observe that GPT2 can generate a reasonable explanation for some causal facts, while the generated explanations may still contain factual mistakes, or be totally irrelevant to the given causal fact (highlighted in yellow and pink, respectively). This indicate that the explanation generation still remains challenging for the GPT2 model. To investigate the role of causal explanations in the causal reasoning process, we trained models to jointly conduct these two tasks. Settings Since this task requires a model to predict a label meanwhile generate an explanation, we conduct the experiments using the GPT2 model, which can be adapted to conduct the predictive causal reasoning task and explanation generation simultaneously. We denote this multi-task finetuned GPT2 model as GPT2 CR-GE . Details for training GPT2 CR-GE is provided in the Appendix. To make the performance comparable, when evaluating the performance of GPT2 CR-GE on the causal expatiations generation task, the same as the settings in the explanation generation task, the premise and the correct hypothesis are taken as the input of GPT2 CR-GE for generating explanations. Results We measure the quality of generated explanations using the same automatic scores and human evaluation settings as the Explanation Generation experiment. The performance of causal reasoning is also measured using accuracy. The results are shown in Table 9, where GPT2 CR denotes the GPT2 model finetuned for the causal reasoning task, and GPT2 EG refers to the GPT2 model finetuned for the explanation generation task. We observe that compared with GPT2 CR , the improved performance of GPT2 CR-EG on causal reasoning indicates that the additional explanation can be helpful for the causal reasoning task, as it prompts model to have a deep understanding of the causal mechanisms. Interestingly, by comparing with GPT2 EG and GPT2 CR-EG , we find that learning to predict the label can also be helpful for the explanation generation process. This indicates the 438 synergistic effect of the causal reasoning and the explanation generation on promoting models' understanding of causal mechanism. Previous studies indicate that models may utilize some superficial cues within the dataset to predict the label. This leads to the vulnerability of models when facing adversarial attacks (Poliak et al., 2018; McCoy et al., 2019). Learning to generate the additional conceptual explanation may promote the understanding of causality to increase the stability of the reasoning model. Hence, we conduct a stability analysis to examine the specific effect of additional explanations. Following Bekoulis et al. (2018) and Yasunaga et al. (2018), we attack the causal reasoning system by adding a perturbation term on the word em-beddings of inputs. The perturbation term is derived using the gradient-based FGM method (Miy-ato et al., 2016). Table 9 shows the change of causal reasoning accuracy (  Accu.) brought by the adversarial attack. For example,  =  6 . 40 means a 6.40% decrease of prediction accuracy after the adversarial attack. We find that, compared to the vanilla GPT2 CR model, the explanation enhanced GPT2 model GPT2 CR-EG demonstrates stronger stability. This suggests that, by training reasoning models to generate correct explanations of the causal facts, the understanding of the causality can be promoted, and then the stability of model performance can be increased. Causal knowledge is critical for various NLP applications. In this section, we investigate if the causality knowledge provided by e-CARE can be used as a resource to boost model performance on other causal-related tasks. To this end, we apply transfer learning by first finetuning a BERT model on e-CARE, then adapting the e-CARE-enhanced model (denoted as BERTE ) on a causal extraction task EventStoryLine 0.9 (Caselli and Vossen, 2017), two causal reasoning tasks BECauSE 2.0 (Dunietz et al., 2017) and COPA (Roemmele et al., 2011), as well as a commonsense reasoning dataset CommonsenseQA (Tal-mor et al., 2019). On the EventStoryLine 0.9 dataset, we conduct experiment only on the instances about within-sentence causal relationship. The results are shown in Table 10. We observe Dataset Metric BERT BERTE EventStoryLine 0.9 F1 (%) 66.5 68.1 BECauSE 2.1 Accu. that the additional training process on e-CARE can consistently increase the model performance on all four tasks. This indicates the potential of e-CARE in providing necessary causality information for promoting causal-related tasks in multiple domains. In this paper, we introduce additional explanation information for the causal reasoning process, and propose a corresponding explanation generation task. Previous literature concluded the explanation generation process as an abductive reasoning process (Hanson, 1958; Peirce, 1974) and highlighted the importance of the abdutive explanation generation, as it may interact with the causal reasoning process to promote the understanding of causal mechanism, and increase the efficiency and reliability of causal reasoning. For example, as Figure 2 shows, one may have an observation that C 1 : adding rock into hydrochloric acid caused E 1 : rock dissolved . Through abductive reasoning, one may come up with a conceptual explanation for the observation that acid is corrosive . After that, one can confirm or rectify the explanation by experiments, or resorting to external references. In this way, new ideas about causality can be involved for understanding the observed causal fact. Then if the explanation is confirmed, it can be further utilized to support the causal reasoning process by helping to explain and validate other related causal facts, 439 such as C 2 : adding rust into sulphuric acid may lead to E 2 : rust dissolved. This analysis highlights the pivotal role of conceptual explanation in learning and inferring causality. In this paper, we introduce the e-CARE dataset to provide causal explanations and support future research towards stronger human-like causal reasoning systems. In this paper, we present an e xplainable CAusal REeasoning dataset e -CARE, which contains over 21K causal questions, together with over 13K unique conceptual explanations about the deep understanding of the causal facts, which also makes e-CARE the largest causal reasoning benchmark. Experimental results show that both the causal reasoning task and especially the explanation generation task remain challenging for the SOTA pretrained language models. Moreover, the additional explanation signal can promote both the prediction accuracy and stability of models, highlighting the vital importance of the conceptual explanations in causal reasoning. We thank the anonymous reviewers for their constructive comments, and gratefully acknowledge the support of the New Generation Artificial Intelligence of China (2020AAA0106501), and the National Natural Science Foundation of China (62176079, 61976073).
abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain objective objective abstain other abstain abstain abstain other other other other other other other other other other abstain abstain other other other other other other other other other other abstain abstain objective other other other other other other method other other method method method abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain other