Open-domain question answering remains a challenging task as it requires models that are capable of understanding questions and answers, collecting useful information, and reasoning over evidence. Previous work typically formulates this task as a reading comprehension or entailment problem given evidence retrieved from search engines. However, existing techniques struggle to retrieve indirectly related evidence when no directly related evidence is provided, especially for complex questions where it is hard to parse precisely what the question asks. In this paper we propose a retriever-reader model that learns to attend on essential terms during the question answering process. We build (1) an essential term selector which first identifies the most important words in a question, then reformulates the query and searches for related evidence; and (2) an enhanced reader that distinguishes between essential terms and distracting words to predict the answer. We evaluate our model on multiple open-domain multiplechoice QA datasets, notably performing at the level of the state-of-the-art on the AI2 Reasoning Challenge (ARC) dataset. Open-domain question answering (QA) has been extensively studied in recent years. Many existing works have followed the search-and-answer' strategy and achieved strong performance (Chen et al., 2017; Kwon et al., 2018; Wang et al., 2018b) spanning multiple QA datasets such as TriviaQA (Joshi et al., 2017), SQuAD (Rajpurkar et al., 2016), MS-Macro (Nguyen et al., 2016), ARC (Clark et al., 2018) among others. However, open-domain QA tasks become inherently more difficult when (1) dealing with questions with little available evidence; (2) solving  Most of the work was done during internship at Microsoft, Redmond. questions where the answer type is free-form text (e.g. multiple-choice) rather than a span among existing passages (i.e., answer span'); or when (3) the need arises to understand long and complex questions and reason over multiple passages, rather than simple text matching. As a result, it is essential to incorporate commonsense knowledge or to improve retrieval capability to better capture partially related evidence (Chen et al., 2017). As shown in Table 1, the TriviaQA, SQuAD, and MS-Macro datasets all provide passages within which the correct answer is guaranteed to exist. However, this assumption ignores the diffi-culty of retrieving question-related evidence from a large volume of open-domain resources, especially when considering complex questions which require reasoning or commonsense knowledge. On the other hand, ARC does not provide passages known to contain the correct answer. Instead, the task of identifying relevant passages is left to the solver. However, questions in ARC have multiple answer choices that provide indirect information that can help solve the question. As such an effective model needs to account for relations among passages, questions, and answer choices. Real-world datasets such as Amazon-QA (a corpus of user queries from Amazon) (McAuley and Yang, 2016) also exhibit the same challenge, i.e., the need to surface related evidence from which to extract or summarize an answer. Figure 1 shows an example of a question in the ARC dataset and demonstrates the difficulties in retrieval and reading comprehension. As shown for Choice 1 (C1), a simple concatenation of the 1 For SQuAD and TriviaQA, since the questions are paired with span-type answers, it is convenient to obtain ranking supervision where retrieved passages are relevant via distant supervision; however free-form questions in ARC and AmazonQA result in a lack of supervision which makes the problem more difficult. For MS-Macro, the dataset is designed to annotate relevant passages though it has free-form answers. question and the answer choice is not a reliable query and is of little help when trying to find supporting evidence to answer the question (e.g. we might retrieve sentences similar to the question or the answer choice, but would struggle to find evidence explaining why the answer choice is cor-rect). On the other hand, a reformulated query consisting of essential terms in the question and Choice 4 can help retrieve evidence explaining why Choice 4 is a correct answer. To achieve this, the model needs to (1) ensure that the retrieved evidence supports the fact mentioned in both the question and the answer choices and (2) capture this information and predict the correct answer. To address these difficulties, we propose an essential-term-aware Retriever-Reader (ET-RR) model that learns to attend on essential terms during retrieval and reading. Specifically, we develop a two-stage method with an essential term selector followed by an attention-enhanced reader. Essential term selector. ET-Net is a recurrent neural network that seeks to understand the question and select essential terms, i.e., key words, from the question. We frame this problem as a classification task for each word in the question. These essential terms are then concatenated with each answer choice and fed into a retrieval engine to obtain related evidence. Attention-Enhanced Reader. Our neural reader takes the triples (question, answer choice, retrieved passage) as input. The reader consists of a sequence of language understanding layers: an input layer, attention layer, sequence modeling layer, fusion layer, and an output layer. The attention and fusion layers help the model to obtain a refined representation of one text sequence based on the understanding of another, e.g. a passage representation based on an understanding of the question. We further add a choice-interaction module to handle the semantic relations and differences between answer choices. Experiments show that this can further improve the model's accuracy. We evaluate our model on the ARC Challenge dataset, where our model achieves an accuracy of 36.61% on the test set, and outperformed all leaderboard solutions at the time of writing (Sep. 2018). To compare with other benchmark datasets, we adapt RACE (Lai et al., 2017) and MCScript (Ostermann et al., 2018) to the open domain setting by removing their supervision in the form of relevant passages. We also consider a large-scale real-world open-domain dataset, Amazon-QA, to evaluate our model's scalability and to compare against standard benchmarks designed for the open-domain setting. Experiments on these three datasets show that ET-RR outperforms baseline models by a large margin. We conduct multiple ablation studies to show the effectiveness of each component of our model. Finally, we perform in-depth error analysis to explore the model's limitations. There has recently been growing interest in building better retrievers for open-domain QA. Wang et al. (2018b) proposed a Reinforced Ranker-Reader model that ranks retrieved evidence and assigns different weights to evidence prior to processing by the reader. Min et al. (2018) demonstrated that for several popular MRC datasets (e.g. SQuAD, TriviaQA) most questions can be answered using only a few sentences rather than the entire document. Motivated by this observation, they built a sentence selector to gather this potential evidence for use by the reader model. Nishida et al. (2018) developed a multi-task learning (MTL) method for a retriever and reader in order to obtain a strong retriever that considers certain passages including the answer text as positive samples during training. The proposed MTL framework is still limited to scenarios where it is feasible to discover whether the passages contain the answer span. Although these works have achieved progress on open-domain QA by improving the ranking or selection of given evidence, few have focused on the scenario where the model needs to start by searching for the evidence itself. Scientific Question Answering (SQA) is a representative open-domain task that requires capability in both retrieval and reading comprehension. In this paper, we study question answering on the AI2 Reasoning Challenge (ARC) scientific QA dataset (Clark et al., 2018). This dataset contains multiple-choice scientific questions from 3rd to 9th grade standardized tests and a large corpus of relevant information gathered from search engines. The dataset is partitioned into Chal-lenge and Easy sets. The challenge set consists of questions that cannot be answered correctly by either of the solvers based on Pointwise Mutual Information (PMI) or Information Retrieval (IR). Existing models tend to achieve only slightly better and sometimes even worse performance than random guessing, which shows that existing models are not well suited to this kind of QA task. Jansen et al. (2017) first developed a rule-based focus word extractor to identify essential terms in the question and answer candidates. The extracted terms are used to aggregate a list of potential answer justifications for each answer candidate. Experiments shown that focus words are beneficial for SQA on a subset of the ARC dataset. Khashabi et al. (2017) also worked on the problem of finding essential terms in a question for solving SQA problems. They published a dataset containing over 2,200 science questions annotated with essential terms and train multiple classifiers on it. Similarly, we leverage this dataset to build an essential term selector using a neural network-based algorithm. More recently, Boratko et al. (2018) developed a labeling interface to obtain high quality labels for the ARC dataset. One finding is that human annotators tend to retrieve better evidence after they reformulate the search queries which are originally constructed by a simple concatenation of question and answer choice. By feeding the evidence obtained by human-reformulated queries into a pre-trained MRC model (i.e., DrQA (Chen et al., 2017)) they achieved an accuracy increase of 42% on a subset of 47 questions. This shows the potential for a human-like retriever to boost performance on this task. Query reformulation has been shown to be effective in information retrieval (Lavrenko and Croft, 2001). Nogueira and Cho (2017) modeled the query reformulation task as a binary term selection problem (i.e., whether to choose the term in the original query and the documents retrieved using the original query). The selected terms are then concatenated to form the new query. Instead of choosing relevant words, Buck et al. (2018) proposed a sequence-to-sequence model to generate new queries. Das et al. (2019) proposed Multistep Retriever-Reader which explores an iterative retrieve-and-read strategy for open-domain question answering. It formulates the query reformulation problem in the embedding space where the vector representation of the question is changed to improve the performance. Since there is no supervision for training the query reformulator, all these methods using reinforcement learning to maximize the task-specific metrics (e.g. Recall for paragraph ranking, F1 and Exact Matching for span-based MRC). Different from these works, we train the query reformulator using an annotated dataset as supervision and then apply the output to a separate reader model. We leave the exploration of training our model end-to-end using reinforcement learning as future work. In this section, we introduce the essential-term-aware retriever-reader model (ET-RR). As shown in Figure 2, we build a term selector to discover which terms are essential in a question. The selected terms are then used to formulate a more efficient query enabling the retriever to obtain related evidence. The retrieved evidence is then fed to the reader to predict the final answer. For a question with q words Q = { w Qt } qt =1 along with its N answer choices C = { C n } Nn =1 where C n = { w C t } ct =1 , the essential-term selector chooses a subset of essential terms E  Q , which are then concatenated with each C n to formulate a query. The query for each answer choice, E + C n , is sent to the retriever (e.g. Elastic Search 2 ), and the top K retrieved sentences based on the scores returned by the retriever are then concatenated into the evidence passage P n = { w Pt } Kt =1 . Next, given these text sequences Q , C , and P = { P n } Nn =1 , the reader will determine a matching score for each triple { Q , C n , P n } . The answer choice C n  with the highest score is selected. We first introduce the reader model in Section 3.1 and then the essential term selector in Section 3.2. To simplify notation, we ignore the subscript n denoting the answer choice until the final output layer. In the input layer, all text inputsthe question, answer choices, and passages, i.e., retrieved evidenceare converted into embedded representations. Similar to Wang (2018), we consider the following components for each word: Word Embedding . embedding with dimensionality d w = 300 . Part-of-Speech Embedding and Named-Entity Embedding . The part-of-speech tags and named entities for each word are mapped to embeddings with dimension 16. 2 https://www.elastic.co/products/elasticsearch Relation Embedding . A relation between each word in P and any word in Q or C is mapped to an embedding with dimension 10. In the case that multiple relations exist, we select one uniformly at random. The relation is obtained by querying ConceptNet (Speer et al., 2017). Feature Embeddings . Three handcrafted features are used to enhance the word representations: (1) Word Match; if a word or its lemma of P exists in Q or C , then this feature is 1 (0 otherwise). (2) Word Frequency; a logarithmic term frequency is calculated for each word. (3) Essential Term; for the i -th word in Q , this feature, denoted as w e i , is 1 if the word is an essential term (0 otherwise). Let w e = [ w e 1 , w e 2 , . . . , w e q ] denote the essential term vector. For Q , C , P , all of these components are concatenated to obtain the final word representations WQ  R q  d Q , WC  R c  d C , WP  R p  d P , where d Q , d C , d P are the final word dimensions of Q , C , and P . As shown in Figure 2, after obtaining word-level embeddings, attention is added to enhance word representations. Given two word embedding sequences WU , WV , word-level attention is calculated as: M (cid:48) UV = WUU  ( WVV ) (cid:62) MUV = softmax( M (cid:48) UV ) WVU = MUV  ( WVV ) , (1) where U  R d U  d w and V  R d V  d w are two matrices that convert word embedding sequences to dimension d w , and M (cid:48) UV contains dot products between each word in WU and WV , and softmax is applied on M (cid:48) UV row-wise. Three types of attention are calculated using Equation (1): (1) question-aware passage representation WQ P  R p  d w ; (2) question-aware choice representation WQC  R c  d w ; and (3) passage-aware choice representation WPC  R c  d w . To model the contextual dependency of each text sequence, we use BiLSTMs to process the word representations obtained from the input layer and attention layer: H q = BiLSTM [ WQ ] H c = BiLSTM [ WC ; WPC ; WQC ] H p = BiLSTM [ WP ; WQP ] , (2) where H q  R q  l , H c  R c  l , and H p  R p  l are the hidden states of the BiLSTMs, ;' is feature-wise concatenation, and l is the size of the hidden states. We further convert each question and answer choice into a single vector: q  R l and c  R l : where the essential-term feature w e from Section 3.1.1 is concatenated with H q , and w sq and w sc are learned parameters. Finally, a bilinear sequence matching is calculated between H p and q to obtain a question-aware passage representation, which is used as the final passage representation:  p = softmax( H p  q ); p = H p (cid:62)  p . When a QA task provides multiple choices for selection, the relationship between the choices can provide useful information to answer the question. Therefore, we integrate a choice interaction layer to handle the semantic correlation between multiple answer choices. Given the hidden state H c n of choice c n and H c i of other choices c i ,  i (cid:54) = n , we calculate the differences between the hidden states and apply max-pooling over the differences: c inter = Maxpool( H c n  1 N  1 (cid:88) i (cid:54) = n H c i ) , (5) where N is the total number of answer choices. Here, c inter characterizes the differences between an answer choice c n and other answer choices. The final representation of an answer choice is updated by concatenating the self-attentive answer choice vector and inter-choice representation as c final = [ c ; c inter ] . For each tuple { q , p n , c n } Nn =1 , two scores are calculated by matching (1) the passage and answer choice and (2) question and answer choice. We use a bilinear form for both matchings. Finally, a softmax function is applied over N answer choices to determine the best answer choice: s pcn = p n W pc c final n ; s qcn = qW qc c final n s = softmax( s pc ) + softmax( s qc ) , (6) where s pcn , s qcn are the scores for answer choice 1  n  N ; s pc , s qc are score vectors for all N choices; and s contains the final scores for each answer choice. During training, we use a cross-entropy loss. Essential terms are key words in a question that are crucial in helping a retriever obtain related evidence. Given a question Q and N answer choices C 1 , . . . , CN , the goal is to predict a binary variable y i for each word Q i in the question Q , where y i = 1 if Q i is an essential term and 0 otherwise. To address this problem, we build a neural model, ET-Net, which has the same design as the reader model for the input layer, attention layer, and sequence modeling layer to obtain the hidden state H q for question Q . In detail, we take the question Q and the concatenation C of all N answer choices as input to Question If an object is attracted to a magnet, the object is most likely made of (A) wood (B) plastic (C) cardboard (D) metal # annotators 5 Annotation If,0; an,0; object,3; is,0; attracted,5; to,0; a,0; magnet,,5; the,0; object,1; is,0; most,0; likely,0; made,2; of,0 Table 2: Example of essential term data. ET-Net. Q and C first go through an input layer to convert to the embedded word representation, and then word-level attention is calculated to obtain a choice-aware question representation WCQ as in Equation (1). We concatenate the word representation and word-level attention representation of the question and feed it into the sequence modeling layer: H q = BiLSTM [ WQ ; WCQ ] . As shown in Figure 2, the hidden states obtained from the attention layer are then concatenated with the embedded representations of Q and fed into a projection layer to obtain the prediction vector y  R q for all words in the question: where w s contains the learned parameters, and W fQ is the concatenation of the POS embedding, NER embedding, relation embedding, and feature embedding from Section 3.1.1. After obtaining the prediction for each word, we use a binary cross-entropy loss to train the model. During evaluation, we take words with y i greater than 0.5 as essential terms. In this section, we first discuss the performance of the essential term selector, ET-Net, on a public dataset. We then discuss the performance of the whole retriever-reader pipeline, ET-RR, on multiple open-domain datasets.For both the ET-Net and ET-RR models, we use 96-dimensional hidden states and 1-layer BiLSTMs in the sequence modeling layer. A dropout rate of 0.4 is applied for the embedding layer and the BiLSTMs' output layer. We use adamax (Kingma and Ba, 2014) with a learning rate of 0.02 and batch size of 32. The model is trained for 100 epochs. Our code is released at https://github.com/ nijianmo/arc-etrr-code . We use the public dataset from Khashabi et al. (2017) which contains 2,223 annotated questions, each accompanied by four answer choices. Table 2 gives an example of an annotated question. As shown, the dataset is annotated for binary classification. For each word in the question, the data measures whether the word is an essential term according to 5 annotators. We then split the dataset into training, development, and test sets using an 8:1:1 ratio and select the model that performs best on the development set. Table 3 shows the performance of our essential term selector and baseline models from Khashabi et al. (2017). The second best model (ET Classifier) is an SVM-based model from Khashabi et al. (2017) requiring over 100 handcrafted features. As shown, our ET-Net achieves a comparable result with ET Classifier in terms of the F1 Score. Table 4 shows example predictions made by ET-Net. As shown, ET-Net is capable of selecting most ground-truth essential terms. It rejects certain words such as organisms which have a high TF-IDF in the corpus but are not relevant to answering a particular question. This shows its ability to discover essential terms according to the context of the question. We train and evaluate our proposed pipeline method ET-RR on four open-domain multiplechoice QA datasets. All datasets are associated with a sentence-level corpus. Detailed statistics are shown in Table 5. ARC (Clark et al., 2018): We consider the Challenge' set in the ARC dataset and use the provided corpus during retrieval. RACE-Open: We adapted the RACE dataset (Lai et al., 2017) to the open-domain setting. Originally, each question in RACE comes Example questions Which unit of measurement can be used to describe the length of a desk ? with a specific passage. To enable passage retrieval, we concatenate all passages into a corpus with sentence deduplication. 3  MCScript-Open: The MCScript (Ostermann et al., 2018) dataset is also adapted to the open-domain setting. Again we concatenate all passages to build the corpus. 4  Amazon-QA: The Amazon-QA dataset (McAuley and Yang, 2016) is an open-domain QA dataset covering over one million questions across multiple product categories. Each question is associated with a free-form answer. We adapt it into a 2-way multiple-choice setting by randomly sampling an answer from other questions as an answer distractor. We split all product reviews at the sentence-level to build the corpus. We consider three categories from the complete dataset in our experiments. In the experiments, ET-RR uses ET-Net to choose essential terms in the question. Table 6 shows example predictions on these target datasets. Then it generates a query for each of the N answer choices by concatenating essential 3 As short questions might not contain any words which can relate the question to any specific passage or sentence, we only keep questions with more than 15 words. terms and the answer choice. For each query, ET-RR obtains the top K sentences returned by the retriever and considers these sentences as a passage for the reader. We set K = 10 for all experiments. We compare ET-RR with existing retrieve-and-read methods on both datasets. As shown in Table 7, on the ARC dataset, ET-RR outperforms all previous models without using pre-trained models and achieves a relative 8.1% improvement over the second best BiLSTM Max-out method (Mi-haylov et al., 2018). Recently, finetuning on pre-trained models has shown great improvement over a wide range of NLP tasks. Sun et al. (2019) proposed a Reading Strategies' method to finetune the pre-trained model OpenAI GPT, a language model trained on the BookCorpus dataset (Rad-ford, 2018). They trained Reading Strategies on the RACE dataset to obtain more auxiliary knowledge and then finetune that model on the ARC corpus. Table 8 demonstrates the performance comparison of ET-RR and Reading Strategies on ARC. As shown, though Reading Strategies trained on both ARC and RACE dataset outperforms ET-RR, ET-RR outperforms Reading Strategies using only the ARC dataset at training time. On the RACE-Open and MCScript-Open datasets, ET-RR achieves a relative improvement of 24.6% and 10.5% on the test set compared with the second best method IR solver respectively. We also evaluate on multiple categories of Model ARC RACE-Open MCScript-Open Test Test Test IR solver 20.26 30.70 60.46 Random 25.02 25.01 50.02 BiDAF 26.54 26.89 50.81 BiLSTM Max-out 33.87 / / ET-RR (Concat) 35.33 36.87 66.46 ET-RR 36.61 38.61 67.71 Table 7: Accuracy on multiple-choice selection on ARC, RACE-Open and MCScript-Open. the Amazon-QA dataset. As shown in Table 9, ET-RR increases the accuracy by 10.33% on average compared to the state-of-the-art model Moqa (McAuley and Yang, 2016). We also compare ET-RR with ET-RR (Concat), which is a variant of our proposed model that concatenates the question and choice as a query for each choice. Among all datasets, ET-RR outperforms ET-RR (concat) consistently which shows the effectiveness of our essential-term-aware retriever. We investigate how each component contributes to model performance. Performance of reader. Our reader alone can be applied on MRC tasks using the given passages. Here, we evaluate our reader on the original RACE dataset to compare with other MRC models as shown in Table 10. As shown, the recently proposed Reading Strategies and OpenAI GPT models, that finetune generative pre-trained models achieve the highest scores. Among non-pre-trained models, our reader outperforms other Model Test ET-RR 36.61  inter-choice 36.36  passage-choice 35.41  question-choice 34.47  passage-question 34.05 Table 11: Ablation test on attention components of ET-RR on ARC. baselines: Bi-attn (MRU) (Tay et al., 2018) and Hierarchical Co-Matching (Wang et al., 2018a) by a relative improvement of 3.8%. Attention components. Table 11 demonstrates how the attention components contribute to the performance of ET-RR. As shown, ET-RR with all attention components performs the best on the ARC test set. The performance of ET-RR without passage-question attention drops the most signifi-cantly out of all the components. It is worth noting that the choice interaction layer gives a further 0.24% boost on test accuracy. Essential term selection. To understand the contribution of our essential-term selector, we compare ET-RR with two variants: (1) ET-RR (Concat) and (2) ET-RR (TF-IDF). For ET-RR (TF-IDF), we calculate the TF-IDF scores and take words with the top 30% of TF-IDF scores in the question to concatenate with each answer choice as a query. 5 Table 12 shows an ablation study comparing different query formulation methods and amounts of retrieved evidence K . As shown, with the essential term selector ET-Net, the model consistently outperforms other baselines, given different numbers of retrievals K . Performance for all models is best when K = 10 . Furthermore, only using TF-IDF to select essential terms in a question is not effective. When K = 10 , the ET-RR (TF-IDF) 5 According to the annotated dataset, around 30% of the terms in each question are labelled as essential. method performs even worse than ET-RR (Con-cat). This illustrates the challenges in understanding what is essential in a question. Though ET-RR consistently outperforms ET-RR (TF-IDF), the improvement is relatively modest on the Test set (around 1.4%). A similar outcome has been reported in Jansen et al. (2017); Khashabi et al. (2017) where essential term extraction methods have shown around 2%-4% gain compared with TF-IDF models and struggle to obtain further improvement on SQA tasks. This consensus might show the discrepancy of essential terms between human and machine (i.e., the essential terms obtained using a human annotated dataset might not be helpful in a machine inference model). Another reason might be the current retrieval method does not effectively use these essential terms and the performance highly depends on the dataset. Note that the ET-RR outperforms ET-RR (TF-IDF) by around 4% on the Dev set. Therefore, how to develop well-formed single or even multi-hop queries using these terms are worth studying in the future. Table 13 shows two major types of error, where the correct answer choice is in bold and the predicted answer choice is in italics . Retrieved supporting evidence but failed to reason over it. For the first question, there exists evidence that can justify the answer candidate (C). However, the model chooses (D) which has more words overlapping with its evidence. This shows that the model still lacks the reasoning capability to solve complex questions. Failed to retrieve supporting evidence. For the second question, the retrieved evidence of both the correct answer (D) and the prediction (B) is not helpful to solve the question. Queries such as what determines the year of a planet' are needed to acquire the knowledge for solving this question. We present a new retriever-reader model (ET-RR) for open-domain QA. Our pipeline has the following contributions: (1) we built an essential term selector (ET-Net) which helps the model understand which words are essential in a question leading to more effective search queries when retrieving related evidence; (2) we developed an attention-enhanced reader with attention and fusion among passages, questions, and candidate answers. Experimental results show that ET-RR outperforms existing QA models on open-domain multiplechoice datasets as ARC Challenge, RACE-Open, MCScript-Open and Amazon-QA. We also perform in-depth error analysis to show the limitations of current work. For future work, we plan to explore the directions of (1) constructing multihop query and (2) developing end-to-end retriever-reader model via reinforcement learning. We thank Jade Huang for proofreading the paper, Liang Wang and Daniel Khashabi for sharing code and the annotated dataset with us. We thank all the reviewers for their constructive suggestions.
abstain abstain abstain objective objective method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain objective objective abstain abstain abstain abstain abstain method abstain abstain method abstain result method method abstain result objective other other other other other other other other method other other other other other other other other other method other other other other other other other other other other other method method method method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method other method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method method method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain result objective other other