Complex question answering over knowledge base (Complex KBQA) is challenging because it requires various compositional reasoning capabilities, such as multi-hop inference, attribute comparison, set operation. Existing benchmarks have some shortcomings that limit the development of Complex KBQA: 1) they only provide QA pairs without explicit reasoning processes; 2) questions are poor in diversity or scale. To this end, we introduce KQA Pro, a dataset for Complex KBQA including ~120K diverse natural language questions. We introduce a compositional and interpretable programming language KoPL to represent the reasoning process of complex questions. For each question, we provide the corresponding KoPL program and SPARQL query, so that KQA Pro serves for both KBQA and semantic parsing tasks. Experimental results show that SOTA KBQA methods cannot achieve promising results on KQA Pro as on current datasets, which suggests that KQA Pro is challenging and Complex KBQA requires further research efforts. We also treat KQA Pro as a diagnostic dataset for testing multiple reasoning skills, conduct a thorough evaluation of existing models and discuss further directions for Complex KBQA. Our codes and datasets can be obtained from https://github.com/shijx12/ KQAPro_Baselines . Thanks to the recent advances in deep models, especially large-scale unsupervised representation learning (Devlin et al., 2019), question answering of simple questions over knowledge base (Simple KBQA), i.e., single-relation factoid questions (Bor-des et al., 2015), begins to saturate (Petrochuk and Zettlemoyer, 2018; Wu et al., 2019; Huang et al., 2019). However, tackling complex questions (Complex KBQA) is still an ongoing challenge, due to the unsatisfied capability of compositional reasoning. As shown in Table 1, to promote the community development, several benchmarks are proposed for Complex KBQA, including LC-QuAD2.0 (Dubey et al., 2019), ComplexWebQuestions (Talmor and Berant, 2018), MetaQA (Zhang et al., 2018), CSQA (Saha et al., 2018), CFQ (Key-sers et al., 2020), and so on. However, they suffer from the following problems: 1) Most of them only provide QA pairs without explicit reasoning processes, making it challenging for models to learn compositional reasoning. Some researchers try to learn the reasoning processes with reinforcement learning (Liang et al., 2017; Saha et al., 2019; Ansari et al., 2019) and searching (Guo et al., 2018). However, the prohibitively huge search space hinders both the performance and speed, especially when the question complexity increases. For example, Saha et al. (2019) achieved a 96.52% F1 score on simple questions in CSQA, whereas only 0.33% on complex questions that require comparative count. We think that intermediate supervision is needed for learning the compositional reasoning, mimicking the learning process of human beings (Holt, 2017). 2) Questions are not satisfactory in diversity and scale. For example, MetaQA (Zhang et al., 2018) questions are generated using just 36 templates, and they only consider relations between entities, ignoring literal attributes; LC-QuAD2.0 (Dubey et al., 2019) and ComplexWebQuestions (Talmor and Berant, 2018) have fluent and diverse human-written questions, but their scale is less than 40K. To address these problems, we create KQA Pro , a large-scale benchmark for Complex KBQA. In KQA Pro, we define a Knowledge-oriented Programming Language ( KoPL ) to explicitly describe 6101 Question 1: SPARQL: KoPL: When did Cleveland Cavaliers pick up LeBron James SELECT DISTINCT the reasoning process for solving complex questions (see Fig. 1). A program is composed of symbolic functions , which define the basic, atomic operations on KBs. The composition of functions well captures the language compositionality (Ba-roni, 2019). Besides KoPL, following previous works (Yih et al., 2016; Su et al., 2016), we also provide the corresponding SPARQL for each question, which solves a complex question by parsing it into a query graph. Compared with SPARQL, KoPL 1) provides a more explicit reasoning process. It divides the question into multiple steps, making human understanding easier and the intermediate results more transparent; 2) allows humans to control the model behavior better, potentially supporting human-in-the-loop. When the system gives a wrong answer, users can quickly locate the error by checking the outputs of intermediate functions. We believe the compositionality of KoPL and the graph structure of SPARQL are two complementary directions for Complex KBQA. To ensure the diversity and scale of KQA Pro, we follow the synthesizing and paraphrasing pipeline in the literature (Wang et al., 2015a; Cao et al., 2020), first synthesize large-scale (canonical question, KoPL, SPARQL) triples, and then paraphrase the canonical questions to natural language questions (NLQs) via crowdsourcing. We combine the following two factors to achieve diversity in questions: (1) To increase structural variety, we leverage a varied set of templates to cover all the possible queries through random sampling and recursive composing; (2) To increase linguistic variety, we filter the paraphases based on their edit distance with the canonical utterance. Finally, KQA Pro consists of 117,970 diverse questions that involve varied reasoning skills ( e.g. , multi-hop reasoning, value comparisons, set operations, etc. ). Besides a QA dataset, it also serves as a semantic parsing dataset. To the best of our knowledge, KQA Pro is currently the largest corpus for NLQ-to-SPARQL and NLQ-to-Program tasks. We reproduce the state-of-the-art KBQA models and thoroughly evaluate them on KQA Pro. From the experimental results, we observe significant performance drops of these models compared with on existing KBQA benchmarks. It indicates that Complex KBQA is still challenging, and KQA Pro could support further explorations. We also treat KQA Pro as a diagnostic dataset for analyzing a model's capability of multiple reasoning skills, and discover weaknesses that are not widely known, e.g. , current models struggle on comparisonal reasoning for lacking of literal knowledge ( i.e. , (Le-Bron James, height, 206 centimetre) ), or perform poorly on questions whose answers are not obver-seved in the training set. We hope all contents of KQA Pro could encourage the community to make further breakthroughs. Complex KBQA aims at answering complex questions over KBs, which requires multiple reasoning capabilities such as multi-hop inference, quantitative comparison, and set operation (Lan et al., 2021). Current methods for Complex KBQA can be grouped into two categories: 1) semantic parsing based methods (Liang et al., 2017; Guo et al., 2018; Saha et al., 2019; Ansari et al., 2019), which parses a question to a symbolic logic form ( e.g. ,  calculus (Artzi et al., 2013),  -DCS (Liang, 2013; Pasupat and Liang, 2015; Wang et al., 2015b; Pasupat and Liang, 2016), SQL (Zhong et al., 2017), AMR (Banarescu et al., 2013), SPARQL (Sun et al., 2020), and etc. ) and then executes it against the KB and obtains the final answers; 2) information retrieval based methods (Miller et al., 2016; Saxena 6102 Dataset multiple kinds of knowledge number of questions naturallanguage querygraphs multi-stepprograms WebQuestions (Berant et al., 2013)  5,810    WebQuestionSP (Yih et al., 2016)  4,737    GraphQuestions (Su et al., 2016)  5,166    LC-QuAD2.0 (Dubey et al., 2019)  30,000    ComplexWebQuestions (Talmor and Berant, 2018)  34,689    MetaQA (Zhang et al., 2018)  400,000    CSQA (Saha et al., 2018)  1.6M    CFQ (Keysers et al., 2020)  239,357    GrailQA (Gu et al., 2021)  64.331    KQA Pro (ours)  117,970    Table 1: Comparison with existing datasets of Complex KBQA. et al., 2020; Schlichtkrull et al., 2018; Zhang et al., 2018; Zhou et al., 2018; Qiu et al., 2020; Shi et al., 2021), which constructs a question-specific graph extracted from the KB and ranks all the entities in the extracted graph based on their relevance to the question. Compared with information retrieval based methods, semantic parsing based methods provides better interpretability by generating expressive logic forms, which represents the intermediate reasoning process. However, manually annotating logic forms is expensive and labor-intensive, and it is challenging to train a semantic parsing model with weak supervision signals ( i.e. , question-answer pairs). Lacking logic form annotations turns out to be one of the main bottlenecks of semantic parsing. Table 1 lists the widely-used datasets in Complex KBQA community and their features. MetaQA and CSQA have a large number of questions, but they ignore literal knowledge, lack logic form annotations, and their questions are written by templates. Query graphs ( e.g. , SPARQLs) are provided in some datasets to help solve complex questions. However, SPARQL is weak in describing the intermediate procedure of the solution, and the scale of existing question-to-SPARQL datasets is small. In this paper, we introduce a novel logic form KoPL, which models the procedure of Complex KBQA as a multi-step program, and provides a more explicit reasoning process compared with query graphs. Furthermore, we propose KQA Pro, a large-scale semantic parsing dataset for Complex KBQA, which contains ~120k diverse natural language questions with both KoPL and SPARQL annotations. It is the largest NLQ-to-SPARQL dataset as far as we know. Compared with these existing datasets, KQA Pro serves as a more well-rounded benchmark. Relation , the link between entities or concepts. Entities are linked to concepts via the relation instance of . Concepts are organized into a tree structure via relation subclass of . Attribute , the literal information of an entity. An attribute has a key and a value, which is one of four types 1 : string, number, date, and year. The number value has an extra unit, e.g. , 206 centimetre. LeBron James) . Literal knowledge , the triple with form (entity, attribute key, attribute value), e.g. , (LeBron James, height, 206 centimetre) . Qualifier knowledge , the triple whose head is a relational or literal triple, e.g. , ((LeBron James, drafted by, Cleveland Cavaliers), point in time, 2003) . A qualifier also has a key and a value. We design KoPL, a compositional and interpretable programming language to represent the reasoning process of complex questions. It models the complex procedure of question answering with a program of intermediate steps. Each step involves a function with a fixed number of arguments. Every program can be denoted as a binary tree. As shown in Fig. 1, a directed edge between two nodes represents the dependency relationship between two 1 Wikidata also has other types like geographical and time. functions. That is, the destination function takes the output of the source function as its argument. The tree-structured program can also be serialized by post-order traversal, and formalized as a sequence with n functions. The general form is shown below. Each function f i takes in a list of textual arguments a i , which need to be inferred according to the question, and a list of functional arguments b i , which come from the output of previous functions. Take function Relate as an example, it has two textual inputs: relation and direction (i.e., forward or backward , meaning the output is object or subject). It has one functional input: a unique entity. Its output is a set of entities that hold the specific relation with the input entity. For example, in Question 2 of Fig. 1, the function Relate ([ father, forward ], [ LeBron James Jr. ]) returns LeBron James, the father of LeBron James Jr. (the direction is omitted in the figure for simplicity). We analyze the generic, basic operations for Complex KBQA, and design 27 functions 2 in KoPL. They support KB item manipulation ( e.g. , Find , Relate , FilterConcept , QueryRelationQualifier , etc. ), various reasoning skills ( e.g. , And , Or , etc. ), and multiple question types ( e.g. , QueryName , SelectBetween , etc. ). By composing the finite functions into a KoPL program 3 , we can model the reasoning process of infinite complex questions. Note that qualifiers play an essential role in disambiguating or restricting the validity of a fact (Galkin et al., 2020; Liu et al., 2021). However, they have not been adequately modeled in current KBQA models or datasets. As far as we know, we are the first to explicitly model qualifiers in Complex KBQA. To build KQA Pro dataset, first, we extract a knowledge base with multiple kinds of knowledge (Section 4.1). Then, we generate canonical questions, corresponding KoPL programs and SPARQL queries with novel compositional strategies (Sec-tion 4.2). In this stage, we aim to cover all the possible queries through random sampling and recursive composing. Finally, we rewrite canonical questions into natural language via crowdsourcing (Section 4.3). To further increase linguistic variety, 2 The complete function instructions are in Appendix A. 3 The grammar rules of KoPL are in Appendix B. we reject the paraphrases whose edit distance with the canonical question is small. We took the entities of FB15k-237 (Toutanova et al., 2015) as seeds, and aligned them with Wikidata via Freebase IDs 4 . The reasons are as follows: 1) The vast amount of knowledge in the full knowledge base ( e.g. , full Freebase (Bollacker et al., 2008) or Wikidata contains millions of entities) may cause both time and space issues, while most of the entities may never be used in questions. 2) FB15k-237 is a high-quality, dense subset of Freebase, whose alignment to Wikidata produces a knowledge base with rich literal and qualifier knowledge. We added 3,000 other entities with the same name as one of FB15k-237 entities to increase the disambiguation difficulty. The statistics of our final knowledge base are listed in Table 2. # Con. To generate diverse complex questions in a scalable manner, we propose to divide the generation into two stages: locating and asking . In locating stage we describe a single entity or an entity set with various restrictions, while in asking stage we query specific information about the target entity or entity set. We define several strategies for each stage. By sampling from them and composing the two stages, we can generate large-scale and diverse questions with a small number of templates. Fig. 2 gives an example of our generation process. For locating stage, we propose 7 strategies and show part of them in the top section of Table 3. We can fill the placeholders of templates by sampling from KB to describe a target entity. We support quantitative comparisons of 4 operations: equal, not equal, less than, and greater than, indicated by  <OP>  of the template. We support optional qualifier restrictions, indicated by ( <QK> is <QV> ), 4 The detailed extracting process is in Appendix C. 6104 Strategy Template Example Locating Stage Entity Name LeBron James Concept + Literal the <C> whose <K> is <OP> <V> ( <QK> is <QV> ) the basketball team whose social media followers is greater than 3,000,000 (point in time is 2021) Concept + Relational the <C> that <P> <E> ( <QK> is <QV> ) the basketball player that was drafted by Cleveland Cavaliers Recursive Multi-Hop unfold <E> in a Concept + Relational description the basketball player that was drafted by the basketball team whose social media followers is greater than 3,000,000 (point in time is 2021) Intersection Condition 1 and Condition 2 the basketball players whose height is greater than 190 centimetres and less than 220 centimetres Asking Stage QueryName What/Who is <E> Who is the basketball player whose height is equal to 206 centimetres? In Recursive Multi-Hop , we replace the entity of a relational condition with a more detailed description, so that we can easily increase the hop of questions. For asking stage, we propose 9 strategies and show some of them in the bottom section of Table 3. Our SelectAmong is similar to argmax and argmin operations in  -DCS. The complete generation strategies are shown in Appendix D due to space limit. Our generated instance consists of five elements: question, SPARQL query, KoPL program, 10 answer choices, and a golden answer. Choices are selected by executing an abridged SPARQL 5 , which randomly drops one clause from the complete SPARQL. With these choices, KQA Pro supports both multiple-choice setting and open-ended setting. We randomly generate lots of questions, and only preserve those with a unique answer. For example, since Akron has different populations in different years, we will drop questions like What is the population of Akron , unless the time constraint ( e.g. , in 2010 ) is specified. After large-scale generation, we release the generated questions on Amazon Mechanical Turk (AMT) and ask the workers to paraphrase them without changing the original meaning. For the convenience of paraphrasing, we visualize the KoPL flowcharts like Fig. 1 to help workers understand complex questions. We allow workers to mark a question as confusing if they cannot understand it or find logical errors. These instances will be removed from our dataset. After paraphrasing, we evaluate the quality by 5 other workers. They are asked to check whether the paraphrase keeps the original meaning and give a fluency rating from 1 to 5. We reject those paraphrases which fall into one of the following cases: (1) marked as different from the original canonical question by more than 2 workers; (2) whose average fluency rating is lower than 3; (3) having a very small edit distance with the canonical question. Our KQA Pro dataset consists of 117,970 instances with 24,724 unique answers. Fig. 3(a) shows the question type distribution of KQA Pro. Within the 9 types, SelectAmong accounts for the least fraction (4.6%), while others account for more or less than 10%. Fig. 3(b) shows that multi-hop questions cover 73.7% of KQA Pro, and 4.7% questions even require at least 5 hops. We compare the question length distribution of different Complex KBQA 6105 Which team picked LeBron James? 3(c). We observe that our KQA Pro has longer questions than others on average. In KQA Pro, the average length of questions/program-s/SPARQLs is 14.95/4.79/35.52 respectively. More analysis is included in Appendix G. 5 Experiments The primary goal of our experiments is to show the challenges of KQA Pro and promising Complex KBQA directions. First, we compare the performance of state-of-the-art KBQA models on current datasets and KQA Pro, to show whether KQA Pro is challenging. Then, we treat KQA Pro as a diagnostic dataset to investigate fine-grained reasoning abilities of models, discuss current weakness and promising directions. We further conduct an experiment to explore the generation ability of our proposed model. Last, we provide a case study to show the interpretablity of KoPL. Benchmark Settings . We randomly split KQA Pro to train/valid/test set by 8/1/1, resulting in three sets with 94,376/11,797/11,797 instances. About 30% answers of the test set are not seen in training. Representative Models . KBQA models typically follow a retrieve-and-rank paradigm, by constructing a question-specific graph extracted from the KB and ranks all the entities in the graph based on their relevance to the question (Miller et al., 2016; Saxena et al., 2020; Schlichtkrull et al., 2018; Zhang et al., 2018; Zhou et al., 2018; Qiu et al., 2020); or follow a parse-then-execute paradigm, by parsing a question to a query graph (Berant et al., 2013; Yih et al., 2015) or program (Liang et al., 2017; Guo et al., 2018; Saha et al., 2019; Ansari et al., 2019) through learning from question-answer pairs. Experimenting with all methods is logistically challenging, so we reproduce a representative subset of mothods: KVMemNet (Miller et al., 2016), a well-known model which organizes the knowledge into a memory of key-value pairs, and iteratively reads memory to update its query vector. EmbedKGQA (Saxena et al., 2020), a state-of-the art model on MetaQA, which incorporates knowledge embeddings to improve the reasoning performance. SRN (Qiu et al., 2020), a typical path search model to start from a topic entity and predict a sequential relation path to find the target entity. RGCN (Schlichtkrull et al., 2018), a variant 6106 of graph convolutional networks, tackling Complex KBQA through the natural graph structure of knowledge base. Our models. Since KQA Pro provides the annotations of SPARQL and KoPL, we directly learn our parsers using supervised learning by regarding the semantic parsing as a sequence-to-sequence task. We explore the widely-used sequence-to-sequence model RNN with attention mechanism (Dong and Lapata, 2016), and the pretrained generative language model BART (Lewis et al., 2020), as our SPARQL and KoPL parsers. For KoPL learning, we design a serializer to translate the tree-structured KoPL to a sequence. For example, the KoPL program in Fig. 2 is serialized as: Find  arg  LeBron James  func  Relate  arg  drafted by  arg  backward  func  FilterConcept  arg  team  func  QueryName . Here,  arg  and  func  are special tokens we designed to indicate the structure of KoPL. To compare machine with Human , we sample 200 instances from the test set, and ask experts to answer them by searching our knowledge base. Implementation Details . For our BART model, we used the bart-base model of HuggingFace 6 . We used the optimizer Adam (Kingma and Ba, 2015) for all models. We searched the learning rate for BART paramters in {1e-4, 3e-5, 1e-5}, the learning rate for other parameters in {1e-3, 1e-4, 1e-5}, and the weight decay in {1e-4, 1e-5, 1e-6}. According to the performance on validation set, we finally used learning rate 3e-5 for BART parameters, 1e-3 for other parameters, and weight decay 1e-5. We compare the performance of KBQA models on KQA Pro with MetaQA and WebQSP (short for WebQuestionSP), two commonly used benchmarks in Complex KBQA. The experimental results are in Table 4, from which we observe that: Although the models perform well on MetaQA and WebQSP, their performances are significantly lower and not satisfying on KQA Pro. It indicates that our KQA Pro is challenging and the Complex KBQA still needs more research efforts. Actually, 1) Both MetaQA and WebQSP mainly focus on relational knowledge, i.e. , multi-hop questions. Therefore, previous models on these datasets are designed to handle only entities and relations. In comparison, KQA Pro includes three types of 6 https://github.com/huggingface/transformers knowledge, i.e. , relations, attributes, and qualifiers, thus is much more challenging. 2) Compared with MetaQA which contains template questions, KQA Pro contains diverse natural language questions and can evaluate models' language understanding abilities. 3) Compared with WebQSP which contains 4,737 fluent and natural questions, KQA Pro covers more question types ( e.g. , verification, counting) and reasoning operations ( e.g. , intersect, union). KQA Pro can serve as a diagnostic dataset for in-depth analyses of reasoning abilities ( e.g. , counting, comparision, logical reasoning, etc. ) for Complex KBQA, since KoPL programs underlying the questions provide tight control over the dataset. We categorize the test questions to measure fine-grained ability of models. Specifically, Multi-hop means multi-hop questions, Qualifier means questions containing qualifier knowledge, Comparison means quantitative or temporal comparison between two or more entities, Logical means logical union or intersection, Count means questions that ask the number of target entities, Verify means questions that take yes or no as the answer, Zero-shot means questions whose answer is not seen in the training set. The results are shown in Table 5, from which we have the following observations: (1) Benefits of intermediate reasoning supervision. Our RNN and BART models outperform current models significantly on all reasoning skills. This is because KoPL program and SPARQL query provide intermediate supervision which benefits the learning process a lot. As (Dua et al., 2020) suggests, future dataset collection efforts should set aside a fraction of budget for intermediate annotations, particularly as the reasoning required 6107 Model Overall Multi-hop Qualifier Compari-son Logical Count Verify Zero-shot KVMemNet 16.61 16.50 18.47 1.17 14.99 27.31 54.70 0.06 SRN -12.33 ---EmbedKGQA 28.36 26.41 25.20 11.93 23.95 32.88 61.05 0.06 RGCN 35.07 34.00 27.61 30.03 35.85 41.91 65.88 0.00 RNN SPARQL 41.98 36.01 19.04 66.98 37.74 50.26 58.84 26.08 RNN KoPL 43.85 37.71 22.19 65.90 47.45 50.04 42.13 34.96 BART SPARQL 89.68 88.49 83.09 96.12 88.67 85.78 92.33 87.88 BART KoPL 90.55 89.46 84.76 95.51 89.30 86.68 93.30 89.59 BART KoPL CG 77.86 77.86 61.46 93.61 77.88 79.17 89.01 76.04 Human 97.50 97.24 95.65 100.00 98.18 83.33 95.24 100.00 Table 5: Accuracy of different models on KQA Pro test set. We hope our dataset KQA Pro with KoPL and SPARQL annotations will help guide further research in Complex KBQA. (2) More attention to literal and qualifier knowledge. Existing models perform poorly in situations requiring comparison capability. This is because they only focus the relational knowledge, while ignoring the literal and qualifier knowledge. We hope our dataset will encourage the community to pay more attention to multiple kinds of knowledge in Complex KBQA. (3) Generalization to novel questions and answers. For zero-shot questions, current models all have a close to zero performance. This indicates the models solve the questions by simply memorizing their training data, and perform poorly on generalizing to novel questions and answers. We further use KQA Pro to test the ability of KBQA models to generalize to questions that contain novel combinations of the elements observed during training. Following previous works, we conduct the productivity experiment (Lake and Baroni, 2018; Shaw et al., 2021), which focuses on generalization to longer sequences or to greater compositional depths than have been seen in training (for example, from a length 4 program to a length 5 program). Specifically, we take the instances with short programs as training examples, and those with long programs as test and valid examples, resulting in three sets including 106,182/5,899/5,899 examples. The performance of BART KoPL drops from 90.55% to 77.86%, which indicates learning to generalize compositionally for pretrained language models requires more research efforts. Our KQA Pro provides an environment for further experimentation on compositional generalization. To further understand the quality of logical forms predicted by the BART parser, we show a case in Fig. 4, for which the SPARQL and KoPL parsers both give wrong predictions. The SPARQL parser fails to understand prior to David Lloyd George and gives a totally wrong prediction for this part. The KoPL parser gives a function prediction which is semantically correct but very different from our generated golden one. It is a surprising result, revealing that the KoPL parser can understand the semantics and learn multiple solutions for each question, similar to the learning process of humans. We manually correct the errors of predicted SPARQL 6108 and KoPL and mark them in red. Compared to SPARQLs, KoPL programs are easier to be understood and more friendly to be modified. In this work, we introduce a large-scale dataset with explicit compositional programs for Complex KBQA. For each question, we provide the corresponding KoPL program and SPARQL query so that KQA Pro can serve for both KBQA and semantic parsing tasks. We conduct a thorough evaluation of various models, discover weaknesses of current models and discuss future directions. Among these models, the KoPL parser shows great interpretability. As shown in Fig. 4, when the model predicts the answer, it will also give a reasoning process and a confidence score (which is ommited in the figure for simplicity). When the parser makes mistakes, humans can easily locate the error through reading the human-like reasoning process or checking the outputs of intermediate functions. In addition, using human correction data, the parser can be incrementally trained to improve the performance continuously. We will leave this as our future work. This work is founded by the National Key Research and Development Program of China (2020AAA0106501), the Institute for Guo Qiang, Tsinghua University (2019GQB0003), Huawei Noah's Ark Lab and Beijing Academy of Artificial Intelligence.
abstain abstain abstain abstain method abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain result abstain abstain abstain method result abstain method method other other other other other other other other other other other objective objective abstain other other other other other other other other other other other method other other other other other other other other other abstain abstain abstain abstain method abstain abstain abstain abstain abstain objective objective objective abstain method method abstain abstain method abstain objective abstain method method abstain objective abstain method method objective abstain method abstain abstain method abstain method abstain abstain result method method abstain method abstain abstain abstain abstain abstain abstain method abstain result abstain objective objective objective objective abstain result abstain result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain other