Given the ubiquitous nature of numbers in text, reasoning with numbers to perform simple calculations is an important skill of AI systems. While many datasets and models have been developed to this end, state-of-the-art AI systems are brittle; failing to perform the underlying mathematical reasoning when they appear in a slightly different scenario. Drawing inspiration from GLUE (Wang et al., 2018) that was proposed in the context of natural language understanding, we propose NUMGLUE, a multi-task benchmark that evaluates the performance of AI systems on eight different tasks, that at their core require simple arithmetic understanding. We show that this benchmark is far from being solved with neural models including state-of-the-art large-scale language models performing significantly worse than humans (lower by 46.4%). Further, NUMGLUE promotes sharing knowledge across tasks, especially those with limited training data as evidenced by the superior performance (average gain of 3.4% on each task) when a model is jointly trained on all the tasks as opposed to task-specific modeling. Finally, we hope that NUMGLUE will encourage systems that perform robust and general arithmetic reasoning within language, a first step towards being able to perform more complex mathematical reasoning 1 . Reasoning with numbers is an important skill that occurs in various day-to-day scenarios and not surprisingly, numbers are ubiquitous in textual data. To train AI reasoning systems that can perform simple mathematical reasoning, many tasks have been proposed (Dua et al., 2019b; Ravichander et al., 2019; Koncel-Kedziorski et al., 2016). Despite these efforts, current state-of-the-art AI 1 https://allenai.org/data/numglue Original Word Problem John had 5 apples. Premise: John had 5 apples. He gave 3 apples to Peter. Hypothesis: John has 2 apples now. Does the hypothesis entail, contradict or is neutral to the premise? Comparison Format John had 5 apples. He gave 3 to Peter. Who has more apples? systems are brittle and fail when problems involving similar mathematical reasoning is posed in a slightly different manner. For instance, presenting a word problem in a different manner as shown in fig. 1, while hardly affecting human performance, is sufficient to confuse state-of-the-art AI systems 2 . This brittleness in reasoning indicates that the models latch on to spurious signals in the specific dataset resulting in solving the dataset while not truly understanding the underlying reasoning skill of simple arithmetic. Further, we believe that building AI systems that can truly understand and apply simple arithmetic reasoning is a mandatory first step towards successfully tackling complex 2 The recently released GPT3-Instruct, a fine-tuned model with 175B parameters produces inconsistent answers for these questions. See supplementary material: GPT3-Instruct's Response for more details. NumGLUE. To this end, we propose NUMGLUE, a multi-task benchmark consisting of eight different tasks that at their core test for arithmetic reasoning skills. For example, as discussed in fig. 1, tasks can involve word problems presented in a slightly different manner or can involve additional reasoning strategies like commonsense reasoning or reading comprehension to be combined with the core skill of simple arithmetic. Our benchmark consists of four new tasks in addition to four existing ones; with  100 K problems spread across eight differet tasks. The motivation behind NUMGLUE is similar to GLUE (Wang et al., 2018, 2019), a multi-task benchmark that aimed at models that demonstrated superior language understanding by learning the underlying linguistic features. NUMGLUE is designed with goal of progressing towards AI systems that are capable of performing arithmetic reasoning in a general setting; achieving superior performance on our benchmark requires the ability to correctly identify and perform the underlying arithmetic reasoning without relying on task or dataset-specific signals. Finally, we hope that NUMGLUE will encourage systems that perform robust and general numeric reasoning within language, a first step towards being able to perform more complex mathematical reasoning. 1. We introduce NUM GLUE a multi-task benchmark consisting of eight different tasks, including 4 new ones, whose solution at its core requires an understanding of simple arithmetic. 2. We demonstrate that NUMGLUE is a challenging benchmark even for state-of-the-art large scale language models, obtaining poor scores not only in zero or few shot settings but also after fine-tuning. This indicates a fundamental barrier for AI systems; one that needs to be breached before complex mathematical challenges can be successfully tackled. 3. Finally, we propose a memory-augmented neural model to demonstrate the utility of such a multi-task meta dataset. Our proposed model when trained on the entirety of NUMGLUE obtains an average improvement of 3.4% on each task as opposed to task-specific training  indicating that joint training leads to beneficial transfer owing to the common theme of arithmetic reasoning. Datasets for Numerical reasoning. Quantitative reasoning has been a challenging problem for a long time. Small question answering datasets were proposed to understand the quantitative aspect of natural language such as the template-based dataset which solved questions with equations as parameters (Kushman et al., 2014), addition-subtraction dataset (Hosseini et al., 2014) and arithmetic problems dataset (Koncel-Kedziorski et al., 2015). Difficulty of questions were increased in subsequent datasets (Roy and Roth, 2016), (Upadhyay et al., 2016). Later, larger datasets were created to facilitate deep learning research (Ling et al., 2017; Dua et al., 2019b). Several other maths datasets have been proposed to improve explainability (Amini et al., 2019), diversity (Miao et al., 2020), scale information in language embeddings (Zhang et al.) and hardness of math questions (Hendrycks et al., 2021). One of the motivations behind creating this benchmark is to test for simple arithmetic reasoning independent of the context or the presentation style of the problem. Further, To the best of our knowledge, our work is the first to consider multiple tasks in the numerical reasoning space. Multi-Task Benchmarks. With increased success of deep learning based models on individual tasks, there has been a significant push both in the NLP community and in the broader AI community towards general purpose models that excel at multiple tasks. Naturally, various benchmarks and challenges that test for such understanding have been proposed. For instance, the BAbI dataset (Weston et al., 2015), GLUE (Wang et al., 2019) and the subsequent harder SuperGLUE (Wang et al., 2019) were proposed to both evaluate and drive progress in language understanding via shared linguistic knowledge across tasks. McCann (McCann et al., 2018) build a multi-task dataset via a novel approach  formatting each task as that of question-answering. In the more restricted setting of reading comprehension, Dua et al. (2019a) and Downey and Rumshisky build a meta-dataset that spans multiple domains 3506 and reasoning skills. Multi-task Models. With the growing interest towards models that go beyond specific datasets, various neural models that can perform mutliple tasks have been proposed. When the underlying reasoning is similar  eg. commonsense reasoning, problem decomposition or linguistic understanding  it has been found that training on multi-task datasets yields more robust and accurate models. For instance, the Multi-task Question Answering Network (McCann et al., 2018), T5 (Raffel et al., 2019), GPT3 (Brown et al., 2020) and GPT3-Instruct models aim to build general purpose language models that are capable of transferring linguistic understanding across tasks. A similar approach is taken by Khashabi et al. (2020) in the setting of question-answering and Lourie et al. (2021) in the scope of commonsense reasoning. Further, Muppet (Aghajanyan et al., 2021) adds an additional step of pre-finetuning between pretraining and finetuning that improves generalization to multiple tasks. As mentioned previously, our NUMGLUE benchmark consists of both new and already existing arithmetic reasoning tasks. We first begin by introducing the novel datasets curated by us before providing a brief overview of existing tasks that are part of NUMGLUE. Finally, in this section, we provide an analysis of the datasets demonstrating that it contains interesting and diverse linguistic and mathematical properties. NUMGLUE Benchmark. Our proposed NUMGLUE benchmark is a collection of eight different tasks that together include  100 K questions. The tasks may either be self-contained or require additional background knowledge (e.g.commonsense reasoning) to arrive at the final solution; however, all the tasks, at their core, involve arithmetic reasoning. Table 1 shows an example question belonging to each task along with indicating the total number of data points associated with each task. It is important to note that tasks are imbalanced with only  400 examples for Task 1 and nearly 50 K questions under Task 5. While we could have under-sampled the questions to create a balanced suite, we retain the imbalanced dataset in order to mimic the real world  for instance, arithmetic word problems are more abundant as opposed to word problems that may require commonsense reasoning in addition to arithmetic reasoning. Data Partition and Evaluation. We randomly partition data in each task into training (70%), development (10%) and test (20%) sets . In the case of reading comprehension tasks (Task 5 and 6), we assign all questions corresponding to a passage to the same split  we do this in order to discourage any data leakage and thereby, allowing models to potentially rely on memorization to arrive at the correct answer. For each task, we report the F1 measure and as an aggregate measure of performance on the NUMGLUE benchmark similar to Dua et al. (2019b), we report the (unweighted) average of the F1 scores corresponding to each task. The novel tasks proposed as part of NUMGLUE are a combination of both freshly collected data and intelligent modifications of already existing datasets. The four novel arithmetic reasoning tasks introduced are as follows 3 : Task 1: Commonsense + Arithmetic Reasoning. Consider the following question  How many faces do 10 dice have? Answering this not only requires simple arithmetic i.e.multiplying the number of faces in a die by ten but also requires knowing that a standard die has six faces. We collect this dataset by first asking the annotator to write down a numerical commonsense fact (e.g.a human has 2 hands, a day has 24 hours etc. ) and then use frame a question that requires using this numerical fact as part of a simple arithmetic calculation. Task 2: Domain Specific + Arithmetic Reasoning. How many units of hydrogen are required to produce 10 units of water? This question, similar to the previously introduced task of arithmetic reasoning questions, requires additional domain-specific knowledge  specifically, that each unit of water contains two units of hydrogen. We 3 We annotate the datasets manually. curate a dataset of such questions that require both domain-specific knowledge and arithmetic reasoning motivated by the finding that QA systems perform poorly on the ARC dataset Clark et al. (2018) consisting of grade-school level science questions. Specifically, the dataset collected by us requires understanding of a small set of chemistry (conservation of mass in chemical reactions) and physics principles ( speed  distance { time ). Task 3: Commonsense + Quantitative Comparison. A golf ball weighs 40g and a baseball weighs 150g. Which has a higher gravitational force? Answering this question requires both knowing that mass is directly proportional to gravitational force and a numerical comparison via subtraction. We collect such quantitative comparison questions by using the QuaRel dataset (Tafjord et al., 2019) containing questions from diverse fields such as physics and economics as the starting point. The annotator chooses a subset of these questions that involve numerically comparable quantities (for instance, in this example, mass of the objects involved) to create the required task of quantitative comparison questions. information (e.g.commonsense knowledge) in addition to simple arithmetic reasoning, this task is self-contained but a stylistic variant of existing math word problems. We source word problems from the Arithmetic Word Problem repository (Roy and Roth, 2016, 2017, 2018) and convert them into the fill-in-the-blanks format. For an example of such a conversion, refer to fig. 1. 3.2 Existing Datasets We now review existing datasets while discussing any modifications made when including them in NUMGLUE. In general, for all the datasets included, we perform a filtering step to clean and control for the quality of the data points being included. This step includes a) discarding questions that do not have answer annotations b) eliminating questions with high lexical overlap with the remainder of the dataset and c) fixing any type mismatches present in the data (e.g.7.0 students  7 students). Task 5: Reading Comprehension (RC) + Explicit Numerical Reasoning. We select a subset from the DROP (Dua et al., 2019b) dataset to create this task. Specifically, the selected questions involve reading comprehension and numerical reasoning but importantly, the required 3508 answer is also a number. Task 6: Reading Comprehension (RC) + Implicit Numerical Reasoning. Consider the following question based on a relevant passage  Which state has the highest income tax rate? Here, while the final answer is a name, arriving at it requires performing comparison (i.e.subtraction). We classify such questions in the DROP dataset as a separate task in NUMGLUE. Task 7: Quantitative NLI EQUATE (Ravichan-der et al., 2019) introduces quantitative NLI questions that require simple arithmetic calculations to be performed in order to accurately classify the relationship between the provided premise and the hypothesis. As noted in fig. 1, many word problems can also be easily converted to this format and is therefore, a diverse and interesting task for evaluating arithmetic reasoning skills of AI systems. Task 8: Arithmetic Word Problems Finally, we arrive at one of the earliest and extensively studied class of arithmetic reasoning problems i.e.word problems. The specific dataset included as part of our NUM GLUEbenchmark is a combination of multiple datasets proposed by Koncel-Kedziorski et al. (2016), (Koncel-Kedziorski et al., 2015) and Kushman et al. (2014). Further, to ensure that the benchmark as a whole is diverse, we eliminate questions that have a high sentence similarity with questions from the fill-in-the-blanks task. In order to ensure a high-quality test set, three independent annotators evaluate each question in the test set across all tasks. A tiny porton of the data marked as invalid or with disagreement between the annotators was excluded, resulting in a verified, high-quality NUMGLUE evaluation suite. We also perform a variety of analysis and find that the novel question tasks we created (task 1-4) have higher quality than the existing question tasks since they have higher average vocabulary (number of unique words per number of samples), higher number of unique nouns, verbs and other POS tags and have less semantic textual similarity among each other (indicating lower repetition). Detailed analysis can be found in the supplementary material: Data Quality Analysis of NUMGLUE. In this section, we establish multiple baselines on our benchmark and discuss their performance. We evaluate several baselines on our benchmark (i) Heuristic, (ii) Zero-shot, (iii) Few-shot, (iv) Fine-tuning and (v) Human. We use two kinds of model architectures (i) Neuro-symbolic, a memory augmented novel architecture that extends Numnet+v2 (Ran et al., 2019) and (ii) End-to-end, GPT3 (Brown et al., 2020). Architectures. In the multi-task setting where the same model is trained on all the NUMGLUE tasks, we use Reading Comprehension (RC) as the common format  converting each task to RC format via a set of hand-coded rules 4 . In addition to being capable of faithfully representing all the constituent tasks, the RC format also allows us to inject additional context in the IR setting without affecting the rest of the pipeline 5 . On the other hand, GPT3 being a generative model does not require such modifications. Importantly, note that both models are inputted the exact same information for the multi-task experiments. Heuristic Baselines with Task Oracle. For this baseline, we assume a task oracle that knows the task a particular question belongs (in a multitask setting)  we use this to make our heuristic baselines more competitive. The first heuristic baseline is random : we randomly select one of the options in case the question has multiple options (task 3 and 7), a number between 0 to 100 for questions having a numerical answer and a random entity present in the passage for questions having a text segment from the passage as the answer. In the majority baseline, we select the most frequent answer for each task such as "Entailment" for NLI questions and similarly, the most frequent number for questions having numerical answer and the major entity present in the passage for questions having span based answer. As the task information is known, we include these baselines under task-specific baselines when discussing results. Zeroshot and Fewshot Baselines. We use GPT3 (Brown et al., 2020) and the more recent GPT3-Instruct 6 . We have two types of few shot baseline (i) task specific and (ii) multi task. In case of task specific fewshot baseline, instances of the same task are used as in-context examples (Brown et al., 2020) whereas in case of multitask few shot baseline, instances from all tasks are used to condition the model. Multitask fewshot is naturally a harder setting as it is task-agnostic. We use default parameters in GPT3 and GPT3-Instruct. In few-shot setting, we experiment after feeding as many examples as it can fit within the tokensize. For few shot experiments, we randomly select 6 newly released by OpenAI as part of the GPT3 finetuned series examples and averaged the results over 5 runs. first consider variations of the fine-tuning baselines in the context of our neuro-symbolic model, Ex-NumNet. We use it as bias-checking baseline  to ensure that solving the benchmark correctly requires considering all of the information presented to it. To this end, we evaluate the performance of our model when finetuned only on the question (Q-only) or the context (C-only). Next, we present task-specific and multi-task baselines where Ex-NumNet is fine-tuned on individual tasks and the entire NUMGLUE benchmark respectively. With the goal of addressing the data imbalance across the tasks, we include an oversampling 3510 Learning Baseline Baseline Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 NumGLUE category name Score HEURISTIC Task-specific Random 0 0.3 46.9 0 0.5 3.4 33 0.4 10.6 Task-specific Majority 1.2 13.9 50 0.5 7.4 3.8 36.5 1.2 14.3 ZERO-SHOT GPT3 0 1 11 2 0 17 6 2 4.9 -GPT3-Instruct 2 1 7 3 3 29 17 3 8.1 FEW-SHOT Task-specific GPT3 44 42 46 40 10 42 35 40 37.4 Task-specific GPT3-Instruct 40 39 51 33 13 43 35 33 35.9 Multi-task GPT3 0 3 27 1 7 28 30 4 12.5 Multi-task GPT3-Instruct 1 2 37 2 6 35 31 7 15.1 FINE-TUNING Multi-task GPT3-13B 21.5 40.7 71.2 11.1 6.3 48.2 48.0 14.2 32.7 FINE-TUNING Multi-task (Q-only) Ex-NumNet 1.2 13.2 25.1 0.5 6.1 25.1 32.8 2.4 13.3 Multi-task (C-only) Ex-NumNet 1.2 14.2 22.8 19.1 0.6 3 0 9.5 8.8 Single-task Ex-NumNet 0 37.8 50.8 22.2 66.6 71.6 85.9 12.2 43.4 Multi-task Ex-NumNet 0 37.5 58 31.4 68.2 70.2 85.7 23.2 46.8 Multi-task + IR Ex-NumNet 5.6 37.5 46.6 36.4 68.6 69.6 85.9 22.4 46.6 Multi-task + CIR Ex-NumNet 7.4 38.8 58 36.8 69.2 70.8 85.8 23.6 48.8 Multi-task + OS Ex-NumNet 7.4 38.8 47.8 35.9 44.3 53.7 85.4 22.4 42.0 -Human 94.4 94.5 97.8 95 94.7 96.1 96.5 92.8 95.2 Table 2: F1 performance of various baselines on the NumGLUE test set across various tasks 1-8. baseline that oversamples data from tasks with limited data so as to ensure that the model sees the same number of examples from each constituent task. In addition, we propose a new architectural modification to Ex-NumNet. Noting that our baseline model Ex-NumNet does not take into account external knowledge, we create a new enhanced architecture in the form of a memory-augmented model that does Information Retrieval (IR) (Khot et al., 2019) with respect to a knowledge base we create, MATH KB to identify the needed knowledge. This is inspired by the observation that formula book and mathematical knowledge make the task easier for humans while solving math questions of various types. We then use this knowledge in the Ex-NumNet setting. Figure 3 illustrates our approach which leverages our newly created knowledge base MATH KB . Conditional IR model is different from the regular IR model in the sense that, IR is performed only for questions of task 1 , 2 and 4, since they require external knowledge to get answered. More details about the model and the IR process can be found in supplementary material: Proposed Memory-Augmented Model (A.5 and A.6). Finally, we discuss fine-tuning baselines in the context of end-to-end models, specifically GPT3. We finetune the GPT3-13B model (for which the finetuning capability has been recently provided by OpenAI 7 ) in the multi-task setting i.e. the desired setting of the NUMGLUE benchmark. Human Baseline. Human baseline was calculated on 100 test set samples of each task (81 of Task 1) by averaging the scores of four annotators. Table 2 shows the performance of various baseline models on the test set of our benchmark. Note that the performance of all baseline models is significantly lesser than the human baseline (Figure 2). We now discuss various insights based on these results. Does the benchmark contain bias that a model can exploit? A challenging dataset requires the model to ideally consider all the information provided to it before arriving at an answer. To ensure that this is indeed the case, we perform ablations where only one portion of the input is provided i.e. either the question or the context. Both these bias-checking baselines perform poorly even in task-specific setting  indicating that both the benchmark and constituent tasks are challenging. Which Tasks are Hard to Solve? Our results show that task 1 which requires numerical commonsense knowledge, is the hardest task to solve. Similarly, tasks 2, 4 and 8 appear to be 7 https://beta.openai.com/docs/guides/fine-tuning 3511 comparatively harder from the rest. One pattern among these tasks is that all of them expect the answer to be numeric. Numeric answer requires accurate calculation. So, models might have difficulty in learning the task directly from data. This hypothesis is also justified from the slight drop in human performance in these tasks.. On the other hand, task 7 has the best performance among all. Further, we see that performance on task 6 is slightly better than task 5  although both tasks are sourced from the same dataset, we observe that models answer span based questions better as compared to numeric answers. Relatively higher performance for task 3 suggests that models find it easier to answer in an MCQ setting. Does IR Help? Results show that knowledge help in improving performance of tasks 1, 2 and 4  where indeed, external knowledge like commonsense or domain-specific knowledge is needed in addition to arithmetic reasoning to arrive at the correct answer. However, task 3 is an exception to this trend and in fact registers a drop in the score when provided with (unnecessary) additional information; we find that this shortcoming is fixed when using conditional information retrieval (CIR) which in fact leads to the strongest baseline presented in this work. Does Oversampling help overcome data imbalance across tasks? Even though oversampling results in higher performance in certain tasks (in comparison with the multitask baseline), specifically the ones with smaller training data, it results in significant drop in performance in the other extreme, i.e tasks with bigger training data. Also, it never performs better than the Conditional IR module in multitask setting. We now present an analysis of the errors made by our baselines to indicate potential avenues for future research. We analyze errors associated with 50 samples each of the 8 tasks and find that there are mainly 4 categories of error models make: (1) producing invalid output (e.g. answering text where the answer is supposed to be a number, answering a text different from the classes allowed in a classification problem), (2) copying a number Error Ex-NumNet GPT3 Invalid output 16 % 7% Copy number 5 % 3% Incorrect calculation 71 % 56% Redundant text 8 % 34% Table 3: Error analysis for the best Ex-NumNet Multi-task+CIR and GPT3 Task-specific model from the question instead of calculating the answer, (3) incorrect calculation  this can be due to multiple reasons including (i) using an incorrect operation e.g. subtraction in place of addition, (ii) incorrect parsing of numbers or (iii) incorrect knowledge of numerical commonsense facts. (4) producing redundant text after producing correct answer. Based on error distribution in Table 3, we observe that the majority of errors come from incorrect calculation. Further, GPT3 is better than Ex NumNet+v2 in producing valid outputs, but it produces more redundant text. Future Directions: Bigger model, more data or . . . ? Table 2 shows that fine-tuned GPT3-13B outperforms other baselines on task 1, 2 and 3. Recall that these tasks require external knowledge and perhaps, this is the reason why GPT3, already pre-trained on a diverse web-scale text corpus has an edge over other baselines on these tasks. In case of the smaller Ex-NumNet, it is interesting that multitask baselines are higher than the single task baselines by 3.4% on average and that information retrieval helps in tasks that require external knowledge. Also notice that, GPT-3 is better on smaller datasets and NumNet is better on large datasets. This may indicate that GPT-3 is a better few-shot learner but not necessarily a better many-shot learner. This non-overlapping performance of GPT-3 and Ex-numnet, end-to-end and neuro-symbolic models respectively, indicates that a potential future direction for research is to combine the best of both the models. We propose NUMGLUE, a multi-task benchmark to test for arithmetic understanding. Our benchmark consists of eight tasks including four new ones. While some of the tasks require external knowledge like commonsense or domain-specific information in addition to arithmetic reasoning, some are self-contained e.g. arithmetic word problems. Further, we demonstrate that our benchmark 3512 is far from being solved  with state-of-the-art large scale models achieving considerably lower performance than humans. This indicates that current AI systems are incapable of performing simple arithmetic reasoning in a general setting  indicating a fundamental hurdle towards AI systems that understand complex mathematical concepts like differential equations or combinatorics. Finally, we present various baselines including a novel architecture (memory augmented Ex-NumNet) that demonstrate the advantages of various modeling choices (e.g. end-to-end vs neuro-symbolic mod-els). Specifically, we show that training in the multi-task setting leads to meaningful sharing of knowledge across tasks as evidenced by an average gain of 3.4% on tasks compared to task-specific modeling. Finally, we hope that our benchmark not only leads to AI systems that are capable of performing simple arithmetic reasoning in a fairly general setting but also results in progress towards more complex mathematical reasoning capability. We thank OpenAI for providing academic access to the GPT3 API, the Aristo team at AI2 for helpful input, the Beaker team for their support with experiments and the anonymous reviewers for their insightful feedback. The support of DARPA SAIL-ON, DARPA CHESS program is gratefully acknowledged. We have verified that all licenses of source datasets used in this paper allow for their use, modification, and redistribution in a research context. The dataset will be distributed in a manner similar to SuperGLUE (Wang et al., 2019) i.e. give full credit assignment to the original data and task creators.
abstain abstain abstain objective abstain objective abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other objective other other other other other other other other other other other other other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain method method other abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain result abstain result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain objective objective abstain objective abstain objective result result other other abstain abstain