We present ASDiv ( A cademia S inica Di-v erse MWP Dataset), a diverse (in terms of both language patterns and problem types) English math word problem (MWP) corpus for evaluating the capability of various MWP solvers. Existing MWP corpora for studying AI progress remain limited either in language usage patterns or in problem types. We thus present a new English MWP corpus with 2,305 MWPs that cover more text patterns and most problem types taught in elementary school. Each MWP is annotated with its problem type and grade level (for indicating the level of difficulty). Furthermore, we propose a metric to measure the lexicon usage diversity of a given MWP corpus, and demonstrate that ASDiv is more diverse than existing corpora. Experiments show that our proposed corpus reflects the true capability of MWP solvers more faithfully. Human math/science tests have been considered more suitable for evaluating AI progress than the Turing test (Clark and Etzioni, 2016). Among them, math word problems (MWPs) are frequently chosen to study natural language understanding and simulate human problem solving (Bakman, 2007; Mukherjee and Garain, 2008; Liang et al., 2016), because the answer is not a span within the given problem text that can be directly extracted. Table 1 shows a typical example of MWP, which consists of a few sentences that involve quantities. Current MWP corpora can be classified into four categories: (1) the Number Word Problem corpus (Shi et al., 2015), which contains number word problems only; (2) the Arithmetic Word Problem corpora (Hosseini et al., 2014; Roy et al., 2015), which involve the four basic arithmetic operations Math Word Problem A sandwich is priced at $0.75. A cup of pudding is priced at $0.25. Tim bought 2 sandwiches and 4 cups of pudding. How much money should Tim pay? Solution: 0.75 x 2 + 0.25 x 4 = 2.5 Table 1: A math word problem ( addition , subtraction , multiplication and division ) with either single-step or multi-step operations; (3) the Algebraic Word Problem corpora (Kushman et al., 2014; Koncel-Kedziorski et al., 2015; Roy and Roth, 2017; Upadhyay & Chang, 2015; Wang et al., 2017), which focus on algebraic MWPs; and (4) the Mixed-type MWP corpora (Huang et al., 2016, Ling et al., 2017, Amini et al., 2019), which are large-scale collections of either daily algebra or GRE/GMAT examination MWPs. Table 2 is a comparison of existing English MWP corpora. However, these existing corpora are either limited in terms of the diversity of the associated problem types (as well as lexicon usage patterns ), or lacking information such as difficulty levels . For example, categories (1), (2), and (3) collect only certain types of MWPs. On the other hand, although large-scale mixed-type MWP corpora contain more problem types, the annotated answers or formulas are sometimes inconsistent, and the corresponding difficulty level is usually not provided. Furthermore, low-diversity corpora are typically characterized by highly similar problems, which usually yields over-optimistic results (Huang et al., 2016) (as the answer frequently can be simply obtained from the existing equation template associated with the most similar MWP in the training-set). Roy and Roth (2017) shown significantly lowered performance if highly similar MWPs are removed. Therefore, dataset diversity is more critical than the dataset size for accurately judging the true capability of an MWP solver. We thus present ASDiv ( A cademia S inica Di-v erse MWP Dataset), a new MWP corpus that contains diverse lexicon patterns with wide problem A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su Institute of Information Science, Academia Sinica, Taiwan miao.iisas@gmail.com, {ccliang, kysu}@iis.sinica.edu.tw 976 type coverage. Each problem provides consistent equations and answers. It is further annotated with the corresponding problem type and grade level , which can be used to test the capability of a system and to indicate the difficulty level of a problem, respectively. The diverse lexicon patterns can be used to assess whether an MWP solver obtains answers by understanding the meaning of the problem text, or simply by finding an existing MWP with similar patterns (Huang et al., 2016). Problem type diverseness is crucial for evaluating whether a system is competitive with humans when solving MWPs of various categories. Besides, to assess text diversity, we propose a lexicon usage diversity metric to measure the diversity of an MWP corpus. This paper makes the following contributions: (1) We construct a diverse (in terms of lexicon us-age), wide-coverage (in problem type), and publicly available 1 MWP corpus, with annotations that can be used to assess the capability of different systems. (2) We propose a lexicon usage diversity metric to measure the diversity of an MWP corpus and use it to evaluate existing corpora. (3) We show that the real performance of state-of-the-art (SOTA) systems is still far behind human performance if evaluated on a corpus that mimics a real human test. A problem type ( PT ) indicates a crucial math operation pattern for solving an MWP. As MWPs of 1 The corpus can be found at: https://github.com/chao-chun/nlu-asdiv-dataset. the same problem type share a similar pattern (in language usages, logic representation, or infer-ences), they thus indicate stereotypical math operation patterns that could be adopted to solve an MWP (Liang et al., 2018). In ASDiv , each MWP is annotated with a specific PT taught at elementary schools. Some examples of selected PTs are shown in Table 5 (Appendix). Currently, we provide 24 different common PTs and classify them into three main categories according to the operations involved and illustrated below. These PTs are usually specified in math textbooks and mostly covered in elementary schools. Basic arithmetic operations: This category includes: Addition , Subtraction , Difference , Multiplication , three different Divisions (i.e., common-di-vision , floor-division, and ceil-division ), Sum , Surplus , Number-Operation , three different T ime-V ariantQ uantities (TVQ), and Multi-step . The first seven types are self-explanatory. Number-Op-eration indicates that the problem description consists mainly of numbers and their relations. TVQ 2 denotes an entity-state related variable (e.g., ini-tial/current/final-state and change ) whose value is updated sequentially according to a sequence of events described in an MWP. Last, in a Multi-step problem, the answer is obtained from multiple arithmetic operations. Aggregative operations: This category includes: (1) Comparison , (2) Set-Operation , (3) Ra 2 E.g.,  the number of apples that Jack has  is a TVQ in  Jack had 5 apples; he then ate 3 of them. How many apples does Jack has now? . tio, (4) Number-Pattern , (5) Algebra-1 , and (6) Al-gebra-2 . The first three types are self-explanatory. Number-Pattern refers to the problems which involve deducing a pattern from a sequence of integers (Table 5 (Appendix) shows an example). Al-gebra-1 and Algebra-2 are algebraic problems with one and two unknown variables, respectively. Additional domain knowledge required: This category includes Greatest Common Divisor , Least Common Multiple , Geometry , and UnitTrans. Additional geometric knowledge (e.g., area = length * width ) is required in Geometry problems. UnitTrans means that the answer is obtained via conversion to the metric system (e.g., converting  miles to  kilometers' ). This corpus was designed based on the following guidelines: (1) The corpus should be as diverse as possible in terms of lexicon usage so that the answer is less likely to be obtained via mechani-cal/statistical pattern matching without understanding the content. (2) The corpus should cover most PTs found in primary school so that it can approximate real human tests. (3) The corpus should be annotated with sufficient information so that it can be used not only to assess the capability of various systems but also to facilitate system development. We first propose a lexicon usage diversity metric, in terms of BLEU (Papineni et al., 2002), to measure the degree of diversity of a given corpus. This metric is from 0 to 1; higher value indicates the corpus is more diverse. We first use Stanford CoreNLP (Manning et al., 2014) to tokenize and tag POSs, and then use NLTK (Bird et al., 2004) to lemmatize each token. Furthermore, we normalize the original sentences with: (1) stop word removal; and (2) named entity and quantity normalization, which replace the associated person names and quantity values with meta symbols in an MWP (i.e., two MWPs are regarded as identical if they differ only in names or quantity-values). This thus places the focus on essential words that matter in the MWP. The obtained sequence is then used to measure the lexicon usage diversity specified below. Let  (cid:3404) (cid:4668) (cid:2869) ,  (cid:2870) ,  ,  (cid:3014) (cid:4669) be a specific set of MWPs in a given corpus with the same PT , where  (cid:3036) is the ith MWP in  . For a given  (cid:3036) , we define its lexicon usage diversity (LD) of  (cid:3036) as  (cid:3036) (cid:3404) 1 (cid:3398) max (cid:3037),(cid:3037)(cid:2999)(cid:3036) (cid:3435) (cid:3036) ,  (cid:3037) (cid:3439) (cid:3397) (cid:3435) (cid:3037) ,  (cid:3036) (cid:3439) 2 , where (cid:3435) (cid:3036) ,  (cid:3037) (cid:3439) is the BLEU score between  (cid:3036) and  (cid:3037) ( j (cid:3405) i ,  (cid:3404) (cid:4668)1,2,  , (cid:4669) ) . We measure the BLEU score bi-directionally with n-grams up to n=4 . This measure is mainly used to identify repeated usage of lexicon and phrases; n=4 suffices for this case . (cid:3036) evaluates the lexicon diversity between  (cid:3036) and all  (cid:3037) (  (cid:3405) (cid:4667) . Furthermore, the mean of all  (cid:3036) (under the same corpus) can be used to indicate the corpus lexicon diversity (CLD). Adding a new MWP with a low LD to an existing corpus introduces little new information to the corpus; thus, it should be either discarded or revised. This diversity metric can help the corpus constructor to decide whether an MWP can be directly adopted or not. Since MathQA is the second-largest dataset in Table 2 (with 37K MWPs), and is cleaner (Amini et al., 2019) than the largest one (AQuA), we first evaluate it with the above LD measurement. Figure 1 shows that its CLD is only 0.05. To understand the reason for the low diversity of MathQA (LD = 0 for 85% of the MathQA MWPs), we investigated this dataset. We observed that MathQA includes various MWP subsets, each of Figure 1: Lexicon usage diversity of various corpora. which shares the same sentence pattern among its members. Figure 1 clearly shows its skewed distribution. Figure 3 (Appendix) shows a subset of which all 105 members share the same sentence pattern. Since most MWP solvers can only solve arithmetic MWPs, we further selected its arithmetic 3 subset, generated their corresponding equations according to the annotated formulas, and then solved the equations using the SymPy 4 package. Afterwards, we verified the consistency between the answer obtained from the annotated formula and the labeled answer. The results show that the annotated formulas of 27% of the problems do not match their labeled answers. We randomly inspected 30 inconsistent MWPs and classified them into three error-types: (1) Incorrect formula (67%), for which the annotated formula cannot be used to solve the given MWP; (2) problematic description (23%), for which the description text is either incomplete or problematic; (3) valueless answer (10%), for which the given answer is either wrong or inappropriate. Table 6 (Appendix) illustrates examples of each error-type. Although building a large corpus via crowd-sourcing is a tempting approach, it can result in a poor-quality corpus if the annotation procedure is not well controlled. We believe the quality of the dataset is more important than its size , if they cannot be achieved simultaneously. To account for the problems observed in MathQA, we first collected MWPs from 28 websites and then either pruned the problem or revised the text if it was highly similar to any existing ones (ac-cording to the proposed lexicon usage diversity metric). This yielded a total of 2,305 MWPs. Next, we hired one master-degree research assistant with a background in automatic MWP solving to annotate the problem type, equation, answer, and grade level manually for each MWP. If annotations were provided with the original MWP (22.6% of the source MWPs included equations and answers; 52% had answers only; 63.5% included grade-level information), we used it directly; otherwise, we annotated them manually 5 . Since MWPs are usually clearly specified (with 3 Associated formula involves only arithmetic operations. 4 https://www.sympy.org/en/index.html. a sure answer), there is no ambiguous interpretation once the answer is given. Therefore, as opposed to other corpora in which annotations (mostly linguistic attributes) are mainly based on human subjective judgment, the MWP an-swer/equation annotation is more objective and must be consistent. As a result, human carefulness , instead of human agreement , is a more critical issue in this task. Since an incorrect math expression usually yields an incorrect answer, we used a program to automatically verify the consistency between the annotated equations and the answers. Inconsistent MWPs were re-annotated and checked again. Afterwards, we randomly selected 480 samples (20 samples per problem type) to verify the final annotation correctness. All those samples were correct, which confirms our above assertion. Figure 2 shows the distribution of different problem categories in six grade levels in elementary school. Most arithmetic operations appear in grade levels 1 to 4, which means students learn basic arithmetic operations in this stage. We further separate Addition/Subtraction from Multiplica-tion/Division to highlight that they are in different difficulty levels for students. Figure 2 also indicates Multiplication/Division is more emphasized in grade 3 and 4. In grades 5 and 6, improved math skills enable students to solve difficult MWPs that require more aggregative operations and additional domain knowledge. Thus, the grade level is a useful indicator of difficulty and can be employed to evaluate the capability of MWP solving systems. We compare the diversity among various MWPs of the same PT (for those corpora without annotated PT Category, diversity is measured over the whole corpus). Lastly, we generate the associated LD distributions (uniformly quantized into 20 intervals between 0 and 1) and calculate the corpus lexicon many, one: Toward rigorous common core standards from the ground up. MathQA-C (CLD=0.08) ASDiv-A (CLD=0.50) ASDiv (CLD=0.49) L -0.68 0.36 U -0.78 0.37 G 0.86 0.68 # 0.36 # Table 3: Accuracies for different systems (CLD denotes the corpus lexicon diversity; L, U and G denote the LCA++ , UnitDep, and GTS systems respectively. denotes failure on this corpus; # indicates performance is significantly lower than -C with p<0.01. G1 G2 G3 G4 G5 G6 L 0.53 0.64 0.49 0.35 0.03 0.01 U 0.55 0.65 0.51 0.34 0.03 0.01 G 0.64 0.60 0.47 0.34 0.07 0.01 Table 4: Performance of various grade levels on the ASDiv. L/U/G are the same as that in Table 3. diversity (CLD, Section 3.1) on corpora frequently adopted for comparing various systems: (1) AI2, (2) IL, (3) KAZB, (4) ALGES, (5) DRAW, (6) AllArith, and (7) MathQA. Figure 1 shows the distributions of CLD for various corpora: there are about 85%, 28%, 22% and 20% identical MWPs (these numbers are the percentages of MWPs with  (cid:3036) =0 w.r.t. each dataset) in MathQA, IL, AI2 and ALGES corpora respectively, whereas ASDiv contains none. We also evaluate syntactic pattern diversity (in terms of POS n-gram ) and the diversity between MWPs in the training set and the test set. Both yield similar trends, too (details are given in the Appendix). 4 Experiments To study the correlation between CLD and system performance, we selected three SOTA MWP solvers to conduct the experiments: two based on statistical models, LCA++ (Roy and Roth, 2015) and UnitDep (Roy and Roth, 2017); and one using a neural network which adopts two-layer gate-feed-forward networks for a Goal-driven Tree-struc-tured approach ( GTS ) (Xie et al., 2019). Since the selected MWP solvers solve only arithmetic MWPs, we first collected 4,117 MWPs from MathQA to construct a subset that its associated formulas satisfy the following two conditions: (1) they involve only arithmetic operations; and (2) they contain neither external constants (which would necessitate external domain knowledge to solve the problem and is out of the scope of this work) nor reused operands (which rarely occur and would complicate the solution procedure). We filtered out inconsistent problems (specified in Section 3.2) and termed the remaining 3,000 MWPs as MathQA-C dataset (-C for consistent ) to evaluate the performance. Similarly, we extracted a subset of 1,218 MWPs that involve only arithmetic operations (and also satisfy the constraints mentioned above) from ASDiv, and termed this the ASDiv-A dataset (-A for arithmetic ). The CLDs for MathQA-C and ASDiv-A were found to be 0.08 and 0.50, respectively. Also, LD = 0 for 82% of the MathQA-C MWPs. Afterwards, we tested the solvers against three MWP corpora: MathQA-C, ASDiv-A, and ASDiv. MathQA-C is reported with 5-fold cross-validation accuracy. For ASDiv-A and ASDiv, we randomly split the MWPs of each PT into five nearly equally-sized subsets, and report the 5-fold cross-validation accuracy. For GTS system, we repeated the experiment 5 times and obtained the averaged answer accuracy. Table 3 compares the answer accuracies of various systems. We observe that the overall performance is only around 36% on ASDiv, which shows that the performance of the current SOTA systems still is not competitive with human performance, and that CLD is correlated with the system performance (i.e., lower diversity implies higher performance) and is a useful metric to evaluate existing corpora. Table 4 further shows the accuracy of different grade levels on ASDiv: the performance of grades 5 and 6 are significantly lower than the performance of grade 1 to 4. As accuracy is highly correlated with the grade level, the grade level is a useful index for indicating the difficulty of MWPs. 5 Conclusion and Future Work We present an MWP corpus which not only is highly diverse in terms of lexicon usage but also covers most problem types taught in elementary school. Each MWP is annotated with the corresponding problem type, equation, and grade level, which are useful for machine learning and assessing the difficulty level of each MWP. We also propose a metric to measure the diversity of lexicon usage of a given corpus. In terms of this metric, we show that in comparison with those corpora widely adopted to compare systems, ours is more suitable for assessing the real performance of an MWP solver. Last, we conduct experiments to show that a low-diverse MWP corpora will exaggerate the true performance of SOTA systems (we are still far behind human-level performance), and that grade level is a useful index for indicating the difficulty of an MWP. 980 References Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, Hannaneh Hajishirzi. 2019. MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms. In Proceedings of NAACL-HLT 2019 . Yefim Bakman. 2007. Robust understanding of word problems with extraneous information. http://lanl.arxiv.org/abs/math.GM/0701393 . Steven Bird and Edward Loper. 2004. NLTK: The Natural Language Toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, 2004, pages 214-217. Peter Clark and Oren Etzioni. 2016. My Computer is an Honor Student but how Intelligent is it? Standardized Tests as a Measure of AI. AI Magazine, pages 512. Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin and Wei-Ying Ma. 2016. How well do computers solve math word problems? Large-scale Dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, Association for Computational Linguistics (ACL), 2016, pages 887896 . Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb Categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014 . Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics (TACL), 3:585597 . Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. Association for Computational Linguistics (ACL), 1:271281, Jun. 2014 . Chao-Chun Liang, Shih-Hong Tsai, Ting-Yun Chang, Yi-Chung Lin and Keh-Yih Su. 2016. A Meaning-based English Math Word Problem Solver with Understanding, Reasoning and Explanation. In Proceedings of the 26th International Conference on Computational Linguistics (COLING): System Demonstrations, pages 151155, Osaka, Japan, December 11-17 2016 . Chao-Chun Liang, Yu-Shiang Wong, Yi-Chung Lin and Keh-Yih Su. 2018. A Meaning-based Statistical English Math Word Problem Solver. In Proceedings of NAACL-HLT 2018 . Wang Ling, Dani Yogatama, Chris Dyer and Phil Blun-som. 2017. Program Induction for Rationale Generation: Learning to Solve and Explain Algebraic Word Problems. Association for Computational Linguistics (ACL), 2017 . Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55-60 . Anirban Mukherjee and Utpal Garain. 2008. A review of methods for automatic understanding of natural language mathematical problems. Artificial Intelligence Review, 29(2):93122 . Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), 2002, pages 311 318 . Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015, pages 17431752 . Subhro Roy and Dan Roth. 2017. Unit Dependency Graph and its Application to Arithmetic Word Problem Solving. AAAI-2017 . Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. 2015. Automatically Solving Number Word Problems by Semantic Parsing and Reasoning. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), Lisbon, Portugal, 2015, pages 11321142 . Shyam Upadhyay and Ming-Wei Chang. 2015. DRAW: A challenging and diverse algebra word problem set. Number MSR-TR-2015-78 Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing . Zhipeng Xie, and Shichao Sun. 2019. A goal-driven tree-structured neural model for math word problems. In Proceedings of the 28th International Joint Conference on Artificial Intelligence. AAAI Press . 981 Appendix Appendix A: Examples of a few Selected Problem Types Table 5 shows examples of selected types in Basic arithmetic operations, Aggregative operations, and Additional domain knowledge required categories. Problem type Examples Basic arithmetic operations Number-Operation I have 3 hundreds, 8 tens, and 3 ones. What number am I? TVQ-Initial Tim's cat had kittens.
method abstain objective abstain objective objective abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain objective objective objective result abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain objective abstain abstain method abstain other abstain abstain abstain result abstain result method abstain abstain result abstain abstain method method abstain other