ANALYSING MATHEMATICAL REASONING ABILITIES OF NEURAL MODELS
Mathematical reasoning-a core ability within human intelligence-presents some unique challenges as a domain: we do not come to understand and solve mathematical problems primarily on the back of experience and evidence, but on the basis of inferring, learning, and exploiting laws, axioms, and symbol manipulation rules. In this paper, we present a new challenge for the evaluation (and eventually the design) of neural architectures and similar system, developing a task suite of mathematics problems involving sequential questions and answers in a free-form textual input/output format. The structured nature of the mathematics domain, covering arithmetic, algebra, probability and calculus, enables the construction of training and test splits designed to clearly illuminate the capabilities and failure-modes of different architectures, as well as evaluate their ability to compose and relate knowledge and learned processes. Having described the data generation process and its potential future expansions, we conduct a comprehensive analysis of models from two broad classes of the most powerful sequence-to-sequence architectures and find notable differences in their ability to resolve mathematical problems and generalize their knowledge.
Summary: This paper is about models for solving basic math problems. The main contribution is a synthetically generated dataset that includes a variety of types and difficulties of math problems; it is both larger and more varied than previous datasets of this type. The dataset is then used to evaluate a number of recurrent models (LSTM, LSTM+attention, transformer); these are very powerful models for general sequence-sequence tasks, but they are not explicitly tailored to math problems. The results are then analyzed and insights are derived explaining where neural models seemingly cope well with math tasks, and where they fall down. 
Strengths: I am happy to see the proposal of a very large dataset with a lot of different axes for measuring and examining the performance of models. There are challenging desiderata involved in building the training+tests sets, and the authors have an interesting and involved methodology to accomplish these. The paper is very clearly written. I'm not aware of a comparable work, so the novelty here seems good.
Weaknesses: The dataset created here is entirely synthetic, and the paper only includes one single small real-world case; it seems like it would be easy to generate a larger and more varied real world dataset as well (possibly from the large literature of extant solved problems in workbooks). It would have been useful to compare the general models here with some specific math problem-focused ones as well. Some details weren't clear to me. More in the comments below.
Verdict: I thought this was generally an interesting paper that has some very nice benefits, but also has some weaknesses that could be resolved. I view it as borderline, but I'm willing to change my mind based on the discussion.
 
 
Comments:
- One area that could stand to be improved is prior work. I'd like to see more of a discussion of *prior data sets* rather than papers proposing models for problems. Since this is the core contribution, this should also be the main comparison. For example, EMLNP 2017 paper "Deep Neural Solver for Math Word Problems" mentions a size 60K problem dataset. A more extensive discussion will help convince the readers that the proposed dataset is indeed the largest and most diverse.
- The authors note that previous datasets are often specific to one type of problem (i.e., single variable equation solving). Why not then combine multiple types of extant problem sets? 
- The authors divide dataset construction into crowdsourcing and synthetic. This seems incomplete to me: there are tens of thousands (probably more) of exercises and problems available in workbooks for elementary, middle, and high school students. These are solved, and only require very limited validation. They are also categorized by difficulty and area. Presumably the cost here would be to physically scan some of these workbooks, but this seems like a very limited investment. Why not build datasets based on workbooks, problem solving books, etc? 
- How do are the difficulty levels synthetically determined?
- When generating the questions, the authors "first sample the answer". What's the distribution you use on the answer? This seems like it dramatically affects the resulting questions, so I'm curious how it's selected.
- The general methodology of generating questions and ensuring that no question is too rare or too frequent and the test set is sufficiently different---these are important questions and I commend the authors for providing a strong methodology.
- I didn't understand the motivation for testing only very general-purpose models (this is described in Section 3). This is certainly a scientific decision, i.e., the authors are determining which models to use in order to determine the possible insights they will derive. But it's not clear to me why testing more sophisticated models that are tailored for math questions would *not* be useful. In fact, assuming that such methods outperform general-purpose models, we could investigate why and where this is the case (in fact the proposed dataset is very useful for this). On the other hand, if these specialized approaches largely fail to outperform general-purpose models, we would have the opposite insights---that these models' benefits are dataset-specific and thus limited. 
- Really would be good to do real-world tests in a more extensive way. A 40-question exam for 16 year olds is probably far too challenging for the current state of general recurrent models. Can you add some additional grades here, and more questions?
- For the number of thinkings steps, how does it scale up as you increase it from 0 to 16? Is there a clear relationships here?
- The 1+1+...+1 example is pretty intriguing, and could be a nice "default" question!
- Minor typo: in the abstract: "test spits" should be "test splits"