Integrating distributed representations with symbolic operations is essential for reading comprehension requiring complex reasoning, such as counting, sorting and arithmetics, but most existing approaches rely on specialized neural modules and are hard to adapt to multiple domains or multi-step reasoning. In this work, we propose the Neural Symbolic Reader (NeRd), which includes a reader, e.g., BERT, to encode the passage and question, and a programmer, e.g., LSTM, to generate a program for multi-step reasoning. By using operators like span selection, the program can be executed over text to generate the answer. Compared to previous works, NeRd is more scalable in two aspects: (1) domain-agnostic, i.e., the same neural architecture works for different domains; (2) compositional, i.e., complex programs can be generated by compositionally applying the symbolic operators. Furthermore, to overcome the challenge of training NeRd with weak supervision, we apply data augmentation techniques and hard ExpectationMaximization (EM) with thresholding. On DROP, a challenging reading comprehension dataset requiring discrete reasoning, NeRd achieves 1.37%/1.18% absolute gain over the state-of-the-art on Exact-Match/F1 metrics. With the same architecture, NeRd significantly outperforms the baselines on MathQA, a math problem benchmark that requires multiple steps of reasoning, by 25.5% absolute gain on accuracy when trained on all the annotated programs, and more importantly, still beats the baselines even with only 20% of the program annotations.
This paper presents a semantic parser that operates over passages of text instead of a structured data source.  This is the first time anyone has demonstrated such a semantic parser (Siva Reddy and several others have essentially used unstructured text as an information source for a semantic parser, similar to OpenIE methods, but this is qualitatively different).  The key insight is to let the semantic parser point to locations in the text that can be used in further symbolic operations.  This is excellent work, and it should definitely be accepted.  I have a ton of questions about this method, but they are good questions.  The rest of this review focuses on things that I thought could be more clear, or that raise new questions, and might sound negative.  Please understand them, however, in terms of my overall score and what I said above.
The three claimed contributions are (1) better numbers, (2) better compositionality / domain applicability, and (3) better interpretability.
(2) and (3) sound a bit like overclaiming in the introduction to me, as there isn't a whole lot of nested composition in the language used by NeRd, and the BERT calculator in principle is almost as compositional and interpretable (also, e.g., NAQANet can add and subtract an arbitrary number of numbers, also, and it tells you which ones they are, just as NeRd does).  Later in the paper the specifics of those claims are made more clear, and while they are justified, they are very narrow claims.  To me, someone who is intimately familiar with this research area, the key contributions (the things that I learned) are (1) using passage-span and key-value predicates actually works, (2) how much difference hard EM and thresholding make, and (3) the data augmentation in this work is pretty clever.  (2) was intuitively clear to me after seeing Dasigi's iterative search paper and Min's hard EM paper, but the difference in results presented here is pretty striking.
Compositionality:
The authors claim that their method is compositional and domain agnostic, while all previous methods had hand-crafted modules for specific question types.  However, I see little reason to believe there's much of a difference here.  You also defined operations that are tailored to the dataset, and are basically identical to the operations that others have used.  I see no evidence that NeRd actually generalizes to program types that are beyond what is captured by other methods.  It's possible that this happens, but there is no evaluation that discusses this, and from all of the examples I'm led to believe that this is basically also just learning a few program templates, the same ones learned by previous methods.  With the weak supervision that you have, are you actually able to find more complex programs during your search?  Some kind of demonstration of actual compositionality on the more complex questions in DROP would make a very strong argument for your claims; without that, they ring a little hollow.
Interpretability:
The use of passage-span as a predicate is really interesting, and it raises a lot of questions.  This predicate lets the model shortcut any interpretable reasoning and do operations entirely inside the encoder/parser.  For example, your first example in table 2 ostensibly requires filtering the numbers in the passage to those that are percentages associated with groups, then filtering them again to those where the percentage is larger than 16, then returning the associated groups. But your method jumps straight to returning a set of passage spans.  This is hardly interpretable.  (In fairness, no prior method provides interpretable reasoning for this kind of operation either.)  But the fact that you have this predicate lets the model do these filters and greater-than comparisons inside the network in an opaque way, while also getting interpretable operations for some questions (table 5 is further confirmation of this, and of the fact that you probably are not capturing many of more the complex, compositional questions in DROP).  But how does the network decide which to do?  Any argmax or max question, and many count questions, could be answered by passage-span alone.  With only weak supervision, and with the parser having the ability to shortcut these more interpretable operations, how often are you actually getting the interpretable one, and what's causing the model to choose it?
Similarly, how often does an argmax or a max operation actually operate on the full set that you would expect it to?  Or does it just do the argmax internal to the network, and output only one item as an argument to the argmax?  If the later, this again hurts your claims of better interpretability over prior methods, as the logic is just as opaque as before.  This also seems like a really hard search problem in how you've set up your DSL - what would make your search over programs actually select all of the correct arguments?  Because you're selecting passage spans directly instead of performing some kind of matching operation, you have to have your search select all of the appropriate spans for this to be "interpretable", and not just hiding the logic inside of the network.  But that seems like a totally intractable search.  You found a clever way to get around this for count questions (even though that still implicitly hides a bunch of filtering logic, as noted above), but I don't know how to make it work for maxes and argmaxes.
Another question raised by the passage-span predicate: the more you use bare passage-span programs for training, the more the network learns to put all of its compositional reasoning inside, in an opaque way, instead of giving you interpretable compositionality.  At one extreme, you end up with something like NABERT (or even less compositional), where basically everything is inside the network.  At the other extreme, where you don't have passage-span, you are left with a crippled semantic parser that can't handle most of the questions.  But using the predicate introduces tension in the model between interpretability and flexibility.  How do we resolve this tension?  (This isn't something I expect your paper to address, it's just a really interesting and important question raised by your work.)
Parser:
Prior work has found benefit in using runtime constraints on parser outputs, or grammar-based decoding.  It looks like you are doing neither of those, yet you're able to output specific token indices and number indices in your programs.  Are you really not doing anything special to handle those?  How does the decoder know token indices?  I feel like something must be missing here, or a simple LSTM decoder is more magical than I thought.
Evaluation:
Why only show results on DROP dev, and not on the test set?  It's possible that your higher numbers are because you were better able to overfit to the dev set, which you presumably used during training.  (I don't think that that's likely, but it's a concern that would be easily avoided by evaluating on test.)