Neural Symbolic Reader: Scalable Integration of Distributed and Symbolic Representations for Reading Comprehension
Integrating distributed representations with symbolic operations is essential for reading comprehension requiring complex reasoning, such as counting, sorting and arithmetics, but most existing approaches are hard to scale to more domains or more complex reasoning. In this work, we propose the Neural Symbolic Reader (NeRd), which includes a reader, e.g., BERT, to encode the passage and question, and a programmer, e.g., LSTM, to generate a program that is executed to produce the answer. Compared to previous works, NeRd is more scalable in two aspects: (1) domain-agnostic, i.e., the same neural architecture works for different domains; (2) compositional, i.e., when needed, complex programs can be generated by recursively applying the predefined operators, which become executable and interpretable representations for more complex reasoning. Furthermore, to overcome the challenge of training NeRd with weak supervision, we apply data augmentation techniques and hard Expectation-Maximization (EM) with thresholding. On DROP, a challenging reading comprehension dataset that requires discrete reasoning, NeRd achieves 2.5%/1.8% absolute improvement over the state-of-the-art on EM/F1 metrics. With the same architecture, NeRd significantly outperforms the baselines on MathQA, a math problem benchmark that requires multiple steps of reasoning, by 25.5% absolute increment on accuracy when trained on all the annotated programs. More importantly, NeRd still beats the baselines even when only 20% of the program annotations are given.
-- define cascade errors when you first use the phrase -- basic english grammar could be fixed but is not interfering with understanding -- what is the early stopping criterion in Alg 1? -- did you try any other values for the initial threshold and decay factor? -- At the end of Section 4.1 DROP: *, but not the same.* - I can*t parse what this last clause is supposed to mean. --the diff sum example in table 2 is confusing; the program appears to sum up the numbers but the result is a subtraction without a sum operation in it. Would be clearer to show the sum in the result line as well rather than distribute the subtraction. Also, shouldn*t it be diff(9, sum(10, 12))? -- I think you should pull at least some commentary about the constant used in Table 3 from Appendix B and include it in the main paper (or at least mention Appendix B is the place to look). Can you add a table in an appendix showing the complete list of operators used? -- Nice results in Table 4 on the dev set. Are there Test set results as well? -- The organization of the Baselines 4.3 section and the Results 4.4 is confusing. For example, you mention that you test different variants of NeRd, operator variants, and mathqa, but then the results are not mentioned for these experiments until the next page. I found myself immediately looking for the numbers/results when you introduce the experiment. I would pair your experiment description with the results rather than grouping all experiment descriptions and then grouping all results, especially when the order of the experiment descriptions does not match the order of the results presented. For example, in baselines you discuss training variants and then operators. Then in Results you discuss operators before variants. It is too disconnected and makes the reader jump around a bunch. Same goes for the drop baselines where you mention a bunch of models, and I would prefer the Results/discussion paired with each one, rather than having to wait for it down below. -- Overall it seems like a solid work; good empirical results showing improvements of each purported contribution. The model itself is a relatively simple construction of basic component, but the combination with the DSL is intuitive and makes sense. I don*t think the novelty in model here is the main selling point anyways; the training variants and the demonstration of how well a DSL approach can do combined with previously introduced methods. -- I find the model description to be slightly unclear. In Fig 1 for example there is an arrow that connects passage to compositional programs. What does that arrow represent? I think you should elaborate on how the attention over the encoded text interacts with the attention over previously generated tokens. Equations would make this far more explicit as is I am left with a lot of questions on how to implement your model. Maybe you can add to your appendix? Or release your code? That is mentioned either.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Neural Symbolic Reader: Scalable Integration of Distributed and Symbolic Representations for Reading Comprehension
Integrating distributed representations with symbolic operations is essential for reading comprehension requiring complex reasoning, such as counting, sorting and arithmetics, but most existing approaches are hard to scale to more domains or more complex reasoning. In this work, we propose the Neural Symbolic Reader (NeRd), which includes a reader, e.g., BERT, to encode the passage and question, and a programmer, e.g., LSTM, to generate a program that is executed to produce the answer. Compared to previous works, NeRd is more scalable in two aspects: (1) domain-agnostic, i.e., the same neural architecture works for different domains; (2) compositional, i.e., when needed, complex programs can be generated by recursively applying the predefined operators, which become executable and interpretable representations for more complex reasoning. Furthermore, to overcome the challenge of training NeRd with weak supervision, we apply data augmentation techniques and hard Expectation-Maximization (EM) with thresholding. On DROP, a challenging reading comprehension dataset that requires discrete reasoning, NeRd achieves 2.5%/1.8% absolute improvement over the state-of-the-art on EM/F1 metrics. With the same architecture, NeRd significantly outperforms the baselines on MathQA, a math problem benchmark that requires multiple steps of reasoning, by 25.5% absolute increment on accuracy when trained on all the annotated programs. More importantly, NeRd still beats the baselines even when only 20% of the program annotations are given.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



-- define cascade errors when you first use the phrase -- basic english grammar could be fixed but is not interfering with understanding -- what is the early stopping criterion in Alg 1? -- did you try any other values for the initial threshold and decay factor? -- At the end of Section 4.1 DROP: *, but not the same.* - I can*t parse what this last clause is supposed to mean. --the diff sum example in table 2 is confusing; the program appears to sum up the numbers but the result is a subtraction without a sum operation in it. Would be clearer to show the sum in the result line as well rather than distribute the subtraction. Also, shouldn*t it be diff(9, sum(10, 12))? -- I think you should pull at least some commentary about the constant used in Table 3 from Appendix B and include it in the main paper (or at least mention Appendix B is the place to look). Can you add a table in an appendix showing the complete list of operators used? -- Nice results in Table 4 on the dev set. Are there Test set results as well? -- The organization of the Baselines 4.3 section and the Results 4.4 is confusing. For example, you mention that you test different variants of NeRd, operator variants, and mathqa, but then the results are not mentioned for these experiments until the next page. I found myself immediately looking for the numbers/results when you introduce the experiment. I would pair your experiment description with the results rather than grouping all experiment descriptions and then grouping all results, especially when the order of the experiment descriptions does not match the order of the results presented. For example, in baselines you discuss training variants and then operators. Then in Results you discuss operators before variants. It is too disconnected and makes the reader jump around a bunch. Same goes for the drop baselines where you mention a bunch of models, and I would prefer the Results/discussion paired with each one, rather than having to wait for it down below. -- Overall it seems like a solid work; good empirical results showing improvements of each purported contribution. The model itself is a relatively simple construction of basic component, but the combination with the DSL is intuitive and makes sense. I don*t think the novelty in model here is the main selling point anyways; the training variants and the demonstration of how well a DSL approach can do combined with previously introduced methods. -- I find the model description to be slightly unclear. In Fig 1 for example there is an arrow that connects passage to compositional programs. What does that arrow represent? I think you should elaborate on how the attention over the encoded text interacts with the attention over previously generated tokens. Equations would make this far more explicit as is I am left with a lot of questions on how to implement your model. Maybe you can add to your appendix? Or release your code? That is mentioned either.
assistant
-- define cascade errors when you first use the phrase -- basic english grammar could be fixed but is not interfering with understanding -- what is the early stopping criterion in Alg 1? -- did you try any other values for the initial threshold and decay factor? -- At the end of Section 4.1 DROP: *, but not the same. * - I can*t parse what this last clause is supposed to mean. --the diff sum example in table 2 is confusing; the program appears to sum up the numbers but the result is a subtraction without a sum operation in it. Would be clearer to show the sum in the result line as well rather than distribute the subtraction. Also, shouldn*t it be diff(9, sum(10, 12))? -- I think you should pull at least some commentary about the constant used in Table 3 from Appendix B and include it in the main paper (or at least mention Appendix B is the place to look). Can you add a table in an appendix showing the complete list of operators used? -- Nice results in Table 4 on the dev set. Are there Test set results as well? -- The organization of the Baselines 4.3 section and the Results 4.4 is confusing. For example, you mention that you test different variants of NeRd, operator variants, and mathqa, but then the results are not mentioned for these experiments until the next page. I found myself immediately looking for the numbers/results when you introduce the experiment. I would pair your experiment description with the results rather than grouping all experiment descriptions and then grouping all results, especially when the order of the experiment descriptions does not match the order of the results presented. For example, in baselines you discuss training variants and then operators. Then in Results you discuss operators before variants. It is too disconnected and makes the reader jump around a bunch. Same goes for the drop baselines where you mention a bunch of models, and I would prefer the Results/discussion paired with each one, rather than having to wait for it down below. -- Overall it seems like a solid work; good empirical results showing improvements of each purported contribution. The model itself is a relatively simple construction of basic component, but the combination with the DSL is intuitive and makes sense. I don*t think the novelty in model here is the main selling point anyways; the training variants and the demonstration of how well a DSL approach can do combined with previously introduced methods. -- I find the model description to be slightly unclear. In Fig 1 for example there is an arrow that connects passage to compositional programs. What does that arrow represent? I think you should elaborate on how the attention over the encoded text interacts with the attention over previously generated tokens. Equations would make this far more explicit as is I am left with a lot of questions on how to implement your model. Maybe you can add to your appendix? Or release your code? That is mentioned either.