ElimiNet: A Model for Eliminating Options for Reading Comprehension with Multiple Choice Questions
The task of Reading Comprehension with Multiple Choice Questions, requires a human (or machine) to read a given {	extit{passage, question}} pair and select one of the given options. The current state of the art model for this task first computes a query-aware representation for the passage and then 	extit{selects} the option which has the maximum similarity with this representation. However, when humans perform this task they do not just focus on option selection but use a combination of 	extit{elimination} and 	extit{selection}. Specifically, a human would first try to eliminate the most irrelevant option and then read the document again in the light of this new information (and perhaps ignore portions corresponding to the eliminated option). This process could be repeated multiple times till the reader is finally ready to select the correct option. We propose 	extit{ElimiNet}, a neural network based model which tries to mimic this process. Specifically, it has gates which decide whether an option can be eliminated given the {	extit{document, question}} pair and if so it tries to make the document representation orthogonal to this eliminatedd option (akin to ignoring portions of the document corresponding to the eliminated option). The model makes multiple rounds of partial elimination to refine the document representation and finally uses a selection module to pick the best option. We evaluate our model on the recently released large scale RACE dataset and show that it outperforms the current state of the art model on 7 out of the 13 question types in this dataset. Further we show that taking an ensemble of our 	extit{elimination-selection} based method with a 	extit{selection} based method gives us an improvement of 7\% (relative) over the best reported performance on this dataset.
 This paper proposes a new reading comprehension model for multi-choice questions and the main motivation is that some options should be eliminated first to infer better passage/question representations. It is a well-written paper, however, I am not very convinced by its motivation, the proposed model and the experimental results. First of all, the improvement is rather limited. It is only 0.4 improvement overall on the RACE dataset; although it outperforms GAR on 7 out of 13 categories; but why is it worse on the other 6 categories? I don’t see any convincing explanations here. Secondly, in terms of the development of reading comprehension models, I don’t see why we need to care about eliminating the irrelevant options. It is hard to generalize to any other RC/QA tasks. If the point is that the options can add useful information to induce better representations for passage/question, there should be some simple baselines in the middle that this paper should compare to. The two baselines SAR and GAR both only induce a representation from paragraph/question, and finally compare to the representation of each option. Maybe a simple baseline is to merge the question and all the options and see if a better document representation can be defined. Some visualizations/motivational examples could be also useful to understand how some options are eliminated and how the document representation has been changed based on that.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
ElimiNet: A Model for Eliminating Options for Reading Comprehension with Multiple Choice Questions
The task of Reading Comprehension with Multiple Choice Questions, requires a human (or machine) to read a given {	extit{passage, question}} pair and select one of the given options. The current state of the art model for this task first computes a query-aware representation for the passage and then 	extit{selects} the option which has the maximum similarity with this representation. However, when humans perform this task they do not just focus on option selection but use a combination of 	extit{elimination} and 	extit{selection}. Specifically, a human would first try to eliminate the most irrelevant option and then read the document again in the light of this new information (and perhaps ignore portions corresponding to the eliminated option). This process could be repeated multiple times till the reader is finally ready to select the correct option. We propose 	extit{ElimiNet}, a neural network based model which tries to mimic this process. Specifically, it has gates which decide whether an option can be eliminated given the {	extit{document, question}} pair and if so it tries to make the document representation orthogonal to this eliminatedd option (akin to ignoring portions of the document corresponding to the eliminated option). The model makes multiple rounds of partial elimination to refine the document representation and finally uses a selection module to pick the best option. We evaluate our model on the recently released large scale RACE dataset and show that it outperforms the current state of the art model on 7 out of the 13 question types in this dataset. Further we show that taking an ensemble of our 	extit{elimination-selection} based method with a 	extit{selection} based method gives us an improvement of 7\% (relative) over the best reported performance on this dataset.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposes a new reading comprehension model for multi-choice questions and the main motivation is that some options should be eliminated first to infer better passage/question representations. It is a well-written paper, however, I am not very convinced by its motivation, the proposed model and the experimental results. First of all, the improvement is rather limited. It is only 0.4 improvement overall on the RACE dataset; although it outperforms GAR on 7 out of 13 categories; but why is it worse on the other 6 categories? I don’t see any convincing explanations here. Secondly, in terms of the development of reading comprehension models, I don’t see why we need to care about eliminating the irrelevant options. It is hard to generalize to any other RC/QA tasks. If the point is that the options can add useful information to induce better representations for passage/question, there should be some simple baselines in the middle that this paper should compare to. The two baselines SAR and GAR both only induce a representation from paragraph/question, and finally compare to the representation of each option. Maybe a simple baseline is to merge the question and all the options and see if a better document representation can be defined. Some visualizations/motivational examples could be also useful to understand how some options are eliminated and how the document representation has been changed based on that.
assistant
 This paper proposes a new reading comprehension model for multi-choice questions and the main motivation is that some options should be eliminated first to infer better passage/question representations. It is a well-written paper, however, I am not very convinced by its motivation, the proposed model and the experimental results. First of all, the improvement is rather limited. It is only 0.4 improvement overall on the RACE dataset; although it outperforms GAR on 7 out of 13 categories; but why is it worse on the other 6 categories? I don’t see any convincing explanations here. Secondly, in terms of the development of reading comprehension models, I don’t see why we need to care about eliminating the irrelevant options. It is hard to generalize to any other RC/QA tasks. If the point is that the options can add useful information to induce better representations for passage/question, there should be some simple baselines in the middle that this paper should compare to. The two baselines SAR and GAR both only induce a representation from paragraph/question, and finally compare to the representation of each option. Maybe a simple baseline is to merge the question and all the options and see if a better document representation can be defined. Some visualizations/motivational examples could be also useful to understand how some options are eliminated and how the document representation has been changed based on that.