Ask the Right Questions: Active Question Reformulation with Reinforcement Learning
We frame Question Answering (QA) as a Reinforcement Learning task, an approach that we call Active Question Answering. We propose an agent that sits between the user and a black box QA system and learns to reformulate questions to elicit the best possible answers. The agent probes the system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer. The reformulation system is trained end-to-end to maximize answer quality using policy gradient. We evaluate on SearchQA, a dataset of complex questions extracted from Jeopardy!. The agent outperforms a state-of-the-art base model, playing the role of the environment, and other benchmarks. We also analyze the language that the agent has learned while interacting with the question answering system. We find that successful question reformulations look quite different from natural language paraphrases. The agent is able to discover non-trivial reformulation strategies that resemble classic information retrieval techniques such as term re-weighting (tf-idf) and stemming.
 This paper proposes active question answering via a reinforcement learning approach that can learn to rephrase the original questions in a way that can provide the best possible answers. Evaluation on the SearchQA dataset shows significant improvement over the state-of-the-art model that uses the original questions. In general, the paper is well-written (although there are a lot of typos and grammatical errors that need to be corrected), and the main ideas are clear. It would have been useful to provide some more details and carry out additional experiments to strengthen the merit of the proposed model. Especially, in Section 4.2, more details about the quality of paraphrasing after training with the multilingual, monolingual, and refined models would be helpful. Which evaluation metrics were used to evaluate the quality? Also, more monolingual experiments could have been conducted with state-of-the-art neural paraphrasing models on WikiQA and Quora datasets (e.g. see https://arxiv.org/pdf/1610.03098.pdf and https://arxiv.org/pdf/1709.05074.pdf). More details with examples should be provided about the variants of AQA along with the oracle model. Especially, step-by-step examples (for all alternative models) from input (original question) to question reformulations to output (answer/candidate answers) would be useful to understand how each module/variation is having an impact towards the best possible answer/ground truth. Although experiments on SearchQA demonstrate good results, I think it would be also interesting to see the results on additional datasets e.g. MS MARCO (Nguyen et al., 2016), which is very similar to the SearchQA dataset, in order to confirm the generalizability of the proposed approach. --------------------------------------------- Thanks for revising the paper, I am happy to update my scores.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Ask the Right Questions: Active Question Reformulation with Reinforcement Learning
We frame Question Answering (QA) as a Reinforcement Learning task, an approach that we call Active Question Answering. We propose an agent that sits between the user and a black box QA system and learns to reformulate questions to elicit the best possible answers. The agent probes the system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer. The reformulation system is trained end-to-end to maximize answer quality using policy gradient. We evaluate on SearchQA, a dataset of complex questions extracted from Jeopardy!. The agent outperforms a state-of-the-art base model, playing the role of the environment, and other benchmarks. We also analyze the language that the agent has learned while interacting with the question answering system. We find that successful question reformulations look quite different from natural language paraphrases. The agent is able to discover non-trivial reformulation strategies that resemble classic information retrieval techniques such as term re-weighting (tf-idf) and stemming.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposes active question answering via a reinforcement learning approach that can learn to rephrase the original questions in a way that can provide the best possible answers. Evaluation on the SearchQA dataset shows significant improvement over the state-of-the-art model that uses the original questions. In general, the paper is well-written (although there are a lot of typos and grammatical errors that need to be corrected), and the main ideas are clear. It would have been useful to provide some more details and carry out additional experiments to strengthen the merit of the proposed model. Especially, in Section 4.2, more details about the quality of paraphrasing after training with the multilingual, monolingual, and refined models would be helpful. Which evaluation metrics were used to evaluate the quality? Also, more monolingual experiments could have been conducted with state-of-the-art neural paraphrasing models on WikiQA and Quora datasets (e.g. see https://arxiv.org/pdf/1610.03098.pdf and https://arxiv.org/pdf/1709.05074.pdf). More details with examples should be provided about the variants of AQA along with the oracle model. Especially, step-by-step examples (for all alternative models) from input (original question) to question reformulations to output (answer/candidate answers) would be useful to understand how each module/variation is having an impact towards the best possible answer/ground truth. Although experiments on SearchQA demonstrate good results, I think it would be also interesting to see the results on additional datasets e.g. MS MARCO (Nguyen et al., 2016), which is very similar to the SearchQA dataset, in order to confirm the generalizability of the proposed approach. --------------------------------------------- Thanks for revising the paper, I am happy to update my scores.
assistant
 This paper proposes active question answering via a reinforcement learning approach that can learn to rephrase the original questions in a way that can provide the best possible answers. Evaluation on the SearchQA dataset shows significant improvement over the state-of-the-art model that uses the original questions. In general, the paper is well-written (although there are a lot of typos and grammatical errors that need to be corrected), and the main ideas are clear. It would have been useful to provide some more details and carry out additional experiments to strengthen the merit of the proposed model. Especially, in Section 4.2, more details about the quality of paraphrasing after training with the multilingual, monolingual, and refined models would be helpful. Which evaluation metrics were used to evaluate the quality? Also, more monolingual experiments could have been conducted with state-of-the-art neural paraphrasing models on WikiQA and Quora datasets (e.g. see https://arxiv.org/pdf/1610.03098.pdf and https://arxiv.org/pdf/1709.05074.pdf). More details with examples should be provided about the variants of AQA along with the oracle model. Especially, step-by-step examples (for all alternative models) from input (original question) to question reformulations to output (answer/candidate answers) would be useful to understand how each module/variation is having an impact towards the best possible answer/ground truth. Although experiments on SearchQA demonstrate good results, I think it would be also interesting to see the results on additional datasets e.g. MS MARCO (Nguyen et al., 2016), which is very similar to the SearchQA dataset, in order to confirm the generalizability of the proposed approach. --------------------------------------------- Thanks for revising the paper, I am happy to update my scores.