Published as a conference paper at ICLR 2018 ASK THE RIGHT QUESTIONS: ACTIVE QUESTION REFORMULATION WITH REINFORCEMENT LEARNING
We frame Question Answering (QA) as a Reinforcement Learning task, an approach that we call Active Question Answering. We propose an agent that sits between the user and a black box QA system and learns to reformulate questions to elicit the best possible answers. The agent probes the system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer. The reformulation system is trained end-to-end to maximize answer quality using policy gradient. We evaluate on SearchQA, a dataset of complex questions extracted from Jeopardy!. The agent outperforms a stateof-the-art base model, playing the role of the environment, and other benchmarks. We also analyze the language that the agent has learned while interacting with the question answering system. We find that successful question reformulations look quite different from natural language paraphrases. The agent is able to discover non-trivial reformulation strategies that resemble classic information retrieval techniques such as term re-weighting (tf-idf) and stemming.
This paper formulates the Jeopardy QA as a query reformulation task that leverages a search engine.  In particular, a user will try a sequence of alternative queries based on the original question in order to find the answer.  The RL formulation essentially tries to mimic this process.  Although this is an interesting formulation, as promoted by some recent work, this paper does not provide compelling reasons why it's a good formulation.  The lack of serious comparisons to baseline methods makes it hard to judge the value of this work.
Detailed comments/questions:
	1. I am actually quite confused on why it's a good RL setting. For a human user, having a series of queries to search for the right answer is a natural process, but it's not natural for a computer program.  For instance, each query can be viewed as different formulation of the same question and can be issued concurrently. Although formulated as an RL problem,  it is not clear to me whether the search result after each episode has been used as the immediate environment feedback. As a result, the dependency between actions seems rather weak.
	2. I also feel that the comparisons to other baselines (not just the variation of the proposed system) are not entirely fair. For instance, the baseline BiDAF model has only one shot, namely using the original question as query.  In this case, AQA should be allowed to use the same budget -- only one query.  Another more realistic baseline is to follow the existing work on query formulation in the IR community.  For example, 20 shorter queries generated by methods like [1] can be used to compare the queries created by AQA.
[1] Kumaran & Carvalho. "Reducing Long Queries Using Query Quality Predictors". SIGIR-09
	
Pros:
	1. An interesting RL formulation for query reformulation
Cons:
	1. The use of RL is not properly justified
	2. The empirical result is not convincing that the proposed method is indeed advantageous 
---------------------------------------
After reading the author response and checking the revised paper, I'm both delighted and surprised that the authors improved the submission substantially and presented stronger results.  I believe the updated version has reached the bar and recommend accepting this paper.