MULTI-STEP RETRIEVER-READER INTERACTION FOR SCALABLE OPEN-DOMAIN QUESTION ANSWERING
This paper introduces a new framework for open-domain question answering in which the retriever and the reader iteratively interact with each other. The framework is agnostic to the architecture of the machine reading model, only requiring access to the token-level hidden representations of the reader. The retriever uses fast nearest neighbor search to scale to corpora containing millions of paragraphs. A gated recurrent unit updates the query at each step conditioned on the state of the reader and the reformulated query is used to re-rank the paragraphs by the retriever. We conduct analysis and show that iterative interaction helps in retrieving informative paragraphs from the corpus. Finally, we show that our multistep-reasoning framework brings consistent improvement when applied to two widely used reader architectures (DR. QA and BIDAF)  on various large open-domain datasets -TRIVIAQA-unfiltered, QUASAR-T, SEARCHQA, and SQUAD-open 1 .
This paper introduces a new framework to interactively interact document retriever and reader for open-domain question answering. While retriever-reader framework was often used for open-domain QA, this bi-directional interaction between the retriever and the reader is novel and effective because
1) If the retriever fails to retrieve the right document at the first step, the reader can give a signal to the retriever so that the retriever can recover its mistake at the next step
2) The idea of `reader state` from the reader to the retriever is new
3) The retriever use question-independent representation of paragraphs, which does not require different representation depending on the question and makes the framework easily scalable.
Strengths
1) The idea of multi-step & bi-directional interaction between the retriever and the reader is novel enough (as mentioned above). The paper contains enough literature studies on existing retriever-reader framework in open-domain setting, and clearly demonstrates how their framework is different from them.
2) The authors run the experiments on 4 different dataset, which supports the argument about the framework’s effectiveness.
Weakness
1) The authors seem to highlight multi-step `reasoning`, while it is not `reasoning` in my opinion. Multi-step reasoning refers to the task which you need evidence from different documents, and/or you need to find first evident to find the second evidence from a different document. I don’t think the dataset here are not multi-step reasoning dataset, and the authors seem not to claim it either. Therefore, I recommend using another term (maybe `multi-step interaction`?) instead of `multi-step reasoning`.
2) While the idea of multi-step interaction and how it benefits the overall performance is interesting, the analysis is not enough. Figure 3 in the paper does not have enough description — for example, I got the left example means step 2 recovers the mistake from step 1, but what does the right example mean?
Questions on result comparison
1) On TriviaQA (both open and full), the authors mentioned the result is on hidden test set — did you submit it to the leaderboard? I don’t see the same numbers on the TriviaQA leaderboard. Also, the authors claim they are SOTA on TriviaQA, but there are higher numbers on the leaderboard (which are submitted prior to the ICLR deadline).
2) There are other published papers with higher result on Quasar-T, SearchQA and TriviaQA (such as https://aclanthology.info/papers/P18-1161/p18-1161 and https://arxiv.org/abs/1805.08092) which the authors did not compare with.
3) In Section 4.2, is there a reason for the specific comparison to AQA (5th line), though AQA is not SOTA on SearchQA? I don’t think it means latent space is better than natural language space. They are totally different model and the only intersection is they contains interaction between two submodules.
4) In Section 5, the authors mentioned their framework outperforms previous SOTA by 15% margin on TriviaQA, but what is that? I don’t see 15% margin in Table 2.
Marginal comments:
1) If I understood correctly, `TriviaQA-open` and `TriviaQA-full` in the paper are officially called `TriviaQA-full` and `open-domain TriviaQA`. How about changing the term for readers to better understand the task? Also, in Section 4, the authors said TriviaQA-open is larger than web/wiki setting, but to my knowledge, this setting is part of the wiki setting.
2) It would be great if the authors make the capitalization consistent. e.g. EM, Quasar-T, BiDAF. Also, the authors can use EM instead of `exact match` after they mentioned EM refers to exact match in Section 4.2.
Overall comment
The idea in the paper is interesting, and their model and experiments are concrete. My only worries is that the terms in the paper are confusing and performance comparison are weak. I would like to update the score when the authors update the paper.
Update 11/27/2018
Thanks for the authors for updating the paper. The updated paper have more clear comparisons with other models, with more & stronger experiments with the additional dataset. Also, the model is claimed to perform multi-step interaction rather than multi-step reasoning, which clearly resolves my initial concern. The analysis, especially ablations in varying number of iterations, was helpful to understand how their framework benefits. I believe these make the paper stronger along with its initial novelty in the framework. In this regard, I vote for acceptance.