Answering questions that require multi-hop reasoning at Web-scale requires retrieving multiple evidence documents, one of which often has little lexical or semantic relationship to the question. This paper introduces a new graphbased recurrent retrieval approach that learns to retrieve reasoning paths over the Wikipedia graph to answer multi-hop open-domain questions. Our retriever trains a recurrent neural network that learns to sequentially retrieve evidence documents in the reasoning path by conditioning on the previously retrieved documents. Our reader ranks the reasoning paths and extracts the answer span included in the best reasoning path. Experimental results show state-of-the-art results in three opendomain QA datasets, showcasing the effectiveness and robustness of our method. Notably, our method achieves significant improvement in HotpotQA, outperforming the previous best model by more than 14 points.
Summary
========
This paper introduces a graph-based recurrent retrieval model for retrieving evidence documents in a multi-hop reasoning question answering task. The main idea is that (1) the graph formed by Wikipedia links between passages can be used as constraint for constructing reasoning chains, and (2) the joint encoding of the question and current passage can be used to retrieve a subsequent passage in the reasoning chain. The paper describes a model for implementing the above retrieval system, and how they jointly train with a reading comprehension model. They demonstrate the effectiveness of the system on HotPotQA, showing improvements over previously published models, and SQuaD-Open, showing competitive results.
Overall Comments
===============
The paper is an interesting, but incremental, improvement to the area of question answering. Overall, there are two main concerns about this work. First, while the results are somewhat strong, the ideas presented are small variations on existing systems. For example, Godbole et al 2019 and Ding et al. 2019 both explore using graphical structural to constraint iterative, multi-hop, retrieval. Also, Feldman et al 2019, describe an encoder based approach to encode question and paragraph context for iterative retrieval. Asides from smaller modeling differences (choice of RNN, training regime, BERT reader, etc.) to account for the difference in results, the main difference seems to be the joint training of the retrieval system with the reader. Secondly, the paper lacks clarity on some formal definitions and definition of the graph, making it hard to understand the content precisely.
Detailed Comments
================
Below are some detailed comments about specific parts of the paper, in order of importance:
1. One important limitation of this technique is the reliance on a linked documents for constructing the retrieval system. It is not clear from the paper how much of the results are obtained from constraining the set of retrieved passages (after the initial retrieval) to Wikipedia links. And whether, for example, substituting Wikipedia links with links derived from an off-the-shelf entity linking system would suffice.
2. Given that the retrieval model is restricted to link structure in Wikipedia that induces the proposed retrieval graph, I assume that there are “reasoning paths” that do not exist in the graph, given Wikipedia’s policy of avoiding adding redundant links within a Wikipedia page. It would have been informative to conduct an “Oracle” experiment: that is, given the initial  set of retrieved nodes and the graph structure, are there *any* paths that provide the correct answer and reasoning chain? That is to say, what is the upper-bound performance on the proposed system given the currently induced Wikipedia graph?
3. In Section 3, and even later on in the paper, it was not clear what “E” denotes. It never seems to be defined, and is used interchangeably with “graph node”, “wikipedia page”, “wikipedia paragraph” and “reasoning path”. Are these the same thing? It would be much clearer to define what E means, and perhaps separate the different concepts (node, passage, reasoning path) properly.
4. In Section 3, it seems that ‘q’ is not defined. Is it the question?
5. In Section 3.1, it is not clear what the graph actually contains. Does it contain all the paragraphs from Wikipedia? Just the paragraphs with links? The first paragraph of every Wikipedia page? What granularity of the wikipedia page becomes an individual node in the graph?
6. In Section 3.1.1., the representation of the starting retrieval (i.e., time-step = 0), h_0, is not defined. Later in the section, the paper mentions the use of TF-IDF for the initial set of nodes, instead of the learned retrieval model. This seems a bit unusual design decision without further explanation. Particularly when taking the results in Table 4, showing TF-IDF based retrieval performs worse that the learned retrieval system from the proposed model.
7. In Section 3, C_{t} (the candidate set of paragraphs) is not defined. This is an important set to define. Is it the set of paragraphs derived from Wikipedia links, starting from the current node?
8. In Section 3.1.2, “Loss function”, the term g_{r} is not defined.
9. In Section 4.4 “Analysis on reasoning path length”, it would have been useful to see the performance of the model with different path lengths. This analysis is somewhat common on multi-hop reasoning tasks, and should be included.
10. Typo in Section 4.4: “..., and out model is likely too terminate …”  should be “ likely to terminate “