Learning multi-hop reasoning has been a key challenge for reading comprehension models, leading to the design of datasets that explicitly focus on it. Ideally, a model should not be able to perform well on a multi-hop question answering task without doing multi-hop reasoning. In this paper, we investigate two recently proposed datasets, WikiHop (Welbl et al., 2018) and HotpotQA (Yang et al., 2018). First, we explore sentence-factored models for these tasks; by design, these models cannot do multi-hop reasoning, but they are still able to solve a large number of examples in both datasets. Furthermore, we find spurious correlations in the unmasked version of WikiHop, which make it easy to achieve high performance considering only the questions and answers. Finally, we investigate one key difference between these datasets, namely span-based vs. multiple-choice formulations of the QA task. Multiple-choice versions of both datasets can be easily gamed, and two models we examine only marginally exceed a baseline in this setting. Overall, while these datasets are useful testbeds, high-performing models may not be learning as much multi-hop reasoning as previously thought. Question answering from text (Richardson et al., 2013; Hill et al., 2015; Hermann et al., 2015; Ra-jpurkar et al., 2016) is a key challenge problem for NLP that tests whether models can extract information based on a query. However, even sophisticated models that perform well on QA benchmarks (Seo et al., 2017; Shen et al., 2017; Yu et al., 2018) may only be doing shallow pattern matching of the question against the supporting passage (Weissenborn et al., 2017). More recent work (Ku-mar et al., 2016; Joshi et al., 2017; Welbl et al., 2018) has emphasized gathering information from different parts of a passage to answer the question, leading to a number of models designed to do multi-hop reasoning . Two recent large-scale datasets have been specifically designed to test multi-hop reasoning: WikiHop (Welbl et al., 2018) and HotpotQA (Yang et al., 2018). In this paper, we seek to answer two main questions. First, although the two datasets are explicitly constructed for multi-hop reasoning, do models really need to do multi-hop reasoning to do well on them? Recent work has shown that large-scale QA datasets often do not exhibit their advertised properties (Chen et al., 2016; Kaushik and Lipton, 2018). We devise a test setting to see whether multi-hop reasoning is necessary: can a model which treats each sentence independently select the sentence containing the answer? This provides a rough estimate of the fraction of questions solvable by a non-multi-hop system. Our results show that more than half of the questions in WikiHop and HotpotQA do not require multihop reasoning to solve. Surprisingly, we find that a simple baseline which ignores the passage and only uses the question and answer can achieve strong results on WikiHop and a modified version of HotpotQA, further confirming this view. Second, we study the nature of the supervision on the two datasets. One critical difference is that HotpotQA is span-based (the answer is a span of the passage) while WikiHop is multiple-choice. How does this difference affect learning and evaluation of multi-hop reasoning systems? We show that a multiple-choice version of HotpotQA is vulnerable to the same baseline that performs well on WikiHop, showing that this distinction may be important from an evaluation standpoint. Furthermore, we show that a state-of-the-art model, BiDAF++, trained on span-based HotpotQA and adapted to the multiple-choice setting outperforms the same model trained natively on the multiple-choice setting. However, even in the span-based setting, the high performance of the sentence-factored models raises questions about whether multi-hop reasoning is being learned. Our conclusions are as follows: (1) Many examples in both WikiHop and HotpotQA do not require multi-hop reasoning to solve, as the sentence-factored model can find the answers. (2) On WikiHop and a multiple-choice version of HotpotQA, a no context baseline does very well. (3) Span-based supervision provides a harder testbed than multiple choice by having more answers to choose from, but given the strong performance of the sentence-factored models, it is unclear whether any of the proposed models are doing a good job at multi-hop reasoning in any setting. WikiHop Welbl et al. (2018) introduced this English dataset specially designed for text understanding across multiple documents. The dataset consists of 40k+ questions, answers, and passages, where each passage consists of several documents collected from Wikipedia. Questions are posed as a query of a relation r followed by a head entity h , with the task being to find the tail entity t from a set of entity candidates E . Annotators followed links between documents and were required to use multiple documents to get the answer. HotpotQA Yang et al. (2018) proposed a new dataset with 113k English Wikipedia-based question-answer pairs. The questions are diverse, falling into several categories, but all require finding and reasoning over multiple supporting documents to answer. Models should choose answers by selecting variable-length spans from these documents. Sentences relevant to finding the answer are annotated in the dataset as supporting facts so models can use these at training time as well. In this section, we seek to answer whether multihop reasoning is really needed to solve these two multi-hop datasets. If a question requires a multi-hop model, then we should not be able to figure out the answer by only looking at the question and each sentence separately. Based on this idea, we propose a sentence-factored modeling setting, where Method Random Factored Factored BiDAF WikiHop 6.5 60.9 66.1 HotpotQA 5.4 45.4 57.2 SQuAD 22.1 70.0 88.0 Table 1: The accuracy of our proposed sentence-factored models on identifying answer location in the development sets of WikiHop, HotpotQA and SQuAD. a model must predict which sentence contains the answer but must score each sentence independently, i.e., without using information from other sentences in this process. Identifying the presence of the answer is generally easier than predicting the answer directly, particularly if a sentence is complicated, and is still sufficient to provide a bound on how strongly multi-hop reasoning is required. Figure 1 shows a typical example from these datasets, where identifying the answer ( Delhi ) requires bridging to an entity not mentioned in the question. Simple Factored Model We encode each passage sentence s i and the question q into a contextual representation h s i and h q using a bidirectional GRU (Chung et al., 2014). Then, S i = h (cid:62) s i W h q ; that is, compute a bilinear product of these representations with trainable weights W to get the score of the i th sentence. Finally, let p i = softmax i ( S i ) ; softmax over the sentences to get a probability distribution. We maximize the marginal log probability of picking a sentence containing the correct answer: log( (cid:80) i : s i  s  p i ) , where s  is the set of sentences containing the answer. During evaluation, we pick the sentence s with the highest score and treat it as correct if it contains the answer. Factored BiDAF We encode the question and each sentence separately using bi-GRUs. Then, we generate the question-aware token representation for each token of sentence by using a co-attention layer (Seo et al., 2017). Finally, we max-pool over each sentence to get the sentence representation and feed those to a FFNN to compute the sentence score. Training and inference are the same as for the simple model. We run this test on both datasets as well as Question: The Oberoi family is part of a hotel company that has a head office in what city? The Oberoi family is an Indian family that is famous for its involvement in hotels, namely through The Oberoi Group . (cid:1)(cid:1) (cid:1)(cid:1) Figure 1: An example from the HotpotQA dev set. SQuAD (Rajpurkar et al., 2016), where multi-hop reasoning is only needed in a few questions. Results in Table 1 indicate that although intentionally created for multi-hop reasoning, for more than half of questions in WikiHop and HotpotQA, we can figure out where the answer is without doing multi-hop reasoning. This result is initially surprising, but one reason it may be possible is suggested by the example from HotpotQA shown in Figure 1. We can see that the model could easily figure out the answer sentence without looking at the bridging entities using lexical cues alone. This observation is also in accordance with the work of Jansen (2018), which demonstrates that high performance for a simple baseline can be achieved in cases when passages have increasing lexical overlap with the question. We note that this method potentially overestimates performance of a non-multi-hop model on HotpotQA, since there are some examples where many plausible answers are in the same sentence and require other context to resolve. However, these still form a minority in the dataset (see Table 3 of Yang et al. (2018)). The results of the previous section show that a model can identify correspondences between questions and answer sentences. One other pair of correlations we can study is suggested in the work of Kaushik and Lipton (2018), namely examining question-answer correlations independent of the passage. We construct a no context baseline to verify whether it is possible to pick the correct answer without consulting the passage. In a sim-National autonomous university of Mexico Arte, Capital, Life, Monterrey, School, Time Employer Gilberto Aceves Navarro Other Candidates Answer Question Bi-GRU Bi-GRU Bi-linear Dot Figure 2: An example of question and candidates from WikiHop. ilar fashion to the factored model, we encode the query q and each answer candidate c i using a bi-GRU and once again compute a bilinear product between them to get the scores over candidates, making no reference to the document. Results of this model on the multiple-choice WikiHop dataset are shown in Table 2. Surprisingly, the no-context baseline achieves high performance, comparable to some recently-published systems, showing that WikiHop is actually possible to solve reasonably well without using the document at all. One possible reason for this is that this model can filter possible answers based on expected answer type (Sugawara et al., 2018), as shown in the example of Figure 2, or perhaps capture other correlations between training and test. This model substantially outperforms the unlearned baseline reported in the WikiHop paper (Welbl et al., 2018) (38.8%) as well as the BiDAF (Seo et al., 2017) results reported there (42.9%). The no context model indicates that having multiple-choice questions may provide an avenue for a dataset to be gamed. In order to investigate the difference in multiple-choice vs. span supervision while controlling for other aspects of dataset difficulty, we first recast each dataset in the other's framework, then investigate the performance of two models each of these settings. To modify Hotpot to be multiple-choice, we Dataset HotpotQA-MC WikiHop-MC Metric Accuracy Accuracy NoContext 68.01 59.70 MC-BiDAF++ 70.01 61.32 MC-MemNet 68.75 61.80 Span2MC-BiDAF++ 76.01 59.85 Table 3: The performance of different models on the dev sets of WikiHop and HotpotQA. randomly select 9 entities in all of the documents as distractors, and add the answer to make a 10-choice candidates set. To modify WikiHop to be span-based, we concatenate all documents and treat the first appearance of the answer mention as the gold span for training. Any answer occurrence is treated as correct for evaluation. MemNet Memory networks (Weston et al., 2015) define a generic model class which can gather information from different parts of the passage. Kumar et al. (2016) and Miller et al. (2016) have demonstrated its effectiveness in certain multi-hop settings. These models process a document over several timesteps. On the i th step, the model takes a question representation q i , attends to the context representation p , gets an attention distribution  i , computes a new memory cell value m i = (cid:80)  i p i , then forms an updated q i +1 = f ( m i , q i ) . The final memory cell m T is used to compute a score s i = g ( m T , c j ) with the j th candidate representation c j . We modify this architecture slightly using a standard hierarchical attention module (Li et al., 2015). We can also modify this architecture to predict an answer span  we use the memory cell m T of the last step, and do a bi-linear product with the context representation p to compute a distribution over start points P start = softmax ( p W start m T ) and end points distribution P end = softmax ( p W end m T ) of the answer span, where W start and W end are two parameter matrix to be learned. We call this Span-MemNet. BiDAF++ Recently proposed by Clark and Gardner (2018), this is a high-performing model on SQuAD. It combines the bi-directional attention flow (Seo et al., 2017) and self-attention Dataset HotpotQA-Span WikiHop-Span Metric EM F1 EM F1 BiDAF++ (Yang+ 18) 42.79 56.19   Span-BiDAF++ 42.45 56.46 24.23 46.13 Span-MemNet 18.75 26.11 13.54 19.23 Table 4: The performance of different models on the dev sets of WikiHop and HotpotQA. mechanisms. We use the implementation described in Yang et al. (2018). We can modify this model for the multiple-choice setting as well. Specifically, we use the start P start and end P end distribution to do a weighted sum over the context p to get a summarized representation D start = (cid:80) P start i p i , D end = (cid:80) P end i p i of the context. Then we concatenate them to do a bilinear dot product with each candidate representation to get the answer score as we described for MemNet. We call this model MC-BiDAF++. Table 3 and Table 4 show our results in the two settings. As a baseline on multiple-choice HotpotQA, we also test the no-context baseline, which achieves an accuracy of 68.01%, around 10% absolute higher than on WikiHop. Our candidates were randomly chosen, so this setting may not be quite as challenging as a well-constructed multiple-choice dataset. From Table 3 and Table 4 we draw the following conclusions. When trained and tested on multiple-choice datasets, our models do not learn multi-hop reasoning. Comparing MC-BiDAF++ and MC-MemNet on the multiple-choice setting of both datasets as shown in Table 3, the models appear to have similar capabilities to learn multi-hop reasoning. However, looking at the no-context baseline for comparison, we find that it is only around 2% lower than the two relatively more complex models. This indicates that much of the performance is achieved by cheating through the correlation between the candidates and question/context. Surprisingly, this is true even for HotpotQA, which seems stronger based on the analysis in Table 1. Span-based data is less hackable, but models still may not be doing multi-hop reasoning. We then compare the results of Span-BiDAF++ E x ac t M a t c h S c o r e 30 40 50 60 70 80 Number of Options 10 20 30 40 50 NoContext BiDAF++ MemNet Span-BiDAF++ Figure 3: Performance of different options on HotpotQA-MC. and Span-MemNet on the span-based settings of both datasets, which are substantially different from the multiple-choice setting as shown in Table 4. BiDAF++ substantially outperforms the MemNet on both datasets, indicating that BiDAF++ is a stronger model for multi-hop reasoning, despite being less explicitly designed for this task. However, this model still underperforms the Factored BiDAF model, indicating that it could just be doing strong single-sentence reasoning. Adding more options does not qualitatively change the multiple choice setting. The span-based model requires dealing with a much larger output space than the multiple-choice setting. To test the effects of this, we conduct another experiment by making more spurious options on HotpotQA-MC using the method described in Section 4. The results are shown in Figure 3. As we increase the number of options, we can see that the performance of all models drops. However, even with more options, the no-context baseline can still achieve comparable performance to the other two more complex models, which indicates that these models still aren't learning multihop reasoning in such a strengthened setting. Span-based training data is more powerful. To further understand the two different supervision signals, we conduct another experiment where we train using span-based supervision and evaluate on the multiple-choice setting. Specifically, during evaluation, we select all document spans that map onto some answer candidate, then max over the scores of all spans to pick the predicted answer candidate. The multiple choice options therefore filter the span model's predictions. From the results in Table 3, we can see that Span2MC-BiDAF++ achieves higher performance compared to MC-BiDAF++ on HotpotQA and nearly comparable performance on WikiHop even with random span selection during training. This shows that with the span-based supervision, the model can learn at least the same thing as the multiple-choice and avoid cheating through learning question-candidate correspondences. There exist several other multi-hop reasoning datasets including WorldTree (Jansen et al., 2018), OpenBookQA (Mihaylov et al., 2018), and Mul-tiRC (Khashabi et al., 2018). These datasets are more complex to analyze since the answers may not appear directly in the passage and may simply be entailed by passage content. We leave a detailed investigation of these for future work. For researchers working on the problem of multi-hop reasoning, we think the following points should be considered: (1) Prefer models using span-based supervision to avoid cheating by using the extra candidate information. (2) If using multiple-choice supervision, check the no-context baseline to see whether there are strong correlations between question and candidates. (3) When constructing a multi-hop oriented dataset, it would be best to do an adversarial test using a sentence-factored model to see whether multi-hop reasoning is really needed. Both HotpotQA and WikiHop contain good examples for evaluating multihop reasoning, but this evaluation is clouded by the presence of easily-solvable examples, which can confuse the learning process as well. This work was partially supported by NSF Grant IIS-1814522, NSF Grant SHF-1762299, a Bloomberg Data Science Grant, and an equipment grant from NVIDIA. The authors acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing HPC resources used to conduct this research. Results presented in this paper were obtained using the Chameleon testbed supported by the National Science Foundation. Thanks as well to the anonymous reviewers for their helpful comments.
abstain abstain abstain objective result objective result abstain abstain abstain abstain abstain abstain abstain abstain result abstain objective result method abstain abstain result result abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method method abstain abstain abstain result result abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain method abstain abstain abstain other other other other