Fusion-in-decoder (FID ) (Izacard and Grave, 2021) is a generative question answering (QA) model that leverages passage retrieval with a pre-trained transformer and pushed the state of the art on single-hop QA. However, the complexity of multi-hop QA hinders the effectiveness of the generative QA approach. In this work, we propose a simple generative approach (PATHFID ) that extends the task beyond just answer generation by explicitly modeling the reasoning process to resolve the answer for multihop questions. By linearizing the hierarchical reasoning path of supporting passages, their key sentences, and finally the factoid answer, we cast the problem as a single sequence prediction task. To facilitate complex reasoning with multiple clues, we further extend the unified flat representation of multiple input documents by encoding cross-passage interactions. Our extensive experiments demonstrate that PATHFID leads to strong performance gains on two multihop QA datasets: HotpotQA and IIRC. Besides the performance gains, PATHFID is more interpretable, which in turn yields answers that are more faithfully grounded to the supporting passages and facts compared to the baseline FID model. Leveraging knowledge to make complex reasoning has been a fundamental problem of artificial intelligence. Open-domain question answering (QA) (Voorhees, 1999) is an integral part of such a line of research with impactful applications (Esteva et al., 2020; Zhang et al., 2020), where the task is to answer general domain questions by gathering evidence from a large collection of documents. While super-human level performance has been achieved on single-passage reading comprehension dataset like SQuAD (Rajpurkar et al., 2016), open-domain QA still has a long way to go, especially for questions requiring more complex reasoning. The main challenge in the task of complex QA, namely multihop QA , is that it requires a QA system to combine multiple pieces of evidence from multiple documents (Welbl et al., 2018; Talmor and Berant, 2018; Yang et al., 2018). Even for single-hop QA, it has been shown challenging for extractive QA models to effectively aggregate evidence from the combined pool of multiple passages, which has been the focus of recent work (Clark and Gardner, 2018; Min et al., 2019; Guu et al., 2020). Recent work (Lewis et al., 2020b; Min et al., 2020) has demonstrated the promise of a generative approach at combining evidences from multiple passages for answer generation. Thanks to large pre-trained transformers like T5 (Raffel et al., 2020), Izacard and Grave (2021) introduced fusion-in-decoder (FID ) that leverages passage retrieval with generative models for open-domain QA, achieving state-of-the-art scores across several single-hop QA benchmarks. However, we observe that the success of the FID model does not extend to multi-hop QA, which is corroborated by the findings in (Xiong et al., 2021). Further, the FID model is a rather opaque model in terms of interpretation of the answer generation process. This capability becomes especially important for multi-hop QA, which requires sequential reasoning across multiple evidences from the pool of retrieved passages. In this work, we propose PATHFID , a generative QA model that learns to generate an answer along with a reasoning path to improve its capability of multi-hop reasoning. PATHFID extends multi-hop QA beyond just answer generation by explicitly modeling the full reasoning path to resolve the answer with a generative sequence-to-sequence model. To this end, we cast the problem as a single sequence prediction task that simultaneously models reasoning path consisting of supporting passages and facts, and eventually the factoid answer. Furthermore, we extend PATHFID to allow for cross-passage interactions between the 974 Figure 1: An example of multi-hop question from HotpotQA dataset. retrieved passages to obtain more expressive representations from the encoder to facilitate modeling a complex reasoning chain by the decoder. Figure 1 shows an example of our task formulation, and Figure 2 shows an overview of our approach. We evaluate our proposed approach on two multihop QA datasets: HotpotQA (Yang et al., 2018) and IIRC (Ferguson et al., 2020). Our extensive experiments demonstrate that (i) PATHFID leads to significant performance gains over FID on answer generation, (ii) PATHFID is the first generative model unlocking the possibility of generating the reasoning path jointly with the answer while achieving competitive performance on supporting fact extraction metric as well. Besides the performance gains, PATHFID is able to expose the underlying reasoning process behind the answer generation, which allows us to conduct a much finer-grained qualitative and quantitative analysis on the model's behavior, providing insights into further improving and better understanding generative models for multi-hop QA. We first describe the multi-hop QA task in a general way. We assume that a collection of K passages are given for a question q : D q = { p 1 , p 2 , . . . , p K } , where D q can be a pre-defined set, or it can also be an output from a text retrieval system (e.g., DPR (Karpukhin et al., 2020) and MDR (Xiong et al., 2021)) in an open-domain QA setting. That is, in the case of the open-domain setting, D q is a subset of a large collection of passages, such as Wikipedia. The task is to generate an answer string a given q and D q . In addition, we aim at identifying which passages provide evidence, and which sentences in them are describing the evidence. Figure 1 shows a comprehensive example of the task definition, where we can see that some sentences (called supporting facts ) in the two paragraphs are crucial to answer the question. Moreover, there is a reasoning flow: the question  the first paragraph  the second paragraph, which is called a reasoning path in previous work (Asai et al., 2020). The overall task is then to predict the reasoning path along with the supporting facts, and the answer. Fusion-in-Decoder (FID ) is a generative reader based on a sequence-to-sequence architecture, initialized from pre-trained models such as T5 (Raf-fel et al., 2020) or BART (Lewis et al., 2020a). It consists of an encoder ( Enc ) and a decoder ( Dec ). First, it constructs a single block of text b n := question: q title: t n context: p n of concatenated evidence from each passage-title pair ( p n , t n ) together with the question ( q ). Then, each of the resulting evidence block b n is independently encoded into | b n |  d -dimensional output representations, which are then concatenated to form a unified input representation X = [ Enc ( b 1 ); Enc ( b 2 ); . . . , Enc ( b N )] (1) of dimension ( (cid:80) n | b n | )  d where | b n | denotes the length of the n -th block b n in number of tokens. Note that, the motivation behind this strategy is to avoid the expensive quadratic self-attention computation on the encoder-side, effectively reducing the complexity from O (( (cid:80) | b n | ) 2 ) to O ( (cid:80) | b n | 2 ) . Then, the overall answer generation is modeled 975 as a conditional generation p  ( a | X ) given X consuming the unified input representation X , where  represents the set of all model parameters. The model is trained to minimize the cross-entropy loss for generating answer tokens on the decoder side. At inference time, FID first computes X based on the retrieved passages, and then decodes the answer token by token following p  ( a i | a <i , X ) with the learned model parameters  . In this section, we introduce a generative reader (PATHFID ) for K -hop QA that jointly generates an alternating sequence of passage-level and fact-level clues on the reasoning path by more explicit fusion of evidence from the pool of input passages to arrive at the correct answer. As illustrated in Figure 2, PATHFID employs a single sequence-to-sequence architecture that independently encodes the input passages after inserting special fact markers ( < f i > ) before the i -th sentence of each passage. Conditioning on the concatenation of token-level input representations per passage, its decoder then generates the linearized hierarchical reasoning path obtained by concatenating the sequence of passage titles and their corresponding supporting fact pointers followed by the answer. Each segment on the reasoning path is separated by special markers in a way that makes it possible to uniquely recover the individual segment predictions after decoding in the inference time. The opaqueness of the FID model, which makes understanding of the reasoning process more difficult, motivated our approach and its emphasis on exposing the reasoning path. Instead of only modeling answer generation, we propose to jointly model it with the full reasoning path in an hierarchical fashion to derive the answer in a unified way using multi-task maximum likelihood training . We utilize the core input encoding architecture from FID approach (Section 2.2) by introducing a new passage representation that will facilitate supporting fact generation on the reasoning path as illustrated in Figure 2. To this end, we independently encode each input passage-title pair ( p n , t n ) along with the question q as a separate block b path n := question: q title: t n context: p path n where we redefine the context representation by inserting special tokens ( < f i > ) before each sentence of the passage as p path n := < f 1 > s (1) n < f 2 > s (2) n    < f l n > s ( l n ) n (2) where s ( i ) n denotes the i -th sentence of passage p n , and l n is the number sentences it contains. Having redefined the input blocks ( b path n ) per passage, we then compute the global input representation similar to Eq. 1 by X path q = [ Enc ( b path 1 ); Enc ( b path 2 ); . . . ; Enc ( b path N )] (3) Note that sentence indicators ( < f i > ) are shared across all passages, encouraging a more hierarchical passage representation by explicitly breaking them down into sentence-level sub-blocks using the same indicator tokens. The hierarchical design of reasoning path is inspired by the human reasoning process for multihop QA task. More precisely, if a question q requires K -hop reasoning, then we process these K passages in a sequential order alternating between their passage-level and sentence-level evidence until we reach the answer. To this end, let R q = { p r 1 , p r 2 , . . . , p r K } with r i  [1 , N ] denote the sequence of passages from the larger pool D q reflecting this reasoning process for locating the answer a for question q . As shown in Figure 2, we define the hierarchical reasoning path as a linearized sequence of blocks of passage titles and supporting facts followed by the answer block Y path q := [ T r 1 ; E r 1 ; T r 2 ; E r 2 ;    ; TK ; E r K ; A ] (4) where T r i represents the i -th title block obtained by inserting a special token ( <title-i> ) before the title t r j and A denotes the answer block derived by prepending a special token ( <answer> ) to the answer a as illustrated in Figure 2. On the other hand, i -th supporting fact block is defined as the sequence of fact indicators following <facts-i> token by E r i := <facts-i> < f j 1 > < f j 2 >    < f j mi > (5) 976 Figure 2: PATHFID model overview. where { j 1 , j 2 , . . . , j m i } denote the indices of key sentences to leverage from passage p r i to transition to the next evidence on the reasoning process R q for question q , and 1  m i  l r i denotes the number of supporting facts. Note that fact indicators < f i > are shared between the contexts p path n of input blocks (Eq. 2) and supporting fact blocks (Eq. 5) on the target reasoning path to allow the decoder to follow along the sequential reasoning R q by pointing to the facts E r i of passage p r i . PATHFID enables more explicit evidence fusion through the reasoning path to guide the model to towards correct answer in a structured way. However, it still relies on the decoder to combine all the clues together, which might still struggle due to lack of cross-passage interactions as input blocks are encoded independently. To address this potential limitation, we propose PATHFID +, where we further extend PATHFID in a way that enables cross-passage interaction by redefining the input block consisting of a pair of passages ( p n 1 , p n 2 ) as b path+ n 1 ,n 2 := question: q <title-1> t n 1 <context-1> p path n 1 <title-2> t n 2 <context-2> p path n 2 assuming that a set of passage pairs ( p n 1 , p n 2 ) are available for model to consume. derive a set of pairs of passages from the initial set D q by D + q = { ( p  , p 1 ) , ( p  , p 2 ) , . . . , ( p  , p N ) } where p  corresponds to the first passage that is possible to immediately hop to from question q , which may be determined by another model, or by executing the original PATHFID on D q in our case. Global input representation X path+ q is obtained similarly (Eq. 3) by except encoding the new blocks b path+ n 1 ,n 2 allowing for cross-passage interactions, while the target reasoning path Y path+ q remains the same as Y path q . Note that <title-i> special markers are shared between new input block b path+ n 1 ,n 2 and target reasoning path Y path+ q to provide the model with additional clue regarding the first passage on the reasoning path while still relaying the complete evidence fusion to the decoder via information redundancy encoded in X path+ q . Having defined global input representation X path q , the decoder autoregressively generates the reasoning path Y path q per token at each step by following self-attention, cross-attention on the entire X path q , and feed-forward modules. So, the overall reasoning path generation is modeled as conditional generation p  path ( Y path q | X path q ) . The model then is trained to minimize J (  path ) =  (cid:80) | Y path q | i =1 log p  ( y i | y <i , X path q ) with teacher forcing over a training set of { ( q, a, D q ) } . In the inference, the decoder consumes the input representation X path q computed by encoder, and generates the full reasoning path token by token. We then post-process the decoded sequence using the answer indicator ( <answer> ) to first obtain the answer, followed by recursively parsing the remaining sequence using the special separator tokens ( <title-k> , <facts-k> ) to reconstruct the title and retrieve its relevant sentences at each hop k . As illustrated in Figure 2, the final result of the inference can be summarized into a dictionary which maps each generated passage title to the list of sentence pointers as well as the final answer. answering datasets: HotpotQA and IIRC . HotpotQA (Yang et al., 2018) is a large-scale human-annotated dataset including 113K multihop questions. It focuses on using documents from Wikipedia as the source of information for answering questions rather than knowledge bases as in other multi-hop QA datasets (Welbl et al., 2018; Talmor and Berant, 2018). The questions in HotpotQA are not constrained by the fixed knowledge-base schema, hence they can cover more diverse topics. The answer for each question in HotpotQA is extracted from 10 paragraphs in the distractor setting, while it is allowed to use the entire Wikipedia for the full wiki setting. There are two main question types bridge (80%) and comparison (20%) in the corpus, where each question is designed in a way that extracting the correct answer requires reasoning over multiple evidence distributed across two passages. While comparison questions do not require the these passages to be processed in a particular order, bridge questions often require identifying the bridge entity in the first passage to correctly hop to the second one that contains the answer. Each question is also provided with the annotation of 2 supporting passages and up to 5 corresponding relevant sentences as their supporting facts. Since our proposed approach is a reader model that reasons over a given set of evidence documents, we primarily focus our experiments on the distractor setting 1 . IIRC (Ferguson et al., 2020) is a dataset of more than 13K human-written questions over paragraphs 1 See Appendix B for PATHFID results in open-domain setting using MDR (Xiong et al., 2021) as the retriever. from English Wikipedia, where crowdworkers had access only to initial paragraph and list of hyper-links to other relevant Wikipedia articles, with the missing information occurring in one or more linked documents. This annotation design encouraged less lexical overlap between the questions and the contexts that actually contain the answer. This dataset presents unique challenges compared to HotpotQA because (1) it additionally requires discrete/numerical reasoning and identification of unanswerable questions, which adds up to 4 different possible answer types (span, binary, numerical, unanswerable), and (2) about 30% of questions require reasoning over more than 2 passages including the main passage. Evaluation Metrics. We use standard metrics exact-match (EM) and F 1 scores for measuring the quality of predicted answers. For HotpotQA experiments, we are also able to evaluate PATHFID on supporting fact predictions using the official metrics (Support-EM, Support-F 1 ), which measures the performance of the reader model in correctly identifying the supporting facts from the relevant passages. Note that this metric implicitly requires correctly identifying relevant passages among the distractors as well. For our experiments on IIRC dataset, similar to the baseline model constructed in the original work (Ferguson et al., 2020), we follow the evaluation methods used by DROP (Dua et al., 2019). Implementation Details. We use pre-trained T5-large encoder-decoder (Raffel et al., 2020) to initialize the models in our experiments. We train the model with batch size of 64 with constant learning rate of 1e-4 for 10 epochs. We use maximum length of 256 (resp. 512) tokens for input blocks of PATHFID (resp. PATHFID +), while the maximum target sequence length is set to be 64. However, the sequence truncation is performed on the reasoning path excluding answer part for sequences of length longer than 64 tokens. All the experiments are conducted on a machine with 4 or 8 many 40GB A100 GPUs. Our code is based on Huggingface Transformers (Wolf et al., 2019). Please see Appendix for further details on the hyperparameter settings. We present our main results on the HotpotQA distractor setting in Table 1. We report results on the HotpotQA development set in comparison with the 978 Answer Support Methods EM F1 EM F1 Baseline (Yang et al., 2018) 44.4 58.3 22.0 66.7 DFGN (Qiu et al., 2019) 55.4 69.2 -QFE (Nishida et al., 2019) 53.7 68.7 58.8 84.7 SAE (Tu et al., 2020) 61.3 74.8 58.1 85.3 SAE-large (Tu et al., 2020) 67.7 80.8 63.3 87.4 Graph Recurrent Retriever (Asai et al., 2020) (base) 52.7 65.8 57.4 84.6 Graph Recurrent Retriever (Asai et al., 2020) (wwm) 68.0 81.2 58.6 85.2 Gated Memory Flow (Shao et al., 2021) 69.6 83.0 64.7 89.0 This Work FID * (Izacard and Grave, 2021) 64.4 77.8 -PATHFID 65.8 78.9 59.3 85.7 PATHFID + 72.7 84.2 64.9 88.7 Table 1: Results on the development set of HotpotQA distractor setting in comparison with previous work. previous published methods. PATHFID reader provides 1.4% absolute gain on answer EM score in comparison to FID model. Moreover, it achieves competitive supporting fact predictions of 59.3% support-EM and 85.7% support-F 1 as a result of path generation compared to strong extractive models such as (Asai et al., 2020). In summary, PATHFID establishes the usefulness of modeling the full reasoning path along with answer generation for multi-hop QA. More notably, PATHFID + achieves a quite significant performance gain across all the central evaluation metrics, demonstrating the importance of cross-passage interactions. Overall results validate the effectiveness of the two central modeling contributions of our proposed method. Next, we present further analysis and discussion on the unique advantages of PATHFID approach under a few central questions which motivated our research at the first place. How faithfully grounded are the generated answers on supporting facts? In Table 2, we present a detailed analysis comparing different models in terms of the faithfulness of their generated answers on both gold and predicted supporting facts. The first row focuses on the passage-level answer grounding computed by the percentage of the answers found in one of the gold supporting passages, while the second row reports the same analysis on sentence-level. We can observe that PATHFID models significantly improves on how faithfully the generated answers are grounded on the supporting facts both at passage-level and sentence-level granularities. The next two rows provide further insight into the quality of the generated supporting facts by PATHFID models by measuring how often the gold answer can be found in them. This analysis shows that the generated supporting facts are of quite high-quality including the gold answer for more than 95.3% and 96.2% at sentence-level and passage-level, respectively. The last two rows measure the faithfulness of the generated answers on the model generated supporting facts, which is not applicable to FID model as it does not perform supporting fact prediction. We observe that the generated answers are quite faithfully grounded on the predicted supporting facts, showing the path generation not only improves the answer EM performance but also successfully grounds them on the evidence it generates as part of the full reasoning path. It is important emphasize here that extractive reader models can be guaranteed to output perfectly grounded answers simply by locating the answer in their predicted supporting facts. On the other hand, it is difficult for generative models to ensure 100% answer grounding simply due to its generative na-979 Answer-EM Support-EM Comparison Bridge Comparison Bridge # Supp Facts FIDPATHFIDFIDPATHFIDFIDPATHFIDFIDPATHFID 2 70.4 71.8 63.3 64.6 -86.7 -70.0 3 66.1 68.2 62.7 63.1 -43.4 -30.7 4 62.2 63.8 64.3 66.5 -5.4 -26.2 >=5 83.3 87.5 60.0 65.0 -0.0 -3.8 Table 3: Performance breakdown on Answer-EM and Support-EM by question type and the number of gold supporting facts (rows). ture. However, we are able to provide additional evidence validating the answers generated by PATHFID are significantly grounded in the supporting facts it generates, which might implicitly indicate that the generated reasoning path tightly aligns with the model's underlying process for answer generation. Although this is a strong evidence, it is still quite implicit in exposing the model's prediction process, so we see our approach as a step in the right direction rather than a complete solution. Performance breakdown by the number of supporting facts and question types . In Table 3, we compare the performance of models by breaking them down based on the number of gold supporting sentences and the question type (e.g., bridge and comparison). Our first observation is that PATHFID provides consistent improvement on answer-EM score over FID across both the question types and different number of supporting facts required to answer the question. The high variance in the answer-EM score on comparison questions can be attributed to the strictness of exact-match metric as well as the imbalanced nature of the dataset where only 5% of the comparison questions have more than 3 supporting facts. Surprisingly, both FID and PATHFID models perform considerably well on the comparison questions even when it requires at least 5 supporting facts. A more important motivation behind the performance breakdown analysis was to understand how the supporting fact prediction of PATHFID would change as the number of gold supporting facts grows. Although it starts degrading on examples with more than 2 supporting facts, it still achieves more than 25% Support-EM for bridge questions with up to 4 supporting facts. Recalling the average performance on the whole dataset is less than 60%, we conclude this result might be satisfactory enough, especially for a fully generative Figure 3: PATHFID model evolution on the HotpotQA Dev set during training. Analyzing the evolution of sub-tasks during joint training with PATHFID . In Figure 3, we present the evolution of PATHFID model on the HotpotQA development set at every 500 training steps. We observe that while the model more quickly picks up the patterns for title generation, it takes much longer for it to reach to a reasonable level of fact prediction. As one would expect, the general trend in the evolution of different segments (title-1, facts-1, title-2, facts-2, answer) of the reasoning path mostly follows the difficulty of the corresponding sub-task although all the sub-tasks are jointly formulated and trained in an end-to-end fashion. On the other hand, it seems counter-intuitive for model to reach to a better accuracy on predicting the facts of the second passage (F2-EM) on the reasoning path earlier despite having a better accuracy on (T1-EM). However, one can also interpret it as a result of stronger feedback provided by the answer segment of the reasoning path as most of the ground-truth answers are contained in the facts of the second passage. In addition to our main experiments presented in greater detail, we also conduct experiments on IIRC dataset to verify the generalization of the proposed approach. To this end, we closely follow the authors' model-free retrieval setting (referred to as Oracle L+C in Table-3) because the model checkpoints for the baseline retrieval model are not available in the public release. We use a python script 2 provided in the open-sourced repository to replicate the same setting for a fair comparison. In Table 5, we present the results on the development set for our proposed PATHFID and PATHFI D+ in comparison with the baseline reported in the original paper (Ferguson et al., 2020) and our implementation of the FiD (Izacard and Grave, 2021) baseline. FID model obtains a comparable F1 with IIRC baseline with a slightly worse exact-match performance. However, the proposed PATHFID approach is able to provide 1.3% and 1.4% improvement in F1 score over the two baselines. Furthermore, PATHFID + extension leads to the best performance achieving 4.7% and 4.2% EM score improvement in absolute value over the FID 2 https://github.com/jferguson144/ IIRC-baseline/blob/main/make_drop_style.py baseline and IIRC baseline, respectively. Our experimental results validate the benefit of the proposed approach on the IIRC dataset, suggesting strong evidence for the generalizability of our approach. 4.4 Analyzing the Benefit of Joint Training In Table 4, we present the results of a case study where we analyze the benefit of multi-task training on the passage chain prediction. The first row of the table shows the results for training PATHFID only to predict the sequence of titles for the gold passages (i.e., [t1-t2] ), which is just a subsequence of the full reasoning path obtained by discarding facts and the answer. The second row is another variant, where we add the answer back to the linearized target sequence while still excluding the segments corresponding to the facts. The last row correspond to the full reasoning path generation, which is corresponding to the original formulation of PATHFID as described in Section 3 and illustrated in Figure 2. Comparing first two rows in Table 4, we can immediately observe that including answer segment in the target reasoning path (i.e., [t1-t2-answer] ) boosts the performance across the board although in principle it makes the task more complicated while utilizing the same underlying model capacity. Further including segments corresponding to FACTS (sentences within supporting passages) in addition to answer segment (i.e., [t1-f1-t2-f2-answer]  full reasoning path) boosts the title-EM even further, especially before applying title reconstruction post-processing step. Although the objective of the first task (i.e., [t1-t2] ) is perfectly aligned with the evaluation metric used in Table 4, the performance of the resulting model remains inferior compared to jointly modeling the same task with the answer (and/or supporting facts) prediction. These two observations elicit a compelling evidence regarding the benefit of jointly modeling the sub-tasks of multi-hop QA as single sequence capturing the full reasoning path. Multi-hop question answering. Research on multi-hop QA aims to tackle complex questions that require reasoning across multiple pieces of evidence in multiple documents (Welbl et al., 2018; Yang et al., 2018; Ferguson et al., 2020). In particular, the HotpotQA dataset (Yang et al., 2018) provides both the closed and open-domain settings to evaluate multi-hop reading comprehension models. Compared to single-hop QA, such complex questions pose additional challenges for both reader and retriever models since they are required to capture relationships between documents, instead of independently processing each document. This is challenging because the number of document combinations exponentially grows due to the sequential nature of the process. Two recent works (Nie et al., 2019; Asai et al., 2020) have tackled this challenge by leveraging hyperlink structure in the underlying Wikipedia corpus, while Xiong et al. (2021) has taken a recursive approach to extend the dense retrieval process to handle sequential search. Most of the reading comprehension (RC) models in existing work (Xiong et al., 2019; Chen et al., 2019; Nishida et al., 2019; Qi et al., 2021; Li et al., 2020; Xiong et al., 2021) follow an extractive architecture (De-vlin et al., 2019) for selection of the answer spans and their corresponding supporting evidence with minor modifications such as initializing the backbone model from a stronger or larger pre-trained models (Clark et al., 2020). On the other hand, some recent works (Inoue et al., 2021) take a more abstractive approach and generate question-focused summaries of input paragraphs as concise explanations to be fed to the RC module. Generative question answering. Especially after the emergence of the SQuAD dataset (Rajpurkar et al., 2016), neural extractive QA models have been widely studied. An underlying assumption is that we can extract a short text span (or a phrase) as an answer, but it is not always the case in reality. Motivated by this, the generative QA approach has also been investigated (Hewlett et al., 2017; Fan et al., 2019). Recent advances on pre-trained transformers have pushed this direction; for example, Lewis et al. (2020a) jointly trained a generative QA model along with a text retrieval model, and Roberts et al. (2020) explored an ambitious approach to directly generate an answer without any evidence documents. We focused on the fusion-in-decoder model (Izacard and Grave, 2021); they claimed that the decoder might be good at aggregating information across multiple documents. However, we have shown that it is not trivial in the multihop reasoning task, and pushed the model's ability to jointly learn to predict reasoning paths. Besides question answering, jointly learning multiple intrinsic capabilities required by the final objective with a generative approach has been shown useful in modeling other NLP tasks such as task-oriented dialogues (Neelakantan et al., 2019; Hosseini-Asl et al., 2020; Peng et al., 2021). Open-domain question answering. Open-domain QA (Voorhees, 1999) is practically important, which requires a system to retrieve relevant documents to answer a given question. The task is recently gaining much attention, thanks to the development of large-scale datasets like HotpotQA, SQuAD Open (Chen et al., 2017), Natural Questions Open (Kwiatkowski et al., 2019; Lee et al., 2019), etc. Pre-trained transformer models like BERT (Devlin et al., 2019) have accelerated the development of neural text retrievers (Lee et al., 2019; Karpukhin et al., 2020; Asai et al., 2020; Xiong et al., 2021; Liu et al., 2021) in the retriever-reader framework (Chen et al., 2017). We have investigated the effectiveness of our method in the multi-hop open-domain QA task (see Appendix B) using an existing external retriever component. In this work, we propose a generative question answering (QA) approach that models multi-hop QA as a single sequence prediction task. It learns to generate an answer along with a reasoning path to improve its capability of multi-hop reasoning. Our experiments on prominent multi-hop QA benchmarks, HotpotQA and IIRC, validate the promise and effectiveness of our proposed method PATHFID and its extension PATHFID +. Future work will explore (1) our PATHFID approach more closely with text retrieval models in open-domain QA scenarios and (2) more explicit grounding on the input information to make our approach even more interpretable and controllable. The authors would like to thank the members of Salesforce AI Research team for fruitful discussions, as well as the anonymous reviewers for their helpful feedback.
abstain abstain objective abstain objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain objective abstain result abstain objective abstain abstain result objective method other other objective method other other other other other other other other other other method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain other abstain abstain abstain abstain abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain method abstain method method method abstain abstain method abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain other abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain result abstain abstain abstain abstain abstain abstain abstain other other other other other other other other other other method other other method abstain other other other other other objective objective abstain objective objective other