Recent work on training neural retrievers for open-domain question answering (OpenQA) has employed both supervised and unsupervised approaches. However, it remains unclear how unsupervised and supervised methods can be used most effectively for neural retrievers. In this work, we systematically study retriever pre-training. We first propose an approach of unsupervised pre-training with the Inverse Cloze Task and masked salient spans, followed by supervised finetuning using question-context pairs. This approach leads to absolute gains of 2 + points over the previous best result in the top-20 retrieval accuracy on Natural Questions and TriviaQA datasets. We next explore two approaches for end-to-end training of the reader and retriever components in OpenQA models, which differ in the manner the reader ingests the retrieved documents. Our experiments demonstrate the effectiveness of these approaches as we obtain state-of-the-art results. On the Natural Questions dataset, we obtain a top-20 retrieval accuracy of 84%, an improvement of 5 points over the recent DPR model. We also showcase good results on answer extraction, outperforming recent models such as REALM and RAG by 3 + points. Our code is available at: https: //github.com/NVIDIA/Megatron-LM . The task of open-domain question answering (OpenQA) consists of finding answers to the information-seeking questions using a large knowledge source such as Wikipedia. This knowledge source is also referred to as evidence and it typically contains millions of documents. Most approaches for OpenQA consist of a two-stage pipeline (Chen et al., 2017; Chen, 2018). In the first stage, given  This work was done during an internship at NVIDIA. a question, a retriever module identifies the most relevant documents, which is often a very small subset of the evidence known as context documents . Traditionally, approaches based on document ranking such as BM25 (Robertson and Zaragoza, 2009) have been used for the retriever. In the second stage, these relevant documents are given as input to the reader module, which understands them and extracts the answer for the question (Figure 1). The main drawback of the BM25 method is that it is not trainable and hence it can't be adapted to tasks involving open-retrieval. Recent work has addressed this limitation by building upon advances in self-supervised learning, such as BERT (Devlin et al., 2019). These approaches model both the retriever and reader using neural networks, allowing the retriever to be trained using task-specific datasets (Lee et al., 2019; Guu et al., 2020). Typically, the retriever model consists of a dual-encoder architecture (Bromley et al., 1994), where one encoder processes the question and the other encoder processes the context document. Prior work has investigated both unsupervised and supervised approaches to train the retriever. Unsupervised approaches include separately training the retriever with Inverse Cloze Task (ICT) (Lee et al., 2019) or training the retriever and reader jointly by pre-6649 dicting masked salient spans (REALM) (Guu et al., 2020), while supervised approaches such as Dense Passage Retrieval (DPR) (Karpukhin et al., 2020) train the retriever using human-annotated sets of question and context pairs. However, there is no study that investigates the comparative advantages of using these two styles of training when the retrieval task is challenging, i.e. , when the evidence contains millions of documents. It is unclear if the unsupervised approaches can further help to improve the performance of strong supervised approaches, and, if so, under what conditions. A core focus of this work is systematically studying these aspects of retriever training. We propose a unified approach to train the retriever: unsupervised pre-training followed by supervised finetuning. We also investigate key design choicessuch as relevance score scaling and longer trainingand showcase their effectiveness. Our results demonstrate that the proposed approach obtains substantial accuracy gains when evaluated on benchmark OpenQA datasets. Extensive experiments also highlight the relative importance of different pre-training strategies, revealing important trade-offs when varying the amount of supervised data available to train the retriever. Furthermore, motivated by recent work (Guu et al., 2020; Lewis et al., 2020a), we also explore two approaches for end-to-end supervised training of the reader and retriever components. In the first approach, the reader considers each retrieved document separately while in the second approach, the reader takes as input all the retrieved documents together. We compare the effectiveness of these approaches on both retrieval accuracy and answer extraction. We show that the first approach leads to an improved retrieval performance, while the second approach results in an improved answer extraction. With end-to-end training, we outperform previous best models to obtain new state-of-the-art results on retrieval accuracy and answer extraction. We also perform experiments by scaling the model size to a large configuration for both retriever and reader and observe consistent improvements, compared with smaller models. In summary, the contributions of this work are:  We demonstrate that our proposed method of unsupervised pre-training of the retriever with ICT followed by supervised finetuning leads to absolute gains of more than 2 points in the top-20 retrieval accuracy over the previous best result on Natural Questions and TriviaQA datasets. We show that masked salient spans -based pretraining of the retriever is more effective when the supervised dataset sizes are small. Our end-to-end training approach obtains new state-of-the-art performance on retrieval accuracy. On Natural Questions, our top-20 accuracy is 84 , which is a 5 points gain over DPR results. We achieve competitive results on answer extraction with gains of more than 3 points over recent models such as REALM (Guu et al., 2020) and RAG (Lewis et al., 2020c). We scale up end-to-end training to large models and show consistent gains in performance. The rest of the paper is organized as follows. Sec. 2 and 3 explain the retriever model and end-to-end training, respectively. Sec. 4-6 describe the experimental details with the results. Sec. 7 reviews the related work followed by conclusion in Sec. 8. In this section, we first describe the retriever architecture and then discuss different approaches to train it, including our proposed approach. Given a collection of documents in the evidence Z = { z 1 ,    , z m } and a question q , the task of the retriever is to select a relevant subset of documents for the question. To do this, the retriever performs a ranking of the evidence documents conditioned on the question and outputs the top-ranked documents. The retriever model consists of two modules: a question encoder ( f Q ) and a context encoder ( f Z ). Such a model is often referred to as a dual-encoder model (Bromley et al., 1994). Here, we detail the training methodology of the dual-encoder model given a questions ( q ) and context documents ( z i ) from Z . First, we compute the relevance score between the question and context. We define the relevance score to be the dot-product between the question and context representations s ( q, z i ;  ) = f Q ( q ) (cid:62) f Z ( z i ) (1) where f Q ( q )  R d and f Z ( z )  R d denote the question and context encoders, respectively, which are parameterized by  = [  Q ,  Z ] . We model the f Q and f Z using BERT-style transformer networks (Devlin et al., 2019; Vaswani et al., 2017). We consider the hidden states of the first token of 6650 the sequence (i.e. [CLS] token) as the encoder's output. The probability of a context document z i being relevant to the question q is calculated as p ( z i | q, Z ;  ) = exp( s ( q, z i ;  ) / ) (cid:80) |Z| j =1 exp( s ( q, z j ;  ) / ) (2) where  is a scaling factor. While previous work had used the setting of  = 1 , in this work, we set  =  d . Bigger scaling factor helps in better optimization when the model hidden size ( d ) is large. We refer to this as relevance score scaling . To train the retriever, we maximize the log-likelihood computed from Eq. 2. In practice, as the evidence set consists of millions of documents, the normalization term would be expensive to compute. Hence, we approximate the denominator of the above equation by using the context documents in the batch as negative examples, a technique that has shown to perform well in practice (Chen et al., 2020). In this section, we discuss different approaches to train the retriever. In all the approaches, we initialize the parameters of both the question and context encoders using BERT weights as implemented in Megatron-LM (Shoeybi et al., 2019). We also experimented with random initialization but it vastly underperformed BERT initialization. In the supervised setting, human-annotated questions, answers, and sometimes context are provided. If the context is not included, then a common approach is to use distant supervision (Mintz et al., 2009) to obtain the context document. Specifi-cally, we select the top-ranked document using BM25 (Robertson and Zaragoza, 2009) from the evidence that contains the answer as the context. We also select other top-ranked documents that do not contain the answer as additional hard negative examples. This approach to train neural retriever was popularized by (Karpukhin et al., 2020). Inverse Cloze Task (ICT): In this setup, we do not consider the human-annotated question-context pairs. Instead, the retriever is trained in an unsupervised manner. Specifically, a randomly sampled sentence from a paragraph is considered as the query while other sentences as the context. This approach was first proposed by (Lee et al., 2019). Masked salient spans training: (Guu et al., 2020) showcased that the ICT initialized retriever can be further improved by training it with an objective where the reader predicts the masked salient spans such as named entities conditioned on the retrieved documents. In this work, we adopt the same approach. However, unlike (Guu et al., 2020) who use BERT for the reader, we use a generative language model based on T5 (Raffel et al., 2020). To improve the retriever training, we propose the approach of unsupervised pre-training of the retriever followed by supervised finetuning. In this approach, we first pre-train the retriever weights with ICT training or masked salient spans training (Sec. 2.2.2). After pre-training, we finetune the retriever with supervised training (Sec. 2.2.1). In this section, we explore two supervised training approaches to end-to-end train the reader and retriever components from the task-specific data. In the first approach, the reader considers each retrieved document separately (Sec. 3.1) while in the second approach, the reader takes as input all retrieved documents together (Sec. 3.2). These approaches are designed such that when predicting the answer conditioned on the question, the learning process improves both the reader and retriever. Background and notation: In end-to-end training, the trainable components consists of the retriever (  ) and reader (  ) parameters. For retriever, we use the dual-encoder architecture and train it as discussed previously in Sec. 2.3. Our reader is a generative model designed according to the sequence-to-sequence modeling paradigm (Sutskever et al., 2014). Specifically, we use pre-trained T5 as the reader. The inputs to the training process are questions ( q ) and its answers ( a ), both in string form. Given a question, first the retriever obtains the k relevant context documents ( K ) from the evidence ( Z ) as K = arg sort z i Z s ( q, z i ;  )[: k ] (3) The reader then takes the question and one or more context documents ( z i ) as input to predict the an-6651 Question (q) Evidence embeddings Top-k inner product search Encoder Decoder T5 Model Top-k documents (z) Reader X Dot-product Relevance score Stale weights from previous checkpoint Evidence Documents Asynchronousembeddingupdate Loss Answer (a) Retriever Figure 2: A schematic diagram illustrating end-to-end supervised training of the retriever and reader components. where N is the number of answer tokens. Next, we describe the two proposed approaches. A block diagram illustrating the end-to-end training process is shown in Figure 2. 3.1 Approach 1: Individual Top-k In this approach, similar to (Guu et al., 2020), the reader's likelihood is first computed conditioned on the question and each retrieved document. The marginal likelihood is defined as the weighted average of the individual likelihoods as p ( a | q ; ,  ) = (cid:88) z i K p ( a | q, z i ;  ) p ( z i | q, Z ;  ) , (5) where p ( z i | q, Z ;  ) is computed using Eq. 2. However, the normalization is done over K instead of Z . The final loss is defined as the negative marginal log-likelihood L ( q, a ) =  log p ( a | q ; ,  ) . We note that the RAG model (Lewis et al., 2020c) also proposed a similar approach, but there are two main differences. The first is that while we update all the parameters of the retriever (both the query and context encoders), RAG just updates the query encoder. The second is that we use T5 model as the reader while RAG uses BART model (Lewis et al., 2020b). These enhancements help us obtain substantial gains over the RAG model, which we will discuss in Sec. 6. In this approach, similar to (Lewis et al., 2020a), the likelihood is defined as the reader's likelihood conditioned on the question, all the retrieved documents, and the retrieval score As the T5 reader consists of separate encoder and decoder modules, it provides the flexibility to customize the input or output of the encoder. We concatenate each retrieved document with the question and feed them as input to the encoder, which computes their hidden representations. Next, we stack the hidden representations of all the retrieved documents, which the decoder jointly attends to during the encoder-decoder attention, thus allowing a more powerful form of information aggregation from multiple retrieved documents. We also add retriever similarity score to bias the encoder-decoder attention as it helps facilitate end-to-end training and enables the reader to pay higher attention to the relevant documents. The interaction score during the encoder-decoder attention is computed as attn ( q, a, z 1: k )  Q ( a ) (cid:62) K ( z 1: k , q )+ p ( z | q ;  ) , (8) where Q is the query vector computed from de-coder's input, K is the key vector computed from encoder's output, and  is a trainable parameter. Final loss is defined according to Eq. 6. We further note that a similar approach for OpenQA was proposed in (Izacard and Grave, 2020) but it only optimizes the reader model and didn't perform end-to-end training of the retriever. In this section, we describe the datasets and model settings. For reproducibility, we provide training details and list the hyperparameters in Appendix A. 4.1 OpenQA Datasets We perform experiments using two widely used QA datasets whose details are provided below and their statistics are shown in Table 1. Natural Questions (NQ): This corpus consists of real questions asked from the Google search engine along with their long and short answer annotations from the top-ranked Wikipedia pages (Kwiatkowski et al., 2019). Following prior work (Karpukhin et al., 2020), we use the same subset of the short answer questions in our experiments, as it is more suited for OpenQA. TriviaQA: This corpus consists of a collection of trivia questions and their answers scraped from multiple sources in the Web (Joshi et al., 2017). Evidence: Following (Karpukhin et al., 2020), we make use of their released preprocessed English Wikipedia dump from December 2018 as the source of evidence documents. Overall, there are 21 , 015 , 324 documents, each 100 words long. We use two models of different sizes, base and large , for the experiments. The base configuration consists of 12 layers, 768 -d hidden size, and 12 attention heads. The BERT-base contains 110 M parameters while the T5-base contains 220 M parameters. The large configuration consists of 24 layers, 1024 -d hidden size, and 16 attention heads. The BERT-large contains 330 M parameters while the T5-large contains 770 M parameters. In this section, we compare different approaches to train the retriever. Retrieval accuracy is evaluated using the top-k metric ( k  { 1 , 5 , 20 , 100 } ). We explore the best training settings for supervised training of the retriever. To do so, we perform a series of experiments on the NQ dataset starting with the training settings from the popular DPR model and then progressively improve it. DPR was initialized with BERT, trained for 40 epochs, with a scaling factor of 1 , and utilized [CLS] token embeddings from the retriever. Our result with this setting is shown in Table 2. We then observe that incorporating relevance score scaling and longer training till 80 epochs helps to improve the top-5 and top-20 accuracy by 1 . 5 2 points. These results also signify that the original DPR model was significantly undertrained and not fully optimized. In addition to score scaling, we further include 1 additional hard-negative example (similar to DPR) for each question-context pair and train the model for 80 epochs. Our results, in sync with the results of DPR, obtain substantial additional gains in performance. These findings highlight that relevance score scaling, longer training, and including a hard negative example are essential to improve the supervised retriever's accuracy . These supervised training results can be considered as a very strong baseline. Hence, we employ these settings in subsequent experiments. We first characterize the zero-shot retriever's performance when its weights are initialized with either BERT or ICT or masked salient spans pre-training (Table 3). As is understood that unsupervised language models do not perform well in information retrieval tasks (Lee et al., 2019), evidently, BERT also leads to a poor retrieval accuracy. We note that ICT initialization is quite effective in providing a non-trivial zero-shot accuracy which is further improved by masked salient spans training by more than 8 points. Both being unsupervised approaches 6653 Model NQ TriviaQA Top-1 Top-5 Top-20 Top-100 Top-1 Top-5 Top-20 Top-100 Base Configuration BERT (zero-shot) 0.9 3.9 9.4 20.3 0.6 2.8 7.2 17.8 ICT (zero-shot) 12.6 32.3 50.6 66.8 19.2 40.2 57.5 73.6 Masked salient spans (zero-shot) 20.0 41.7 59.8 74.9 31.7 53.3 68.2 79.4 BERT + Supervised 48.6 68.8 79.0 85.8 57.5 72.2 80.0 85.1 ICT + Supervised 48.4 72.1 81.8 88.0 58.4 73.9 81.7 86.3 Masked salient spans + Supervised 50.3 71.9 82.1 87.8 60.6 74.8 81.8 86.6 Large Configuration ICT (zero-shot) 13.0 31.8 49.3 66.1 20.1 41.6 58.5 74.1 BERT + Supervised 51.4 71.0 81.0 87.2 60.4 74.5 81.4 86.0 ICT + Supervised 52.4 72.7 82.6 88.3 61.9 76.2 82.9 87.1 Table 3: Effect of unsupervised pre-training on retrieval accuracy when evaluated on NQ and TriviaQA test sets. demonstrate their utility in effectively bootstrapping the retriever almost from scratch. We next empirically analyze our proposed approach of pre-training with ICT and masked salient spans followed by supervised finetuning. We observe that it provides absolute improvements of 2 3 points over the already strong supervised training results, with the gains being consistent across both the datasets. These results highlight that even after finetuning the retriever with thousands of labeled examples, it does not lead to catastrophic forgetting of the discriminative properties learned by the retriever during ICT and masked salient spans pre-training. Another merit is that being unsupervised, large text collections can be leveraged to pre-train the retriever, a considerable advantage over data-augmentation methods which rely on the availability of human-annotated question-context pairs. Furthermore, when comparing ICT with masked salient spans initialization, we note that their accuracy gains are roughly similar . We study the effect on accuracy when the retriever is pre-trained with BERT, ICT, or masked salient spans and the amount of supervised training data is varied. We train the retriever with 1% , 2% , 5% , 10 50% , of NQ's training data and plot the top-20 accuracy in Figure 3. Results reveal that in the low-resource regime, masked salient spans pre-training is much more effective than ICT, consistently leading to large gains . As the fraction of training data increases to beyond 40% towards a high-resource setup, the gains from salient spans pre-training saturates to that of ICT . We believe that these findings will have important implications for future research in OpenQAwith only a few hundred ex-0.0 0.01 0.02 0.05 0.1 0.2 0.4 0.5 1.0 fraction of the training data 10 20 30 40 50 60 70 80 T o p 20 a cc u r a cy BERT Init ICT Init Masked Salient Spans Init Figure 3: Effect of amount of training data on retrieval accuracy when evaluated on NQ test set. amples, performing expensive masked salient span training is beneficial while if the training data has thousands of examples, ICT is just as optimal as masked salient spans training. For end-to-end training, retriever weights are initialized with the previous best setting of ICT pretraining and supervised finetuning. The number of retrieved evidence documents for the reader is considered as a hyperparameter and is selected via performance on the dev set. The focus here is to analyze the effect on retrieval accuracy when updating the retriever weights using question-answer pairs in an end-to-end setting (Sec. 3). From the results in Table 4, we observe that for Individual Top-k , when only the query encoder is updated, it tends to improve retrieval accuracy. In addition, when the context encoder is also updated, the retrieval accuracy improves to 75% at top-5, a big gain of 8 points over the previous best DPR retriever. Larger models further help to improve the performance leading to new state-of-the-art results. On the other hand, in Joint Top-k , updating the 6654 Model NQ TriviaQA Q C Top-1 Top-5 Top-20 Top-100 Top-1 Top-5 Top-20 Top-100 Base Configuration DPR (Karpukhin et al., 2020)  67.1 78.4 85.4   79.4 85.0 ICT + Supervised 48.4 72.1 81.8 88.0 58.4 73.9 81.7 86.3 Individual Top-k (cid:51) (cid:55) 54.5 73.7 83.2 88.6 61.4 75.6 82.1 86.7 Individual Top-k (cid:51) (cid:51) 56.8 75.0 84.0 89.2 63.5 76.8 83.1 87.0 Joint Top-k (cid:51) (cid:55) 51.1 72.1 81.8 87.8 59.1 74.1 81.3 86.3 Large Configuration ICT + Supervised 52.4 72.7 82.6 88.3 61.9 76.2 82.9 87.1 Individual Top-k (cid:51) (cid:51) 57.5 76.2 84.8 89.8 66.4 78.7 84.1 87.8 Joint Top-k (cid:51) (cid:55) 53.7 73.3 83.2 88.0 61.2 75.9 82.7 87.0 Table 4: Effect of end-to-end training using question-answer pairs on retrieval accuracy. We also do not update the context encoder for Joint Top-k as it did not result in improvements during our initial experiments. These results showcase that when the retriever is already well-initialized, the objective function of Individual Top-k method is designed such that it significantly improves the retrieval accuracy while the Joint Top-k method does not result in improvements. As we will show next, that the usefulness of this method lies in answer extraction. Retrieval score scaling is used when computing the probability distribution of the retrieved documents according to Equation 2, where the retrieval score is normalized by the scaling factor (  ). To study the effect of  on the retrieval accuracy, we perform an ablation study with different values of  on the NQ retrieval task, whose results can be seen in Table 5. More specifically, we choose different values of  as a multiple of  d , where d is the hidden size of the model. Our results indicate that the choice of  =  d works well in practice. Here, we briefly explain the intuition regarding the usage of the scaling factor. In our preliminary experiments on retriever training and end-to-end training without the scaling factor, we observed that a few of the top-k document's similarity score with the query was very high that in turn led to it being assigned a high retrieval probability score. This high score was leading to a skewed probability distribution with most of the mass being centered over the top-1 or top-2 retrieved documents. A larger value of scaling factor results in a more even distribution of probability mass over the top-k documents, which in turn leads to better results in both retrieval accuracy and in the end-to-end training. We next present the results of end-to-end training on answer extraction. To train the model, retriever weights are initialized with ICT pre-training and supervised finetuning while the reader is initialized with pre-trained T5 weights. The number of retrieved evidence documents for the reader is tuned on the dev set. Results are reported using the conventional Exact Match (EM) metric. We compare our results as presented in Table 6 with the recent related approaches in OpenQA. For the base configuration on NQ, our model outperforms both REALM and DPR by more than 4 points. For the large configuration, we compare with the RAG model (Lewis et al., 2020c), where our approach outperforms it by 3 . 5 + points on NQ and by 2 . 8 points on TriviaQA. Our improved results are because of a more accurate initial retriever, stronger reader, and updating both the query and context encoders during training . Our analysis in Figure 4 reveals that updating the context encoder improves the results for both the base and large configurations. Quite surprisingly, we also observe that the performance of Individual Top-k approach is sensitive to the number of top-k documents and can also decrease with an in-crease in top-k documents. We leave an in-depth investigation of this as a future work. We compare our results with the recent Fusion-in-Decoder (FiD) approach (Izacard and Grave, 2020) that also performs joint encoder-decoder attention. It consists of DPR as the retriever and T5 as the reader, which are initialized with their open-source weights. However, unlike our approach, FiD just finetunes the reader weights. Our results in Table 7 show that for the base configuration, Joint Top-k outperforms the FiD model by 1 point on NQ, highlighting the significance of end-to-end training. For the large configuration, we obtain a gain of 0.7 points on TriviaQA. Our analysis in Figure 5 portrays that the EM scores improve with more retrieved documents. This highlights that in contrast to Individual Top-k , the Joint Top-k better aggregates the information Model NQ TriviaQA Base Configuration FiD (Izacard and Grave, 2020) 48.2 65.0 Joint Top-k 49.2 64.8 Large Configuration FiD (Izacard and Grave, 2020) 51.4 67.6 Joint Top-k 51.4 68.3 Table 7: Results on answer extraction using Joint Top-k approach. contained in the retrieved documents. This Figure also illustrates the effect of similarity enriched attention on answer extraction for the base configuration. For values of top-k=5, 10, and 25, using retrieval-similarity enriched encoder-decoder attention, we consistently observe a gain of 0.8-1 EM points (comparing orange plot and blue plot in Figure 5), while there is a smaller gain when top-k=50. This signifies that with more retrieved documents, the utility of end-to-end training tends to diminish, thus explaining the lower gains observed in retrieval performance for Joint Top-k in Table 4. Based on the discussions in Sec. 5.4 and Sec. 6, we remark that end-to-end training using the two approaches has a complementary effect on the retrieval accuracy and answer extraction. While the Individual Top-k approach helps to significantly improve the retrieval performance, the Joint Top-k approach is more useful for answer extraction. (Yih et al., 2011) proposed a discriminative approach to train a retriever by learning dense representations of query and context documents based on word frequency. However, this approach was data-hungry and not scalable. Recently, (Lee et al., 6656 2019; Karpukhin et al., 2020) address this by leveraging pre-trained BERT weights (Devlin et al., 2019) to train a dual-encoder retriever by using smaller amounts of question-context pairs. In particular, (Lee et al., 2019) first pre-train the retriever in an unsupervised manner using ICT and then jointly train the retriever and reader for OpenQA. On the other hand, (Karpukhin et al., 2020) perform supervised training of the retriever using hard-negative examples, yielding impressive results on several retrieval benchmarks. To improve the retrieval accuracy of the dual-encoder model, (Chang et al., 2020) explore several paragraph-level pre-training strategies including the application of ICT. They demonstrated the effectiveness of pre-training over sparse-retrieval approaches such as BM25. Their evidence consisted of the training documents that was further increased to 1 M documents for OpenQA. Our work differs from them in several ways. First, our OpenQA setup is more challenging as the evidence consists of 21 M documents. Second, we pre-train with two strategies consisting of ICT and masked salient-spans and finetune using strong supervised methods, which leads to much improved results. Third, we further update the retriever with end-to-end training leveraging question-answer pairs, which further improves the retrieval accuracy leading to new state-of-the-art results. A new line of work investigates task-specific pretraining of language models. For example, (Guu et al., 2020) predicts masked salient spans consisting of named entities to pre-train the reader and retriever components for OpenQA. Similarly, (Lewis et al., 2020a) perform cross-lingual pre-training where the objective is to predict a sequence using its paraphrases in different languages, demonstrating improved zero-shot performance in document translation tasks. We propose approaches to improve the retrieval accuracy of the dual-encoder model for the OpenQA task. We first perform a systematic investigation of the importance of pre-training with ICT and masked salient spans tasks for supervised training of the retriever. We then present two approaches for end-to-end training of the reader and retriever components in OpenQA. In one approach, the reader considers each retrieved document individually while in the other approach where the reader considers all the retrieved documents jointly. Overall, these methods help achieve state-of-the-art results on both retrieval and answer extraction. This work was done during the first author's internship at NVIDIA. It was also partially supported by Canada CIFAR AI Chair held by Prof. Hamilton. We would like to thank the anonymous reviewers for providing valuable feedback and recommendations. We would also like to thank the administrators of the Selene supercomputer for their assistance in facilitating the large-scale runs. To understand the ethical context of our work on open-domain question answering, it is important to consider the real-world use cases and potential individuals who may interact with systems developed based on our proposed methods. The potential real-world applications could be search engines or virtual assistants, where our techniques can improve the question-answering ability. However, it is worthwhile to mention that our trained systems can not be deployed off-the-shelf for such applications, given that our models were trained on the Natural Questions and TriviaQA datasets with the goal of matching the specific training data distribution. Real-world applications building on our work should be re-trained using a custom training dataset that is relevant to the kind of queries that originates in practice. Our system represents a prototype model for answering questions over Wikipedia and can easily be extended to be used in sensitive contexts such as legal or health-care settings. However, extensive and robust quality assurance testing will be needed as our system was not designed to meet those criteria. More generally, there is the possibility of social biases which could be introduced by the training data. Since we did not control or regularize our model to remove such biases, we would urge the users to undertake the necessary quality-assurance testing to evaluate and understand the extent to which such biases might be present. User should also understand how much these biases are impacting their trained system and to make modifications to their training data and procedures accordingly.
abstain abstain method objective abstain objective objective result result other abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method objective objective objective abstain abstain abstain method objective objective result objective result objective result result result abstain abstain abstain abstain abstain other other other other abstain objective method method objective other method other abstain method other abstain method method method abstain abstain method abstain abstain method abstain abstain abstain abstain method method method method method abstain abstain abstain abstain method method method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method method other abstain abstain method method method abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain abstain abstain result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain result method result abstain result abstain abstain result abstain abstain abstain result result result abstain abstain result result result method result abstain method result result result abstain abstain abstain result abstain abstain abstain other other other other other other other other objective objective abstain objective other other other objective objective method abstain abstain other other other other abstain abstain abstain method abstain method abstain method abstain