Open-domain question answering can be reformulated as a phrase retrieval problem, without the need for processing documents on-demand during inference (Seo et al., 2019). However, current phrase retrieval models heavily depend on sparse representations and still underper-form retriever-reader approaches. In this work, we show for the first time that we can learn dense representations of phrases alone that achieve much stronger performance in open-domain QA. We present an effective method to learn phrase representations from the supervision of reading comprehension tasks, coupled with novel negative sampling methods. We also propose a query-side fine-tuning strategy, which can support transfer learning and reduce the discrepancy between training and inference. On five popular open-domain QA datasets, our model DensePhrases improves over previous phrase retrieval models by 15%  25% absolute accuracy and matches the performance of state-of-the-art retriever-reader models. Our model is easy to parallelize due to pure dense representations and processes more than 10 questions per second on CPUs. Finally, we directly use our pre-indexed dense phrase representations for two slot filling tasks, showing the promise of utilizing DensePhrases as a dense knowledge base for downstream tasks. 1 1 Introduction Open-domain question answering (QA) aims to provide answers to natural-language questions using a large text corpus (Voorhees et al., 1999; Fer-rucci et al., 2010; Chen and Yih, 2020). While a dominating approach is a two-stage retriever-reader approach (Chen et al., 2017; Lee et al., 2019; Guu et al., 2020; Karpukhin et al., 2020), we focus on  Work partly done while visiting Princeton University. a recent new paradigm solely based on phrase retrieval (Seo et al., 2019; Lee et al., 2020). Phrase retrieval highlights the use of phrase representations and finds answers purely based on the similarity search in the vector space of phrases. 2 Without relying on an expensive reader model for processing text passages, it has demonstrated great runtime efficiency at inference time. Despite great promise, it remains a formidable challenge to build vector representations for every single phrase in a large corpus. Since phrase representations are decomposed from question representations, they are inherently less expressive than cross-attention models (Devlin et al., 2019). Moreover, the approach requires retrieving answers correctly out of billions of phrases (e.g., 6  10 10 phrases in English Wikipedia), making the scale of the learning problem difficult. Consequently, existing approaches heavily rely on sparse representations for locating relevant documents and paragraphs while still falling behind retriever-reader models (Seo et al., 2019; Lee et al., 2020). In this work, we investigate whether we can build fully dense phrase representations at scale for open-domain QA. First, we aim to learn strong phrase representations from the supervision of reading comprehension tasks. We propose to use data augmentation and knowledge distillation to learn better phrase representations within a single passage. We then adopt negative sampling strategies such as in-batch negatives (Henderson et al., 2017; Karpukhin et al., 2020), to better discriminate the phrases at a larger scale. Here, we present a novel method called pre-batch negatives , which leverages preceding mini-batches as negative examples to compensate the need of large-batch training. Lastly, we present a query-side fine-tuning strategy that dras-2 Following previous work (Seo et al., 2018), phrase' denotes any contiguous segment of text up to L words (including single words), which is not necessarily a linguistic phrase. tically improves phrase retrieval performance and allows for transfer learning to new domains, without re-building billions of phrase representations. As a result, all these improvements lead to a much stronger phrase retrieval model, without the use of any sparse representations (Table 1). We evaluate our model, DensePhrases , on five standard open-domain QA datasets and achieve much better accuracies than previous phrase retrieval models (Seo et al., 2019; Lee et al., 2020), with 15% 25% absolute improvement on most datasets. Our model also matches the performance of state-of-the-art retriever-reader models (Guu et al., 2020; Karpukhin et al., 2020). Due to the removal of sparse representations and careful design choices, we further reduce the storage footprint for the full English Wikipedia from 1.5TB to 320GB, as well as drastically improve the throughput. Finally, we envision that DensePhrases acts as a neural interface for retrieving phrase-level knowledge from a large text corpus. To showcase this possibility, we demonstrate that we can directly use DensePhrases for fact extraction, without rebuilding the phrase storage. With only fine-tuning the question encoder on a small number of subject-relation-object triples, we achieve state-of-the-art performance on two slot filling tasks (Petroni et al., 2021), using less than 5% of the training data. We first formulate the task of open-domain question answering for a set of K documents D = { d 1 , . . . , d K } . We follow the recent work (Chen et al., 2017; Lee et al., 2019) and treat all of English Wikipedia as D , hence K  5  10 6 . However, most approachesincluding oursare generic and could be applied to other collections of documents. The task aims to provide an answer  a for the input question q based on D . In this work, we focus on the extractive QA setting, where each answer is a segment of text, or a phrase , that can be found in D . Denote the set of phrases in D as S ( D ) and each phrase s k  S ( D ) consists of contiguous words w start ( k ) , . . . , w end ( k ) in its document d doc ( k ) . In practice, we consider all the phrases up to L = 20 words in D and S ( D ) comprises a large number of 6  10 10 phrases. An extractive QA system returns a phrase  s = argmax s S ( D ) f ( s |D , q ) where f is a scoring function. The system finally maps  s to an answer string  a : TEXT ( s ) =  a and the evaluation is typically done by comparing the predicted answer  a with a gold answer a  . Although we focus on the extractive QA setting, recent works propose to use a generative model as the reader (Lewis et al., 2020; Izacard and Grave, 2021), or learn a closed-book QA model (Roberts et al., 2020), which directly predicts answers without using an external knowledge source. The extractive setting provides two advantages: first, the model directly locates the source of the answer, which is more interpretable, and second, phrase-level knowledge retrieval can be uniquely adapted to other NLP tasks as we show in 7.3. Retriever-reader. A dominating paradigm in open-domain QA is the retriever-reader approach (Chen et al., 2017; Lee et al., 2019; Karpukhin et al., 2020), which leverages a first-stage document retriever f retr and only reads top K (cid:48) (cid:28) K documents with a reader model f read . The scoring function f ( s | D , q ) is decomposed as: f ( s | D , q ) = f retr ( { d j 1 , . . . , d j K (cid:48) } | D , q )  f read ( s | { d j 1 , . . . , d j K (cid:48) } , q ) , (1) where { j 1 , . . . , j K (cid:48) }  { 1 , . . . , K } and if s /  S ( { d j 1 , . . . , d j K (cid:48) } ) , the score will be 0. It can easily adapt to passages and sentences (Yang et al., 2019; Wang et al., 2019). However, this approach suffers from error propagation when incorrect documents are retrieved and can be slow as it usually requires running an expensive reader model on every retrieved document or passage at inference time. Phrase retrieval. Seo et al. (2019) introduce the phrase retrieval approach that encodes phrase and question representations independently and performs similarity search over the phrase representations to find an answer. Their scoring function f is computed as follows: f ( s | D , q ) = E s ( s, D ) (cid:62) E q ( q ) , (2) where E s and E q denote the phrase encoder and the question encoder respectively. As E s (  ) and E q (  ) representations are decomposable, it can support maximum inner product search (MIPS) and improve the efficiency of open-domain QA models. Previous approaches (Seo et al., 2019; Lee et al., 2020) leverage both dense and sparse vectors for phrase and question representations by taking their concatenation: E s ( s, D ) = [ E sparse ( s, D ) , E dense ( s, D )] . 3 However, since the sparse vectors are difficult to parallelize with dense vectors, their method essentially conducts sparse and dense vector search separately. The goal of this work is to only use dense representations, i.e., E s ( s, D ) = E dense ( s, D ) , which can model f ( s | D , q ) solely with MIPS, as well as close the gap in performance. We introduce DensePhrases, a phrase retrieval model that is built on fully dense representations. Our goal is to learn a phrase encoder as well as a question encoder, so we can pre-index all the possible phrases in D , and efficiently retrieve phrases for any question through MIPS at testing time. We outline our approach as follows: 3 Seo et al. (2019) use sparse representations of both paragraphs and documents and Lee et al. (2020) use contextualized sparse representations conditioned on the phrase. We first learn a high-quality phrase encoder and an (initial) question encoder from the supervision of reading comprehension tasks (4.1), as well as incorporating effective negative sampling to better discriminate phrases at scale (4.2, 4.3). Then, we fix the phrase encoder and encode all the phrases s  S ( D ) and store the phrase indexing offline to enable efficient search (5). Finally, we introduce an additional strategy called query-side fine-tuning (6) by further updating the question encoder. 4 We find this step to be very effective, as it can reduce the discrepancy between training (the first step) and inference, as well as support transfer learning to new domains. Our base architecture consists of a phrase encoder E s and a question encoder E q . Given a passage p = w 1 , . . . , w m , we denote all the phrases up to L tokens as S ( p ) . Each phrase s k has start and end in-dicies start ( k ) and end ( k ) and the gold phrase is s   S ( p ) . Following previous work on phrase or span representations (Lee et al., 2017; Seo et al., 2018), we first apply a pre-trained language model M p to obtain contextualized word representations for each passage token: h 1 , . . . , h m  R d . Then, we can represent each phrase s k  S ( p ) as the concatenation of corresponding start and end vectors: E s ( s k , p ) = [ h start ( k ) , h end ( k ) ]  R 2 d . A great advantage of this representation is that we eventually only need to index and store all the word vectors (we use W ( D ) to denote all the words in D ), instead of all the phrases S ( D ) , which is at least one magnitude order smaller. Similarly, we need to learn a question encoder E q (  ) that maps a question q =  w 1 , . . . ,  w n to a vector of the same dimension as E s (  ) . Since the start and end representations of phrases are produced by the same language model, we use another two different pre-trained encoders M q, start and M q, end to differentiate the start and end positions. We apply M q, start and M q, end on q separately and obtain representations q start and q end 4 In this paper, we use the term question and query interchangeably as our question encoder can be naturally extended to unnatural queries. taken from the [CLS] token representations respectively. Finally, E q (  ) simply takes their concatenation: E q ( q ) = [ q start , q end ]  R 2 d . (4) Note that we use pre-trained language models to initialize M p , M q, start and M q, end and they are fine-tuned with the objectives that we will define later. In our pilot experiments, we found that SpanBERT (Joshi et al., 2020) leads to superior performance compared to BERT (Devlin et al., 2019). SpanBERT is designed to predict the information in the entire span from its two endpoints, therefore it is well suited for our phrase representations. In our final model, we use SpanBERT-base-cased as our base LMs for E s and E q , and hence d = 768 . 5 See Table 5 for an ablation study. In this section, we start by learning dense phrase representations from the supervision of reading comprehension tasks, i.e., a single passage p contains an answer a  to a question q . Our goal is to learn strong dense representations of phrases for s  S ( p ) , which can be retrieved by a dense representation of the question and serve as a direct 5 Our base model is largely inspired by DenSPI (Seo et al., 2019), although we deviate from theirs as follows. (1) We remove coherency scalars and don't split any vectors. (2) DenSPI uses a shared encoder for phrases and questions while we use 3 separate language models initialized from the same pre-trained model. (3) We use SpanBERT instead of BERT. answer (4.1). Then, we introduce two different negative sampling methods (4.2, 4.3), which encourage the phrase representations to be better discriminated at the full Wikipedia scale. See Figure 1 for an overview of DensePhrases. To learn phrase representations in a single passage along with question representations, we first maximize the log-likelihood of the start and end positions of the gold phrase s  where TEXT ( s  ) = a  . The training loss for predicting the start position of a phrase given a question is computed as: z start 1 , . . . , z start m = [ h (cid:62) 1 q start , . . . , h (cid:62) m q start ] , P start = softmax ( z start 1 , . . . , z start m ) , L start =  log P start start ( s  ) . (5) We can define L end in a similar way and the final loss for the single-passage training is L single = L start + L end 2 . This essentially learns reading comprehension without any cross-attention between the passage and the question tokens, which fully decomposes phrase and question representations. Data augmentation Since the contextualized word representations h 1 , . . . , h m are encoded in a query-agnostic way, they are always inferior to query-dependent representations in cross-attention models (Devlin et al., 2019), where passages are fed along with the questions concatenated by a special token such as [SEP] . We hypothesize that one key reason for the performance gap is that reading comprehension datasets only provide a few annotated questions in each passage, compared to the set of possible answer phrases. Learning from this supervision is not easy to differentiate similar phrases in one passage (e.g., s  = Charles, Prince of Wales and another s = Prince George for a question q = Who is next in line to be the monarch of England? ). Following this intuition, we propose to use a simple model to generate additional questions for data augmentation, based on a T5-large model (Raf-fel et al., 2020). To train the question generation model, we feed a passage p with the gold answer s  highlighted by inserting surrounding special tags. Then, the model is trained to maximize the log-likelihood of the question words of q . After training, we extract all the named entities in each training passage as candidate answers and feed the passage p with each candidate answer to generate questions. We keep the question-answer pairs only when a cross-attention reading comprehension model 6 makes a correct prediction on the generated pair. The remaining generated QA pairs { ( q 1 ,  s 1 ) , ( q 2 ,  s 2 ) , . . . , ( q r ,  s r ) } are directly augmented to the original training set. Distillation We also propose improving the phrase representations by distilling knowledge from a cross-attention model (Hinton et al., 2015). We minimize the KullbackLeibler divergence between the probability distribution from our phrase encoder and that from a standard SpanBERT-base QA model. The loss is computed as follows: L distill = KL ( P start || P start c ) + KL ( P end || P end c ) 2 , (7) where P start (and P end ) is defined in Eq. (5) and P start c and P end c denote the probability distributions used to predict the start and end positions of answers in the cross-attention model. Eventually, we need to build phrase representations for billions of phrases. Therefore, a bigger challenge is to incorporate more phrases as negatives so the representations can be better discriminated 6 SpanBERT-large, 88.2 EM on SQuAD. (a) In-batch Negatives ( ) B  1 at a larger scale. While Seo et al. (2019) simply sample two negative passages based on question similarity, we use in-batch negatives for our dense phrase representations, which has been shown to be effective in learning dense passage representations before (Karpukhin et al., 2020). As shown in Figure 2 (a), for the i -th example in a mini-batch of size B , we denote the hidden representations of the gold start and end positions h start ( s  ) and h end ( s  ) as g start i and g end i , as well as the question representation as [ q start i , q end i ] . Let G start , G end , Q start , Q end be the B  d matrices and each row corresponds to g start i , g end i , q start i , q end i respectively. Basically, we can treat all the gold phrases from other passages in the same mini-batch as negative examples. We compute S start = Q start G start (cid:124) and S end = Q end G end (cid:124) and the i -th row of S start and S end return B scores each, including a positive score and B  1 negative scores: s start 1 , . . . , s start B and s end 1 , . . . , s end B . Similar to Eq. (5), we can compute the loss function for the i -th example as: P start_ib i = softmax ( s start 1 , . . . , s start B ) , P end_ib i = softmax ( s end 1 , . . . , s end B ) , L neg =  log P start_ib i + log P end_ib i 2 , (8) We also attempted using non-gold phrases from other passages as negatives but did not find a meaningful improvement. The in-batch negatives usually benefit from a large batch size (Karpukhin et al., 2020). However, it is challenging to further increase batch sizes, as they are bounded by the size of GPU memory. Next, we propose a novel negative sampling method called pre-batch negatives , which can effectively utilize the representations from the preceding C mini-batches (Figure 2 (b)). In each iteration, we maintain a FIFO queue of C mini-batches to cache phrase representations G start and G end . The cached phrase representations are then used as negative samples for the next iteration, providing B  C additional negative samples in total. 7 These pre-batch negatives are used together with in-batch negatives and the training loss is the same as Eq. (8), except that the gradients are not back-propagated to the cached pre-batch negatives. After warming up the model with in-batch negatives, we simply shift from in-batch negatives ( B  1 negatives) to in-batch and pre-batch negatives (hence a total number of B  C + B  1 negatives). For simplicity, we use L neg to denote the loss for both in-batch negatives and pre-batch negatives. Since we do not retain the computational graph for pre-batch negatives, the memory consumption of pre-batch negatives is much more manageable while allowing an increase in the number of negative samples. Finally, we optimize all the three losses together, on both annotated reading comprehension examples and generated questions from 4.1: where  1 ,  2 ,  3 determine the importance of each loss term. We found that  1 = 1 ,  2 = 2 , and  3 = 4 works well in practice. See Table 5 and Table 6 for an ablation study of different components. Indexing After training the phrase encoder E s , we need to encode all the phrases S ( D ) in the entire English Wikipedia D and store an index of the phrase dump. We segment each document d i  D into a set of natural paragraphs, from which we obtain token representations for each paragraph using E s (  ) . Then, we build a phrase dump H = [ h 1 , . . . , h |W ( D ) | ]  R |W ( D ) | d by stacking the token representations from all the paragraphs in D . Note that this process is computationally expensive and takes about hundreds of GPU hours with a large disk footprint. To reduce the 7 This approach is inspired by the momentum contrast idea proposed in unsupervised visual representation learning (He et al., 2020). Contrary to their approach, we have separate encoders for phrases and questions and back-propagate to both during training without a momentum update. size of phrase dump, we follow and modify several techniques introduced in Seo et al. (2019) (see Appendix E for details). After indexing, we can use two rows i and j of H to represent a dense phrase representation [ h i , h j ] . We use faiss (Johnson et al., 2017) for building a MIPS index of H . 8 Search For a given question q , we can find the answer  s as follows:  s = argmax s ( i,j ) E s ( s ( i,j ) , D ) (cid:62) E q ( q ) , = argmax s ( i,j ) ( Hq start ) i + ( Hq end ) j , (10) where s ( i,j ) denotes a phrase with start and end indices as i and j in the index H . We can compute the argmax of Hq start and Hq end efficiently by performing MIPS over H with q start and q end . In practice, we search for the topk start and topk end positions separately and perform a constrained search over their end and start positions respectively such that 1  i  j < i + L  |W ( D ) | . So far, we have created a phrase dump H that supports efficient MIPS search. In this section, we propose a novel method called query-side fine-tuning by only updating the question encoder E q to correctly retrieve a desired answer a  for a question q given H . Formally speaking, we optimize the marginal log-likelihood of the gold answer a  for a question q , which resembles the weakly-supervised QA setting in previous work (Lee et al., 2019; Min et al., 2019). For every question q , we retrieve top k phrases and minimize the objective: L query =  log (cid:80) s   S ( q ) , TEXT ( s )= a  exp (cid:0) f ( s |D ,q ) (cid:1) (cid:80) s   S ( q ) exp (cid:0) f ( s |D ,q ) (cid:1) , (11) where f ( s |D , q ) is the score of the phrase s (Eq. (2)) and  S ( q ) denotes the top k phrases for q (Eq. (10)). In practice, we use k = 100 for all the experiments. There are several advantages for doing this: (1) we find that query-side fine-tuning can reduce the discrepancy between training and inference, and hence improve the final performance substantially (8). Even with effective negative sampling, the model only sees a small portion of passages compared to the full scale of D and this training objective can effectively fill in the gap. (2) This training strategy allows for transfer learning to unseen 8 We use IVFSQ4 with 1M clusters and set n-probe to 256. domains, without rebuilding the entire phrase index. More specifically, the model is able to quickly adapt to new QA tasks (e.g., WebQuestions) when the phrase dump is built using SQuAD or Natural Questions. We also find that this can transfers to non-QA tasks when the query is written in a different format. In 7.3, we show the possibility of directly using DensePhrases for slot filling tasks by using a query such as (Michael Jackson, is a singer of, x) . In this regard, we can view our model as a dense knowledge base that can be accessed by many different types of queries and it is able to return phrase-level knowledge efficiently. Datasets. We use two reading comprehension datasets: SQuAD (Rajpurkar et al., 2016) and Natural Questions (NQ) (Kwiatkowski et al., 2019) to learn phrase representations, in which a single gold passage is provided for each question. For the open-domain QA experiments, we evaluate our approach on five popular open-domain QA datasets: Natural Questions, WebQuestions (WQ) (Berant et al., 2013), CuratedTREC (TREC) (Baudi and ediv`y, 2015), TriviaQA (TQA) (Joshi et al., 2017), and SQuAD. Note that we only use SQuAD and/or NQ to build the phrase index and perform query-side fine-tuning (6) for other datasets. We also evaluate our model on two slot filling tasks, to show how to adapt our DensePhrases for other knowledge-intensive NLP tasks. We focus on using two slot filling datasets from the KILT benchmark (Petroni et al., 2021): T-REx (Elsahar et al., 2018) and zero-shot relation extraction (Levy et al., 2017). Each query is provided in the form of {subject entity} [SEP] {relation}" and the answer is the object entity. Appendix C provides the statistics of all the datasets. Implementation details. We denote the training datasets used for reading comprehension (Eq. (9)) as C phrase . For open-domain QA, we train two versions of phrase encoders, each of which are trained on C phrase = { SQuAD } and { NQ , SQuAD } , respectively. We build the phrase dump H for the 2018-12-20 Wikipedia snapshot and perform query-side fine-tuning on each dataset using Eq. (11). For slot filling, we use the same phrase dump for open-domain QA, C phrase = { NQ , SQuAD } and perform query-side fine-tuning on randomly sampled 5K Model SQuAD NQ (Long) EM F1 EM F1 Query-Dependent BERT-base 80.8 88.5 69.9 78.2 SpanBERT-base 85.7 92.2 73.2 81.0 Query-Agnostic DilBERT (Siblini et al., 2020) 63.0 72.0 -DeFormer (Cao et al., 2020) -72.1 -DenSPI  73.6 81.7 68.2 76.1 DenSPI + Sparc  76.4 84.8 -DensePhrases (ours) 78.3 86.3 71.9 79.6 Table 2: Reading comprehension results, evaluated on the development sets of SQuAD and Natural Questions. or 10K training examples to see how rapidly our model adapts to the new query types. See Appendix D for details on the hyperparameters and Appendix A for an analysis of computational cost. Reading comprehension. In order to show the effectiveness of our phrase representations, we first evaluate our model in the reading comprehension setting for SQuAD and NQ and report its performance with other query-agnostic models (Eq. (9) without query-side fine-tuning). This problem was originally formulated by Seo et al. (2018) as the phrase-indexed question answering (PIQA) task. Compared to previous query-agnostic models, our model achieves the best performance of 78.3 EM on SQuAD by improving the previous phrase retrieval model (DenSPI) by 4 . 7% (Table 2). Although it is still behind cross-attention models, the gap has been greatly reduced and serves as a strong starting point for the open-domain QA model. Open-domain QA. Experimental results on open-domain QA are summarized in Table 3. Without any sparse representations, DensePhrases outperforms previous phrase retrieval models by a large margin and achieves a 15%  25% absolute improvement on all datasets except SQuAD. Training the model of Lee et al. (2020) on C phrase = { NQ , SQuAD } only increases the result from 14.5% to 16.5% on NQ, demonstrating that it does not suffice to simply add more datasets for training phrase representations. Our performance is also competitive with recent retriever-reader models (Karpukhin et al., 2020), while running much faster during inference (Table 1). Table 4 summarizes the results on the two slot filling datasets, along with the baseline scores provided by Petroni et al. (2021). The only extractive baseline is DPR + BERT, which performs poorly in zero-shot relation extraction. On the other hand, our model achieves competitive performance on all datasets and achieves state-of-the-art performance on two datasets using only 5K training examples. Ablation of phrase representations. Table 5 shows the ablation result of our model on SQuAD. Upon our choice of architecture, augmenting training set with generated questions (QG = (cid:51) ) and performing distillation from cross-attention models (Distill = (cid:51) ) improve performance up to EM = 78.3. We attempted adding the generated questions to the training of the SpanBERT-QA model but find a 0.3% improvement, which validates that data sparsity is a bottleneck for query-agnostic models. Effect of batch negatives. We further evaluate the effectiveness of various negative sampling methods introduced in 4.2 and 4.3. Since it is computationally expensive to test each setting at the full Wikipedia scale, we use a smaller text corpus D small of all the gold passages in the development sets of Natural Questions, for the ablation study. Empirically, we find that results are generally well correlated when we gradually increase the size of |D| . As shown in Table 6, both in-batch and pre-batch negatives bring substantial improvements. While using a larger batch size ( B = 84 ) is beneficial for in-batch negatives, the number of preceding batches in pre-batch negatives is optimal when C = 2 . Surprisingly, the pre-batch negatives also improve the performance when D = { p } . Effect of query-side fine-tuning. We summarize the effect of query-side fine-tuning in Table 7. For the datasets that were not used for training the phrase encoders (TQA, WQ, TREC), we observe a 15% to 20% improvement after query-side fine-tuning. Even for the datasets that have been used (NQ, SQuAD), it leads to significant improvements (e.g., 32.6%  40.9% on NQ for C phrase = {NQ}) and it clearly demonstrates it can effectively reduce the discrepancy between training and inference. Learning effective dense representations of words is a long-standing goal in NLP (Bengio et al., 2003; Collobert et al., 2011; Mikolov et al., 2013; Peters et al., 2018; Devlin et al., 2019). Beyond words, dense representations of many different granularities of text such as sentences (Le and Mikolov, 2014; Kiros et al., 2015) or documents (Yih et al., 2011) have been explored. While dense phrase representations have been also studied for statistical machine translation (Cho et al., 2014) or syntactic parsing (Socher et al., 2010), our work focuses on learning dense phrase representations for QA and any other knowledge-intensive tasks where phrases can be easily retrieved by performing MIPS. This type of dense retrieval has been also studied for sentence and passage retrieval (Humeau et al., 2019; Karpukhin et al., 2020) (see Lin et al., 2020 for recent advances in dense retrieval). While DensePhrases is explicitly designed to retrieve phrases that can be used as an answer to given queries, retrieving phrases also naturally entails retrieving larger units of text, provided the datastore maintains the mapping between each phrase and the sentence and passage in which it occurs. In this study, we show that we can learn dense representations of phrases at the Wikipedia scale, which are readily retrievable for open-domain QA and other knowledge-intensive NLP tasks. We learn both phrase and question encoders from the supervision of reading comprehension tasks and introduce two batch-negative techniques to better discriminate phrases at scale. We also introduce query-side fine-tuning that adapts our model to different types of queries. We achieve strong performance on five popular open-domain QA datasets, while reducing the storage footprint and improving latency significantly. We also achieve strong performance on two slot filling datasets using only a small number of training examples, showing the possibility of utilizing our DensePhrases as a knowledge base. We thank Sewon Min, Hyunjae Kim, Gyuwan Kim, Jungsoo Park, Zexuan Zhong, Dan Friedman, Chris Sciavolino for providing valuable comments and feedback. This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HR20C0021) and National Research Foundation of Korea (NRF-2020R1A2C3010638). It was also partly supported by the James Mi *91 Research Innovation Fund for Data Science and an Amazon Research Award. Our work builds on standard reading comprehension datasets such as SQuAD to build phrase representations. SQuAD, in particular, is created from a small number of Wikipedia articles sampled from top-10,000 most popular articles (measured by PageRanks), hence some of our models trained only on SQuAD could be easily biased towards the small number of topics that SQuAD contains. We hope that excluding such datasets during training or inventing an alternative pre-training procedure for learning phrase representations could mitigate this problem. Although most of our efforts have been made to reduce the computational complexity of previous phrase retrieval models (further detailed in Appendices A and E), leveraging our phrase retrieval model as a knowledge base will inevitably increase the minimum requirement for the additional experiments. We plan to apply vector quantization techniques to reduce the additional cost of using our model as a KB.
abstain abstain objective objective objective result abstain result abstain method abstain abstain abstain abstain abstain abstain abstain objective objective objective method objective method abstain abstain result abstain result method objective result objective method other other abstain other method other other abstain objective other other other other other other other other other other other objective method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain method method method abstain method abstain abstain other method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain method method other method method method abstain method other abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method method method method method abstain abstain abstain method abstain method method abstain method method abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain other other method other other result abstain abstain result result other other other method method method method method