To alleviate the data scarcity problem in training question answering systems, recent works propose additional intermediate pre-training for dense passage retrieval (DPR). However, there still remains a large discrepancy between the provided upstream signals and the downstream question-passage relevance, which leads to less improvement. To bridge this gap, we propose the H yper L ink-induced P re-training (HLP), a method to pre-train the dense retriever with the text relevance induced by hyperlink-based topology within Web documents. We demonstrate that the hyperlink-based structures of dual-link and co-mention can provide effective relevance signals for large-scale pre-training that better facilitate downstream passage retrieval. We investigate the effectiveness of our approach across a wide range of open-domain QA datasets under zero-shot, few-shot, multihop, and out-of-domain scenarios. The experiments show our HLP outperforms the BM25 by up to 7 points as well as other pre-training methods by more than 10 points in terms of top-20 retrieval accuracy under the zero-shot scenario. Furthermore, HLP significantly outperforms other pre-training methods under the other scenarios. Open-domain question answering (OpenQA) aims to answer factual open questions with a large external corpus of passages. Current approaches to OpenQA usually adopt a two-stage retriever-reader paradigm (Chen et al., 2017; Zhu et al., 2021) to fetch the final answer span. The performance of OpenQA systems is largely bounded by the retriever as it determines the evidential documents for the reader to examine. Traditional retrievers, such as TF-IDF and BM25 (Robertson and Zaragoza, 2009), are considered incapable of adapting to sce-Our code and trained models are available at https: //github.com/jzhoubu/HLP . narios where deep semantic understanding is required. Recent works (Lee et al., 2019; Karpukhin et al., 2020; Qu et al., 2021) show that by fine-tuning pre-trained language models on sufficient downstream data, dense retrievers can significantly outperform traditional term-based retrievers. Considering the data-hungry nature of the neural retrieval models, extensive efforts (Lee et al., 2019; Chang et al., 2019; Sachan et al., 2021) have been made to design self-supervised tasks to pre-train the retriever. However, these pre-training tasks construct relevance signals largely depending on easily attainable sentence-level or document-level contextual relationships. For example, the relationship between a sentence and its originated context (shown by the ICT query in Figure 1) may not be sufficient enough to facilitate question-passage matching for the tasks of OpenQA. We also find that these pre-trained retrievers still fall far behind BM25 in our pilot study on the zero-shot experiment. In order to address the shortcomings of the matching-oriented pre-training tasks as mentioned above, we propose a pre-training method with better surrogates of real natural question-passage (Q-P) pairs. We consider two conditions of relevance within Q-P pairs, which is similar to the process of distantly supervised retriever learning (Mintz et al., 2009; Chen et al., 2017). 1) Evidence Existence The evidence, such as entities and their corresponding relations, should exist across the query and the targeted passage as they both discuss similar facts or events related to the answer. 2) Answer Containing The golden passage should contain the answer of the query, which means that a text span within the passage can provide the information-seeking target of the query. In this paper, we propose H yper L ink-induced P re-training (HLP), a pre-training method to learn effective Q-P relevance induced by the hyperlink topology within naturally-occurring Web documents. Specifically, these Q-P pairs are automatically extracted from the online documents with relevance adequately designed via hyperlink-based topology to facilitate downstream retrieval for question answering. Figure 1 shows an example of comparison between the human-written query and different pseudo queries. By the guidance of hyperlinks, our HLP query hold the relevance of answer containing with the passage (query title occurs in the passage). Meanwhile, the HLP query can introduce far more effective relevance of evidence existence than other pseudo queries by deeply mining the hyperlink topology, e.g., the dual-link structure. In figure 1, both HLP query and the passage both contain information corresponding to the same fact of Mitja Okorn directed the film of Letters to Santa . This makes our pseudo query low-cost and a good surrogate for the manually written query. Our contributions are two-fold. First, we present a hyperlink-induced relevance construction methodology that can better facilitate downstream passage retrieval for question answering, and specifically, we propose a pre-training method: Hyperlink-induced Pre-training (HLP). Second, we conduct evaluations on six popular QA datasets, investigating the effectiveness of our approach under zero-shot, few-shot, multi-hop, and out-of-domain (OOD) scenarios. The experiments show HLP outperforms BM25 in most of the cases under the zero-shot scenario and other pre-training methods under all scenarios. Dense Retriever Pre-training Previous works have attempted to conduct additional pre-training for dense retrievers on various weakly supervised data. Borisov et al. (2016) and Dehghani et al. (2017) pre-trained ranking models on click-logs and BM25-induced signals respectively for web search. Lee et al. (2019) proposed the inverse cloze task (ICT) to pre-train a dense retrieval model, which randomly selects sentences as pseudo queries, and matched them to the passages that they originate from. Besides, Chang et al. (2019) proposed the pre-training task of wiki link prediction (WLP) and body first selection (BFS) tasks. Similar to our work, the WLP task also leveraged the hyperlinks within Wikipedia to construct relevant text pairs. However, as shown in figure 1, the WLP pseudo query can only ensure the weak doc-wise contextual relationship with the passage. Guu et al. (2020) proposed the masked-salient-span pre-training task which optimizes a retrieval model by the distant supervision of language model objective. As a follow-up, Sachan et al. (2021) combined ICT with the masked-salient-span task and further improved the pre-training effectiveness. Data Augmentation via Question Generation Ma et al. (2021), Reddy et al. (2021) and Oguz et al. (2021) all investigate training a dense retriever on questions synthesized by large question generative (QG) models. Targeting on the zero-shot setting, Ma et al. (2021) trained a question generator on general-domain question passage pairs from community platforms and publicly available academic datasets. Reddy et al. (2021) focused more on domain transfer and trained the QG model on QA datasets of Wikipedia articles. Oguz et al. (2021) uses the synthetically generated questions from PAQ dataset (Lewis et al., 2021) and the post-comment pairs from dataset of Reddit conversations for retrieval pre-training. Recently, Shinoda et al. (2021) reveals that the QG models tend to generate questions with high lexical overlap which amplify the bias of QA dataset. Different to these studies, our method focuses on a more general setting where the retriever is only trained with the naturally occurring web documents, and has no access to any downstream datasets. In this section, we firstly discuss the background of OpenQA retrieval, then our methodology and training framework. Passage Retrieval Given a question q , passage retrieval aims to provide a set of relevant passages p from a large corpus D . Our work adopts Wikipedia as source corpus and each passage is a disjoint segment within a document from D . OpenQA Q-P Relevance For OpenQA, a passage p is considered relevant to the query q if p conveys similar facts and contains the answer to q . These two conditions of relevance, namely evidence existence and answer containing, are properly introduced into the HLP Q-P pairs under the guidance of desired hyperlink structure. We will discuss more in this section. To better formulate the relevance of pseudo Q-P pairs, we denote the sequence of passages within a document as A = [ a 1 , a 2 , ..., a n A ] where A  D . The corresponding topical entity and the title of document A and its passage splits are denoted as e A and t A , respectively. We use m A to indicate a mention of entity e A , which is a hypertext span linking to document A . Note that the mention span m A is usually identical to the document title t A or a variant version of it. Further, we define F ( p ) as the entity-level factual information conveyed by the passage p , which is a set consists of the topical entity e P and the entities mentioned within passage p . Evidence Existence in HLP With appropriately designed hyperlink topologies, our HLP Q-P pairs guarantee the co-occurrence of entities which are presented as hypertext or topics in q and p . This is considered as evidence across the Q-P pairs: F ( q )  F ( p )  =  (1) Furthermore, we conjecture that HLP is more likely to achieve fact-level relevance than entity-level overlap. We conduct human evaluation in Section 6.3 and case studies in Appendix G to support this conjecture. Moreover, we demonstrate that any Q-P pair containing hyperlink-induced factual evidence, which can be represented as triples, is included in our proposed topologies, which are included in Appendix E. Answer Containing in HLP We consider the document title t Q as the information-seeking target of q . Accordingly, the relevance of answer containing can be formulated as t Q  p (2) The rationale behind this is that both the natural question and the Wikipedia document are intended to describe related facts and events regarding a targeted object, whereas the object is an answer for a question but a topical entity for a Wikipedia document. This similarity leads us to take the document title as the information-seeking target of its context. Based on analysis of how queries match their evidential passages in the NQ (Kwiatkowski et al., 2019) dataset, we propose two kinds of hyperlink topology for relevance construction: Dual-link and Co-mention. We present our exploratory data analysis on NQ dataset in Appendix C. Here we discuss the desired hyperlink topologies and the corresponding relevance of the pseudo Q-P pairs. Dual-link (DL) Among all NQ training samples, 55% of questions mention the title of their corresponding golden passage. This observation motivates us to leverage the topology of dual-link (DL) for relevance construction. We consider a passage pair ( a i , b j ) follows the dual-link topology if they link to each other. An example of a DL pair ( a i , b j ) is shown in Figure 2, in which passage b j mentions the title of document A as m A , satisfying the condition of answer containing: t A  m A and m A  b j (3) Further, since the passages a i and b j both mention the topical entity of the other, the entities e A and e B appear in both passages as evidence: { e A , e B }  F ( a i )  F ( b j ) (4) Co-mention (CM) Among all NQ training samples, about 40% of questions fail to match the dual-link condition but mention the same third-party entity as their corresponding golden passages. In light of this observation, we utilize another topology of Co-mention (CM). We consider that a passage pair ( c k , d l ) follows the Co-mention topology if they both link to a third-party document E and d l links to c k . Figure 2 illustrates a CM pair ( c l , d k ) where answer containing is ensured as the title of c k occurs in d l : t C  m C and m C  d l (5) Since both c l and d k mention a third-party entity e E , and that e C is a topical entity in c l while a mentioned entity in d k , we have entity-level evidence across c l and d k as: { e C , e E }  F ( c k )  F ( d l ) (6) In practice, we use sentence-level queries which contain the corresponding evidential hypertext, and we do not prepend the title to the passage in order to reduce the superficial entity-level overlap. To improve the quality of CM pairs, we filter out those with a co-mentioned entity which has a top 10% highest-ranked in-degree among the Wikipedia entity. We also present pseudo code in Appendix D to illustrate how we construct our pseudo Q-P pairs. Furthermore, we highlight that HLP has the following advantages: 1) it introduces more semantic variants and paraphrasing for better text matching. 2) The hypertext reflects potential interests or needs of users in relevant information, which is consistent to the downstream information-seeking propose. We adopt a BERT-based bi-encoder to encode queries and passages separately into d-dimension vectors. The output representation is derived from the last hidden state of the [CLS] token and the final matching score is measured by the inner product: h q = BERTQ ( q )([CLS]) h p = BERTP ( p )([CLS]) S( p, q ) = h T q  h p Let B = { q i , p + i , p  i } n i =1 be a mini-batch with n instances. Each instance contains a question q i paired with a positive passage p + i and a negative passage p  i . With in-batch negative sampling, each question q i considers all the passages in B except its own gold p + i as negatives, resulting in 2 n  1 negatives per question in total. We use the negative log likelihood of the positive passage as our loss for optimization: L ( q i , p + i , p  i, 1 , ..., p  i, 2 n  1 ) =  log e S ( q i ,p + i ) e S ( q i ,p + i ) + (cid:80) 2 n  1 j =1 e S ( q i ,p  i,j ) 4 Experimental Setup In this session, we discuss the pre-training corpus preparation, downstream datasets, the hyperparameter and the basic setup for our experiments. We adopt Wikipedia as our source corpus D for pretraining as it is the largest encyclopedia covering diverse topics with good content quality and linking structures. We choose the snapshot 03-01-2021 of an English Wikipedia dump, and process it with WikiExtractor 2 to obtain clean context. After filtering out documents with blank text or a title less than three letters, following previous work (Karpukhin et al., 2020), we split the remaining documents into disjoint chunks of 100 words as passages, resulting in over 22 million passages in the end. We evaluate our method on several open-domain question answering benchmarks which are shown below. Natural Questions (NQ) (Kwiatkowski et al., 2019) is a popular QA dataset with real queries from Google Search and annotated answers from Wikipedia. TriviaQA (Joshi et al., 2017) contains question-answer pairs scraped from trivia websites. entity-level answers from Freebase. HotpotQA (Fullwiki) (Yang et al., 2018) is a human-annotated multi-hop question answering dataset. BioASQ (Tsatsaronis et al., 2015) is a competition on biomedical semantic indexing and question answering. We evaluate its factoid questions from task 8B. MS MARCO (Passage Ranking) (Nguyen et al., 2016) consists of real-world user queries and a large collection of Web passages extracted by Bing search engine. Retrieval Corpus For downstream retrieval, we use the 21M Wikipedia passages provided by DPR (Karpukhin et al., 2020) for NQ, TriviaQA and WQ. For BioASQ, we take the abstracts of PubMed articles from task 8A with the same split to Reddy et al. (2021)'s work. For HotpotQA and MS MARCO, we use the official corpus. During the pre-training, we train the bi-encoder for 5 epochs with parameters shared, using a batch size of 400 and an Adam optimizer (Kingma and Ba, 2014) with a learning rate 2  10  5 , linear scheduling with 10% warm-up steps. Our HLP and all the reproduced baselines are trained on 20 million Q-P pairs with in-batch negative sampling, and the best checkpoints are selected based on the average rank of gold passages evaluated on the NQ dev set. The pre-training takes around 3 days using eight NVIDIA V100 32GB GPUs. For the downstream, we use the same hyper-parameters for all experiments. Specifically, we fine-tune the pre-trained models for 40 epochs with a batch size of 256 and the same optimizer and learning rate settings to the pre-training. We conduct evaluation on respective dev sets to select best checkpoints, and we use the last checkpoint if there is no dev set or test set (e.g. HotpotQA). Most existing baselines have been implemented under different experimental settings, which have a substantial effect on the retrieval performance. To ensure fairness, we reproduce several pre-training methods (ICT, WLP, BFS, and their combination) under the same experimental setting, such as batch size, base model, amount of pre-training data, and so on. The only difference between our method and the re-implemented baselines is the self-supervision signal derived from the respective pre-training samples. Our reproduced BM25 baseline is better than that reported in Karpukhin et al. (2020), and the re-implemented pre-training methods also perform better than those reported by the recent work 3 . In addition, we include the work REALM (Guu et al., 2020) as a baseline which has recently been reproduced by Sachan et al. (2021) using 240 GPUs and is named masked salient spans (MSS). We note that most related works gain improvements from varying downstream setting or synthetic pre-training with access to the downstream data of respective domain, which is out of the scope of our interests. Table 1 shows the retrieval accuracy of different models on three popular QA datasets under zero-shot and full-set fine-tuning settings. Under zero-shot setting, HLP consistently outperforms BM25 except for the top-5 retrieval accuracy of TriviaQA, while all other pre-training baselines are far behind. We attribute the minor improvement over BM25 on TriviaQA to a high overlap between questions and passages, which gives term-based retriever a clear advantage. We investigate the coverage of the question tokens that appear in the gold passage and find that the overlap is indeed higher in TriviaQA (62.8%) than NQ (60.7%) and WQ (57.5%). After fine-tuning, all models with intermediate pre-training give better results than the vanilla DPR while our HLP achieves the best in nearly all cases. 3 Our reproduced ICT and BFS surpass the reproduction from recent work (Oguz et al., 2021) by 15 and 12 points, respectively, in terms of top-20 retrieval accuracy on NQ test set under zero-shot setting. Among ICT, WLP and BFS, we observe that WLP is the most competitive with or without fine-tuning, and additional improvements can be achieved by combining three of them. This observation indicates that pre-training with diverse relevance leads to better generalization to downstream tasks, while document-wise relevance is more adaptable for the OpenQA retrieval. The advantage of document-wise relevance may come from the fact that texts in different documents are likely written by different parties, providing less superficial cues for text matching, which is beneficial for the downstream retrieval. Our HLP learns both coarse-grained document-wise relationships as well as the fine-grained entity-level evidence, which results in a significant improvement. To investigate the retrieval effectiveness in a more realistic scenario, we conduct experiments for few-shot learning. Specifically, we fine-tune the pre-trained models on large datasets (NQ, TriviaQA) with m ( m  { 16 , 256 , 1024 } ) samples and present the few-shot retrieval results in Table 2. With only a few hundred labeled data for fine-tuning, all the models with intermediate pretraining perform better than those without, and HLP outperforms the others by a larger margin when m is smaller. Moreover, among three reimplemented baselines, WLP gains the largest improvement with increasing number of samples, outperforming ICT and BFS when a thousand labelled samples are provided for fine-tuning. While HLP is pre-trained on Wikipedia pages, we conduct additional experiments on BioASQ and MS MARCO datasets with non-Wikipedia corpus to further verify its out-of-domain (OOD) generalization. Following Gururangan et al. (2020), we measure the similarity between corpus by computing the vocabulary overlap of the top 10K frequent words (excluding stopwords). We observe a vocabulary overlap of 36.2% between BioASQ and Wikipedia while 61.4% between MS MARCO and Wikipedia, indicating that these two domains differ considerably from our pre-training corpus. and MS MARCO datasets are presented in Table 4. For BioASQ, HLP is competitive with both BM25 and AugDPR(Reddy et al., 2021) while significantly outperforming ICT, WLP, and BFS. Note that AugDPR is a baseline that has access to NQ labeled data whereas our HLP is trained in an unsupervised way. For MS MARCO, HLP consistently outperforms other pre-training methods but falls behind BM25 under zero-shot setting. We conjecture the performance degradation on MS MARCO is attributed to two factors: 1) the Q-P lexical overlap of MS MARCO (65.7%) is higher than that in BioASQ (48.7%) as well as other datasets; 2) the information-seeking target of the MS MARCO query is the entire passage rather than a short answer span, which is biased towards our proposed answer containing. we also observe that pre-training exclusively with DL pairs achieves better results in MS MARCO, indicating the generality of relevance induced by DL topology. We evaluate our methods on HotpotQA in a single-hop manner. Specifically, for each query, we randomly selects one golden passage from the two as a positive passage and one additional passage with high TF-IDF scores as a negative passage. Our models are further fine-tuned on the HotpotQA training set and evaluated on the bridge and the comparison type questions from the development set, respectively. The results of our study are shown in Table 5 which reveals that HLP consistently outperforms others methods, with up to a 11-point improvement on top-5 retrieval accuracy of bridge questions. Furthermore, WLP yields a 4-point advantages in average over ICT and BFS on bridge questions, showing that document-wise relevance contributes to better associative abilities. We include a case study in Appendix F. Bridge Comparison top5 top20 top100 top5 top20 top100 No Pre-train 25.0 40.5 58.0 83.0 94.2 97.4 ICT 28.1 43.8 61.8 84.8 94.4 98.3 WLP 32.1 49.1 66.0 89.7 97.3 99.2 BFS 29.0 44.7 62.1 87.4 95.8 98.7 HLP 36.9 53.0 68.5 94.4 98.5 99.5 Table 5: Retrieval accuracy on questions from HotpotQA dev set, measured as the percentage of top-k retrieved passages which include both golds. To better understand how different key factors affect the results, we conduct ablation experiments with results shown in Table 3. Hyperlink-based Topologies Our proposed dual-link (DL) and co-mention (CM) Q-P pairs, provide evidence induced by different hyperlink-based topologies. To examine their respective effectiveness, we pre-train retrievers on Q-P pairs derived from each topology and their combinations. We present zero-shot retrieval results in Table 3, which show that retrievers pre-trained on DL pairs has a distinct advantage over that on CM pairs, while combining both gives extra improvement. is essential for learning a high-quality encoder. Besides in-batch negative, our reported HLP employs one additional negative for each query. We further explore the impact of the additional negatives during pre-training. In our ablation study, pre-training with additional negatives improves the results significantly, which may be attributed to using more in-batch pairs for text matching. More details on implementation and negative sampling strategies can be found in Appendix B. 6.2 Analysis on Q-P Overlap We carry out extensive analysis on the Q-P lexical overlap in the task of retrieval. Specifically, we tokenize q , p using the BERT tokenizer and measure the Q-P overlap as the proportion of the question tokens that appear in the corresponding passage. Based on the degree of Q-P overlap, we divided the NQ dev set into five categories for further analysis. Distribution of Q-P Overlap Figure 3 shows both the pre-training and the retrieved pairs of HLP have a more similar overlap distribution with the downstream NQ dataset than the other methods, which implies the consistency between the relevance provided by HLP and that in real information-seeking scenario. Retrieval Performance vs. Q-P Overlap Figure 4 shows the top-20 retrieval accuracy on the samples with varying degrees of Q-P overlap. Both figures show that the retrievers are more likely to return answer-containing passages when there is higher Q-P overlap, suggesting that all these models can exploit lexical overlap for passage retrieval. Under the zero-shot setting, HLP outperforms all the methods except BM25 when r is larger than 0.8, which reflects the strong reasoning ability of HLP and the overlap-dependent nature of the term-based retrievers. After fine-tuning, models with additional pre-training perform better than the vanilla DPR while HLP outperforms all other methods in most of the cases. It is important to note that HLP is pre-trained on more high-overlap text pairs while it performs better than all the other methods when fewer overlaps are provided. We speculate that this is because the overlapping in HLP Q-P pairs mostly comes from the factual information, such as entity, which introduces fewer superficial cues, allowing for better adaptation to the downstream cases. We conduct human evaluation to investigate the proportion of Q-P pairs that convey the similar fact-level information. Specially, we randomly selected one hundred examples from our constructed Q-P pairs and asked annotators to identify whether the query and the corresponding passage convey similar facts. Each case is evaluated by three annotators and the result is determined by their votes. Our results are shown in Table 6, and we further present case studies in Appendix G. DL CM WLP Votes 61% 40% 15% Table 6: Human evaluation on pseudo Q-P pairs constructed by different methods. This paper proposes Hyperlink-induced Pretraining (HLP), a pre-training method for OpenQA passage retrieval by leveraging the online textual relevance induced by hyperlink-based topology. Our experiments show that HLP gains significant improvements across multiple QA datasets under different scenarios, consistently outperforming other pre-training methods. Our method provides insights into OpenQA passage retrieval by analyzing the underlying bi-text relevance. Future work involves addressing tasks like MS MARCO where the granularity of the information-seeking target is at the passage level. This work is partially supported by the Hong Kong RGC GRF Project 16202218, CRF Project C6030-18G, C1031-18G, C5026-18G, AOE Project AoE/E-603/18, China NSFC No. 61729201. We thank all the reviewers for their insightful comments.
abstain abstain objective objective objective result abstain abstain abstain abstain other abstain abstain abstain abstain abstain result objective method objective objective objective abstain abstain abstain abstain abstain abstain objective objective objective abstain other other other other method other other other other other other other other method abstain abstain method abstain abstain abstain method abstain method abstain method method abstain method abstain abstain method abstain abstain method abstain abstain method other method method method abstain method abstain abstain method other abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method method method abstain method method method abstain abstain method method method abstain abstain abstain abstain abstain result result result abstain abstain result abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain other abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain objective result method abstain other other