Language model (LM) pretraining captures various knowledge from text corpora, helping downstream NLP tasks. However, existing methods such as BERT model a single document, failing to capture document dependencies and knowledge that spans across documents. In this work, we propose LinkBERT , an effective LM pretraining method that incorporates document links, such as hyperlinks. Given a pretraining corpus, we view it as a graph of documents, and create LM inputs by placing linked documents in the same context. We then train the LM with two joint self-supervised tasks: masked language modeling and our newly proposed task, document relation prediction. We study LinkBERT in two domains: general domain (pretrained on Wikipedia with hyperlinks) and biomedical domain (pretrained on PubMed with citation links). LinkBERT outperforms BERT on various downstream tasks in both domains. It is especially effective for multi-hop reasoning and few-shot QA (+5% absolute improvement on HotpotQA and TriviaQA), and our biomedical LinkBERT attains new state-of-the-art on various BioNLP tasks (+7% on BioASQ and USMLE). We release the pretrained models, LinkBERT and BioLinkBERT , as well as code and data. 1 1 Introduction Pretrained language models (LMs), like BERT and GPTs (Devlin et al., 2019; Brown et al., 2020), have shown remarkable performance on many natural language processing (NLP) tasks, such as text classification and question answering (Raffel et al., 2020), becoming the foundation of modern NLP systems. The Jeerson Memorial, the Martin Luther King Jr. Memorial, the Franklin Delano Roosevelt Memorial, and the George Mason Memorial are situated adjacent to the Tidal Basin. [The National Cherry Blossom Festival] ... It is a spring celebration commemorating the March 27, 1912, gift of Japanese cherry trees from Mayor of Tokyo City Yukio Ozaki to the city of Washington, D.C. Mayor Ozaki gifted the trees to enhance the growing friendship between the United States and Japan. ... Of the initial gift of 12 varieties of 3,020 trees, the Yoshino Cherry (70% of total) and Kwanzan Cherry (13% of total) now dominate. ... By performing self-supervised learning on text, such as masked language modeling (Devlin et al., 2019), LMs learn to encode various knowledge from text corpora and produce informative language representations for downstream tasks (Bosselut et al., 2019; Bommasani et al., 2021). Equal senior authorship. However, existing LM pretraining methods typically consider a single document in each input context (Liu et al., 2019; Joshi et al., 2020), and do not model links between documents. This can pose limitations because documents often have rich dependencies with each other (e.g. hyperlinks, references), and knowledge can span across documents. As a simple example, in Figure 1, the Wikipedia article Tidal Basin, Washington D.C. (left) describes that the basin hosts National Cherry Blossom Festival, and the hyperlinked article (right) reveals the background that the festival celebrates Japanese cherry trees. Taken together, the hyperlink offers new, multi-hop knowledge Tidal Basin has Japanese cherry trees, which is not available in the single article Tidal Basin alone. Acquiring such multi-hop knowledge in pretraining could be useful for various applications including question answering. In fact, document links like hyperlinks and references are ubiquitous (e.g. web, books, scientific literature), and guide how we humans acquire knowledge and 8003 (cid:13)(cid:28)(cid:41)(cid:34)(cid:48)(cid:28)(cid:34)(cid:32)(cid:1)(cid:14)(cid:42)(cid:31)(cid:32)(cid:39) (cid:548)(cid:29)(cid:99)(cid:166)(cid:549)(cid:573) (cid:179)(cid:308)(cid:274)(cid:573) (cid:179)(cid:313)(cid:268)(cid:232)(cid:335)(cid:573) (cid:28)(cid:232)(cid:403)(cid:313)(cid:346)(cid:573) (cid:548)(cid:166)(cid:42)(cid:155)(cid:549)(cid:573) (cid:548)(cid:109)(cid:1)(cid:166)(cid:96)(cid:549)(cid:573) (cid:548)(cid:109)(cid:1)(cid:166)(cid:96)(cid:549)(cid:573) (cid:415)(cid:393)(cid:274)(cid:274)(cid:403) (cid:1057)(cid:1057)(cid:1057)(cid:1) (cid:1057)(cid:1057)(cid:1057)(cid:1) (cid:548)(cid:166)(cid:42)(cid:155)(cid:549)(cid:573) (cid:20)(cid:32)(cid:34)(cid:40)(cid:32)(cid:41)(cid:47)(cid:1)(cid:2) (cid:20)(cid:32)(cid:34)(cid:40)(cid:32)(cid:41)(cid:47)(cid:1)(cid:3) (cid:94)(cid:232)(cid:390)(cid:232)(cid:346)(cid:274)(cid:403)(cid:274)(cid:573)(cid:261)(cid:308)(cid:274)(cid:393)(cid:393)(cid:453) (cid:14)(cid:28)(cid:46)(cid:38)(cid:32)(cid:31)(cid:1)(cid:39)(cid:28)(cid:41)(cid:34)(cid:48)(cid:28)(cid:34)(cid:32)(cid:1)(cid:40)(cid:42)(cid:31)(cid:32)(cid:39)(cid:36)(cid:41)(cid:34)(cid:1)(cid:1090)(cid:14)(cid:13)(cid:14)(cid:1091) (cid:5)(cid:42)(cid:30)(cid:48)(cid:40)(cid:32)(cid:41)(cid:47)(cid:1)(cid:45)(cid:32)(cid:39)(cid:28)(cid:47)(cid:36)(cid:42)(cid:41)(cid:1)(cid:43)(cid:45)(cid:32)(cid:31)(cid:36)(cid:30)(cid:47)(cid:36)(cid:42)(cid:41)(cid:1)(cid:1090)(cid:5)(cid:19)(cid:17)(cid:1091) (cid:404) (cid:99)(cid:313)(cid:346)(cid:332)(cid:274)(cid:268)(cid:573) (cid:404) (cid:158)(cid:232)(cid:346)(cid:268)(cid:355)(cid:344)(cid:573) (cid:404) (cid:29)(cid:355)(cid:346)(cid:415)(cid:313)(cid:301)(cid:423)(cid:355)(cid:423)(cid:403)(cid:573) (cid:4)(cid:42)(cid:45)(cid:43)(cid:48)(cid:46)(cid:1)(cid:42)(cid:33)(cid:1)(cid:39)(cid:36)(cid:41)(cid:38)(cid:32)(cid:31)(cid:1)(cid:31)(cid:42)(cid:30)(cid:48)(cid:40)(cid:32)(cid:41)(cid:47)(cid:46) (cid:36)(cid:355)(cid:261)(cid:573)(cid:475) (cid:36)(cid:355)(cid:261)(cid:573)(cid:478) (cid:36)(cid:355)(cid:261)(cid:573)(cid:477) (cid:36)(cid:355)(cid:261)(cid:573)(cid:479) (cid:36)(cid:355)(cid:261)(cid:573)(cid:474) (cid:36)(cid:355)(cid:261)(cid:573)(cid:476) (cid:20)(cid:32)(cid:34)(cid:40)(cid:32)(cid:41)(cid:47)(cid:1)(cid:2) (cid:20)(cid:32)(cid:34)(cid:40)(cid:32)(cid:41)(cid:47)(cid:1)(cid:3) (cid:36)(cid:355)(cid:261) (cid:573) (cid:474)(cid:573)(cid:403)(cid:274)(cid:301) (cid:573) (cid:390)(cid:605)(cid:474) (cid:29)(cid:355)(cid:346)(cid:415)(cid:313)(cid:301)(cid:423)(cid:355)(cid:423)(cid:403) (cid:158)(cid:232)(cid:346)(cid:268)(cid:355)(cid:344) (cid:99)(cid:313)(cid:346)(cid:332)(cid:274)(cid:268) (cid:355)(cid:393) (cid:355)(cid:393) (cid:36)(cid:355)(cid:261) (cid:573) (cid:474)(cid:573)(cid:403)(cid:274)(cid:301) (cid:573) (cid:390) (cid:36)(cid:355)(cid:261) (cid:573) (cid:478)(cid:573)(cid:403)(cid:274)(cid:301) (cid:573) (cid:392) (cid:36)(cid:355)(cid:261) (cid:573) (cid:474)(cid:573)(cid:403)(cid:274)(cid:301) (cid:573) (cid:390) (cid:36)(cid:355)(cid:261) (cid:573) (cid:476)(cid:573)(cid:403)(cid:274)(cid:301) (cid:573) (cid:392) (cid:36)(cid:355)(cid:261) (cid:573) (cid:474)(cid:573)(cid:403)(cid:274)(cid:301) (cid:573) (cid:390) (cid:4)(cid:45)(cid:32)(cid:28)(cid:47)(cid:32)(cid:1)(cid:13)(cid:14)(cid:1)(cid:36)(cid:41)(cid:43)(cid:48)(cid:47)(cid:46) (cid:17)(cid:45)(cid:32)(cid:47)(cid:45)(cid:28)(cid:36)(cid:41)(cid:1)(cid:47)(cid:35)(cid:32)(cid:1)(cid:13)(cid:14) (cid:35)(cid:52)(cid:43)(cid:32)(cid:45)(cid:39)(cid:36)(cid:41)(cid:38)(cid:1058)(cid:1)(cid:45)(cid:32)(cid:33)(cid:32)(cid:45)(cid:32)(cid:41)(cid:30)(cid:32)(cid:1058)(cid:1)(cid:32)(cid:47)(cid:30)(cid:1057) Figure 2: Overview of our approach, LinkBERT . In this work, we propose LinkBERT , an effective language model pretraining method that incorporates document link knowledge. Given a text corpus, we obtain links between documents such as hyperlinks, and create LM inputs by placing linked documents in the same context window, besides the existing option of placing a single document or random documents as in BERT. Specifically, as in Figure 2, after sampling an anchor text segment, we place either (1) the contiguous segment from the same document, (2) a random document, or (3) a document linked from anchor segment, as the next segment in the input. We then train the LM with two joint objectives: We use masked language modeling (MLM) to encourage learning multi-hop knowledge of concepts brought into the same context by document links (e.g. Tidal Basin and Japanese cherry in Figure 1). Simultaneously, we propose a Document Relation Prediction (DRP) objective, which classifies the relation of the second segment to the first segment ( contiguous , random , or linked ). DRP encourages learning the relevance and bridging concepts (e.g. National Cherry Blossom Festival) between documents, beyond the ability learned in the vanilla next sentence prediction objective in BERT. Viewing the pretraining corpus as a graph of documents, LinkBERT is also motivated as self-supervised learning on the graph, where DRP and MLM correspond to link prediction and node feature prediction in graph machine learning (Yang et al., 2015; Hu et al., 2020). Our modeling approach thus provides a natural fusion of language-based and graph-based self-supervised learning. We train LinkBERT on two domains: the general domain, using Wikipedia articles with hyperlinks (4), and the biomedical domain, using PubMed articles with citation links (6). We then evaluate the pretrained models on a wide range of downstream tasks including question answering, in both domains. LinkBERT consistently improves on baseline LMs across domains and tasks. For the general domain, LinkBERT outperforms BERT on MRQA benchmark (+4% absolute in F1-score) as well as GLUE benchmark. For the biomedical domain, LinkBERT exceeds PubmedBERT (Gu et al., 2020) and attains new state-of-the-art on BLURB biomedical NLP benchmark (+3% absolute in BLURB score) and MedQA-USMLE reasoning task (+7% absolute in accuracy). Overall, LinkBERT attains notably large gains for multi-hop reasoning, multi-document understanding, and few-shot question answering, suggesting that LinkBERT internalizes significantly more knowledge than existing LMs by pretraining with document link information. Retrieval-augmented LMs. Several works (Lewis et al., 2020b; Karpukhin et al., 2020; Oguz et al., 2020; Xie et al., 2022) introduce a retrieval module for LMs, where given an anchor text (e.g. question), retrieved text is added to the same LM context to improve model inference (e.g. answer prediction). These works show the promise of placing related documents in the same LM context at inference time, but they do not study pretraining. Guu et al. (2020) pretrain an LM with a retriever that learns to retrieve text for answering masked tokens in the anchor text. In contrast, our focus is not on retrieval, but on pretraining a general-purpose LM that internalizes knowledge that spans across documents, which is orthogonal to the above works (e.g., our pretrained LM could be used to initialize the LM component of these works). Additionally, we focus on incorporating document links such as hyperlinks, which can offer salient knowledge that common lexical retrieval methods may not provide (Asai et al., 2020). to pretrain LMs. Caciularu et al. (2021) place documents (news articles) about the same topic into the same LM context, and Levine et al. (2021) place sentences of high lexical similarity into the same context. Our work provides a general method to incorporate document links into LM pretraining, where lexical or topical similarity can be one instance of document links, besides hyperlinks. We focus on hyperlinks in this work, because we find they can bring in salient knowledge that may not be obvious via lexical similarity, and yield a more performant LM (5.5). Additionally, we propose the DRP objective, which improves modeling multiple documents and relations between them in LMs (5.5). Hyperlinks and citation links for NLP. Hyperlinks are often used to learn better retrieval models. Chang et al. (2020); Asai et al. (2020); Seonwoo et al. (2021) use Wikipedia hyperlinks to train retrievers for open-domain question answering. Ma et al. (2021) study various hyperlink-aware pretraining tasks for retrieval. While these works use hyperlinks to learn retrievers, we focus on using hyperlinks to create better context for learning general-purpose LMs. Separately, Calixto et al. (2021) use Wikipedia hyperlinks to learn multilingual LMs. Citation links are often used to improve summarization and recommendation of academic papers (Qazvinian and Radev, 2008; Yasunaga et al., 2019; Bhagavatula et al., 2018; Khadka et al., 2020; Cohan et al., 2020). Here we leverage citation networks to improve pretraining general-purpose LMs. Graph-augmented LMs. Several works augment LMs with graphs, typically, knowledge graphs (KGs) where the nodes capture entities and edges their relations. Zhang et al. (2019); He et al. (2020); Wang et al. (2021b) combine LM training with KG embeddings. Sun et al. (2020); Yasunaga et al. (2021); Zhang et al. (2022) combine LMs and graph neural networks (GNNs) to jointly train on text and KGs. Different from KGs, we use document graphs to learn knowledge that spans across documents. A language model (LM) can be pretrained from a corpus of documents, X = { X ( i ) } . An LM is a composition of two functions, f head ( f enc ( X )) , where the encoder f enc takes in a sequence of tokens X = ( x 1 ,x 2 ,...,x n ) and produces a contextualized vector representation for each token, ( h 1 , h 2 ,..., h n ) . The head f head typically uses these representations to perform self-supervised tasks in the pretraining step, and perform particular downstream tasks in the fine-tuning step. We build on BERT (Devlin et al., 2019), which pretrains an LM with the following two self-supervised tasks. Masked language modeling (MLM). Given a sequence of tokens X , a subset of tokens Y  X is masked, and the task is to predict the original tokens from the modified input. Y accounts for 15% of the tokens in X ; of those, 80% are replaced with [MASK] , 10% with a random token, and 10% are kept unchanged. Next sentence prediction (NSP). The NSP task takes two text segments 2 ( XA ,X B ) as input, and predicts whether XB is the direct continuation of XA . Specifically, BERT first samples XA from the corpus, and then either (1) takes the next segment XB from the same document, or (2) samples XB from a random document in the corpus. The two segments are joined via special tokens to form an input instance, [CLS] XA [SEP] XB [SEP] , where the prediction target of [CLS] is whether XB indeed follows XA ( contiguous or random ). In this work, we will further incorporate document link information into LM pretraining. Our approach (4) will build on MLM and NSP. We present LinkBERT, a self-supervised pretraining approach that aims to internalize more knowledge into LMs using document link information. Specifically, as shown in Figure 2, instead of viewing the pretraining corpus as a set of documents X = { X ( i ) } , we view it as a graph of documents, G = ( X , E ) , where E = { ( X ( i ) , X ( j ) ) } denotes links between documents (4.1). The links can be existing hyperlinks, or could be built by other methods that capture document relevance. We then consider pretraining tasks for learning from document links (4.2): We create LM inputs by placing linked documents in the same context window, besides the existing options of a single document or random documents. We use the MLM task to learn concepts brought together in the context by document links, and we also introduce the Document Relation Prediction (DRP) task to learn relations between documents. Finally, we discuss strategies for obtaining informative pairs of linked documents to feed into LM pretraining (4.3). Given a pretraining corpus, we link related documents so that the links can bring together knowledge that is not available in single documents. We focus 2 A segment is typically a sentence or a paragraph. on hyperlinks, e.g., hyperlinks of Wikipedia articles (5) and citation links of academic articles (6). Hyperlinks have a number of advantages. They provide background knowledge about concepts that the document writers deemed usefulthe links are likely to have high precision of relevance, and can also bring in relevant documents that may not be obvious via lexical similarity alone (e.g., in Figure 1, while the hyperlinked article mentions Japanese and Yoshino cherry trees, these words do not appear in the anchor article). Hyperlinks are also ubiquitous on the web and easily gathered at scale (Aghajanyan et al., 2021). To construct the document graph, we simply make a directed edge ( X ( i ) ,X ( j ) ) if there is a hyperlink from document X ( i ) to document X ( j ) . For comparison, we also experiment with a document graph built by lexical similarity between documents. For each document X ( i ) , we use the common TF-IDF cosine similarity metric (Chen et al., 2017; Yasunaga et al., 2017) to obtain topk documents X ( j ) 's and make edges ( X ( i ) ,X ( j ) ) . We use k =5 . Creating input instances. Several works (Gao et al., 2021; Levine et al., 2021) find that LMs can learn stronger dependencies between words that were shown together in the same context during training, than words that were not. To effectively learn knowledge that spans across documents, we create LM inputs by placing linked documents in the same context window, besides the existing option of a single document or random documents. Specifically, we first sample an anchor text segment from the corpus (Segment A; XA  X ( i ) ). For the next segment (Segment B; XB ), we either (1) use the contiguous segment from the same document ( XB  X ( i ) ), (2) sample a segment from a random document ( XB  X ( j ) where j  = i ), or (3) sample a segment from one of the documents linked from Segment A ( XB  X ( j ) where ( X ( i ) ,X ( j ) )  E ). We then join the two segments via special tokens to form an input instance: [CLS] XA [SEP] XB [SEP] . Training objectives. To train the LM, we use two objectives. We apply the MLM objective to encourage the LM to learn multi-hop knowledge of concepts brought together in the same context by document links. We also propose a Document Relation Prediction (DPR) objective, which classifies the relation r of segment XB to segment XA ( r { contiguous , random , linked } ). By distinguishing linked from contiguous and random , DRP encourages the LM to learn the relevance and existence of bridging concepts between documents, besides the capability learned in the vanilla NSP objective. To predict r , we use the representation of [CLS] token, as in NSP. Taken together, we optimize: L = LMLM + LDRP (1) =  (cid:88) i log p ( x i | h i )  log p ( r | h [CLS] ) (2) where x i is each token of the input instance, [CLS] XA [SEP] XB [SEP] , and h i is its representation. Graph machine learning perspective. Our two pretraining tasks, MLM and DRP, are also motivated as graph self-supervised learning on the document graph. In graph self-supervised learning, two types of tasks, node feature prediction and link prediction, are commonly used to learn the content and structure of a graph. In node feature prediction (Hu et al., 2020), some features of a node are masked, and the task is to predict them using neighbor nodes. This corresponds to our MLM task, where masked tokens in Segment A can be predicted using Segment B (a linked document on the graph), and vice versa. In link prediction (Bordes et al., 2013; Wang et al., 2021a), the task is to predict the existence or type of an edge between two nodes. This corresponds to our DRP task, where we predict if the given pair of text segments are linked (edge), contiguous (self-loop edge), or random (no edge). Our approach can be viewed as a natural fusion of language-based (e.g. BERT) and graph-based self-supervised learning. As described in 4.1, 4.2, our method builds links between documents, and for each anchor segment, samples a linked document to put together in the LM input. Here we discuss three key axes to consider to obtain useful linked documents in this process. Relevance. Semantic relevance is a requisite when building links between documents. If links were randomly built without relevance, LinkBERT would be same as BERT, with simply two options of LM inputs ( contiguous or random ). Relevance can be achieved by using hyperlinks or lexical similarity metrics, and both methods yield substantially better performance than using random links (5.5). Salience. Besides relevance, another factor to consider ( salience ) is whether the linked document can offer new, useful knowledge that may not be obvious to the current LM. Hyperlinks are potentially more advantageous than lexical similarity links in this regard: LMs are shown to be good at recognizing lexical similarity (Zhang et al., 2020), and hyperlinks can bring in useful background knowledge that 8006 may not be obvious via lexical similarity alone (Asai et al., 2020). Indeed, we empirically find that using hyperlinks yields a more performant LM (5.5). Diversity. In the document graph, some documents may have a very high in-degree (e.g., many incoming hyperlinks, like the United States page of Wikipedia), and others a low in-degree. If we uniformly sample from the linked documents for each anchor segment, we may include documents of high in-degree too often in the overall training data, losing diversity. To adjust so that all documents appear with a similar frequency in training, we sample a linked document with probability inversely proportional to its in-degree, as done in graph data mining literature (Henzinger et al., 2000). We find that this technique yields a better LM performance (5.5). We experiment with our proposed approach in the general domain first, where we pretrain LinkBERT on Wikipedia articles with hyperlinks (5.1) and evaluate on a suite of downstream tasks (5.2). We compare with BERT (Devlin et al., 2019) as our baseline. We experiment in the biomedical domain in 6. Data. We use the same pretraining corpus used by BERT: Wikipedia and BookCorpus (Zhu et al., 2015). For Wikipedia, we use the WikiExtractor 3 to extract hyperlinks between Wiki articles. We then create training instances by sampling contiguous , random , or linked segments as described in 4, with the three options appearing uniformly (33%, 33%, 33%). For BookCorpus, we create training instance by sampling contiguous or random segments (50%, 50%) as in BERT. We then combine the training instances from Wikipedia and BookCorpus to train LinkBERT. In summary, our pretraining data is the same as BERT, except that we have hyperlinks between Wikipedia articles. Implementation. We pretrain LinkBERT of three sizes, -tiny, -base and -large, following the configurations of BERT tiny (4.4M parameters), BERT base (110M params), and BERT large (340M params) (Devlin et al., 2019; Turc et al., 2019). We use -tiny mainly for ablation studies. For -tiny, we pretrain from scratch with random weight initialization. We use the AdamW (Loshchilov and Hutter, 2019) optimizer with (  1 , 2 ) = (0 . 9 , 0 . 98) , warm up the learning rate for the first 5,000 steps and then linearly decay it. We train for 10,000 steps with a peak learning rate 5e-3, weight decay 0.01, and batch size of 2,048 sequences with 512 tokens. Training takes 1 day on two GeForce RTX 2080 Ti GPUs with fp16. For -base, we initialize LinkBERT with the BERT base checkpoint released by Devlin et al. (2019) and continue pretraining. We use a peak learning rate 3e-4 and train for 40,000 steps. Other training hyperparameters are the same as -tiny. Training takes 4 days on four A100 GPUs with fp16. For -large, we follow the same procedure as -base, except that we use a peak learning rate of 2e-4. Training takes 7 days on eight A100 GPUs with fp16. Baselines. We compare LinkBERT with BERT. Specifically, for the -tiny scale, we compare with BERT tiny , which we pretrain from scratch with the same hyperparameters as LinkBERT tiny . The only difference is that LinkBERT uses document links to create LM inputs, while BERT does not. For -base scale, we compare with BERT base , for which we take the BERT base release by Devlin et al. (2019) and continue pretraining it with the vanilla BERT objectives on the same corpus for the same number of steps as LinkBERT base . Extractive question answering (QA). Given a document (or set of documents) and a question as input, the task is to identify an answer span from the document. We evaluate on six popular datasets from the MRQA shared task (Fisch et al., 2019): HotpotQA (Yang et al., 2018), TriviaQA (Joshi et al., 2017), NaturalQ (Kwiatkowski et al., 2019), SearchQA (Dunn et al., 2017), NewsQA (Trischler et al., 2017), and SQuAD (Rajpurkar et al., 2016). As the MRQA shared task does not have a public test set, we split the dev set in half to make new dev and test sets. We follow the fine-tuning method BERT (Devlin et al., 2019) uses for extractive QA. More details are provided in Appendix B. GLUE. The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018) is a popular suite of sentence-level classification tasks. Following BERT, we evaluate on CoLA (Warstadt et al., 2019), SST-2 (Socher et al., 2013), MRPC (Dolan and Brockett, 2005), QQP , STS-B (Cer et al., 2017), MNLI (Williams et al., 2017), QNLI (Rajpurkar et al., 2016), and RTE (Dagan et al., 2005; Haim et al., 2006; Giampiccolo 8007 HotpotQA TriviaQA SearchQA NaturalQ NewsQA SQuAD Avg. attains comparable or moderately improved performance. et al., 2007), and report the average score. More fine-tuning details are provided in Appendix B. 5.3 Results Table 1 shows the performance (F1 score) on MRQA datasets. LinkBERT substantially outperforms BERT on all datasets. On average, the gain is +4.1% absolute for the BERT tiny scale, +2.6% for the BERT base scale, and +2.5% for the BERT large scale. Table 2 shows the results on GLUE, where LinkBERT performs moderately better than BERT. These results suggest that LinkBERT is especially effective at learning knowledge useful for QA tasks (e.g. world knowledge), while keeping performance on sentence-level language understanding too. We further study when LinkBERT is especially useful in downstream tasks. Improved multi-hop reasoning. In Table 1, we find that LinkBERT obtains notably large gains on QA datasets that require reasoning with multiple documents, such as HotpotQA (+5% over BERT tiny ), TriviaQA (+6%) and SearchQA (+8%), as opposed to SQuAD (+1.4%) which just has a single document per question. To further gain qualitative insights, we studied in what QA examples LinkBERT succeeds but BERT fails. Figure 3 shows a representative example from HotpotQA. Answering the question needs 2-hop reasoning: identify Roden Brothers were taken over by Birks Group from the first document, and then Birks Group is headquartered in Montreal from the second document. While BERT tends to simply predict an entity near the question entity (Toronto in the first document, which is just 1-hop), LinkBERT correctly predicts the answer in the second document (Montreal). Our intuition is that because LinkBERT is pretrained with pairs of linked documents rather than purely single documents, it better learns how to flow information (e.g., do attention) across tokens when multiple related documents are given in the context. In summary, these results suggest that pretraining with linked documents helps for multi-hop reasoning on downstream tasks. relations. While the MRQA datasets typically use ground-truth documents as context for answering questions, in open-domain QA, QA systems need to use documents obtained by a retriever, which may include noisy documents besides gold ones (Chen et al., 2017; Dunn et al., 2017). In such cases, QA systems need to understand the document relations to perform well (Yang et al., 2018). To simulate this setting, we modify the SQuAD dataset such that we prepend or append 12 distracting documents to the original document given to each question. Table 3 shows the result. While BERT incurs a large performance drop (-2.8%), LinkBERT is robust to distracting documents (-0.5%). This result suggests that pretraining with document links improves the ability to understand document relations and 8008 Three days after undergoing a laparoscopic Whipple's procedure, a 43-year-old woman has swelling of her right leg . LinkBERT prediction: Montreal (  ) BERT prediction: Toronto (  ) HotpotQA example Question : Roden Brothers were taken over in 1953 by a group headquartered in which Canadian city? LinkBERT prediction: Montreal (  ) BERT prediction: Toronto (  ) HotpotQA example LinkBERT predicts: Montreal (  ) BERT predicts: Toronto (  ) Figure 3: Case study of multi-hop reasoning on HotpotQA. Answering the question needs to identify Roden Brothers were taken over by Birks Group from the first document, and then Birks Group is headquartered in Montreal from the second document. While BERT tends to simply predict an entity near the question entity (Toronto in the first document), LinkBERT correctly predicts the answer in the second document (Montreal). Doc A : Roden Brothers was founded June 1, 1891 in Toronto, Ontario, Canada by Thomas and Frank Roden. In the 1910s the firm became known as Roden Bros. Ltd. and were later taken over by Henry Birks and Sons in 1953. ... In 1974 Roden Bros. Ltd. published the book, "Rich Cut Glass" with Clock House Publications in Peterborough, Ontario, which was a reprint of the 1917 edition published by Roden Bros., Toronto. Doc B : Birks Group (formerly Birks & Mayors) is a designer, manufacturer and retailer of jewellery, timepieces, silverware and gifts, with stores and manufacturing facilities located in Canada and the United States. As of June 30, 2015, it operates stores under three different retail banners: ... The company is headquartered in Montreal , Quebec, with American corporate offices located in Tamarac, Florida. Table 5 shows the ablation result on MRQA datasets. First, if we ignore relevance and use random document links instead of hyperlinks, we get the same performance as BERT (-4.1% on average; random in Table 5). Second, using lexical similarity links instead of hyperlinks leads to 1.8% performance drop (TF-IDF). Our intuition is that hyperlinks can provide more salient knowledge that may not be obvious via lexical similarity alone. Nevertheless, using lexical similarity links is substantially better than BERT (+2.3%), confirming the efficacy of placing relevant documents together in the input for LM pretraining. Finally, removing the diversity adjustment in document sampling leads to 1% performance drop (No diversity). In summary, our insight is that to create informative inputs for LM pretraining, the linked documents must be semantically relevant and ideally be salient and diverse. relevance. In particular, our intuition is that the DRP objective helps the LM to better recognize document relations like (anchor document, linked document) in pretraining, which helps to recognize relations like (question, right document) in downstream QA tasks. We indeed find that ablating the DRP objective from LinkBERT hurts performance (5.5). The strength of understanding document relations also suggests the promise of applying LinkBERT to various retrieval-augmented methods and tasks (e.g. Lewis et al. 2020b), either as the main LM or the dense retriever component. Improved few-shot QA performance. We also find that LinkBERT is notably good at few-shot learning. Concretely, for each MRQA dataset, we fine-tune with only 10% of the available training data, and report the performance in Table 4. In this few-shot regime, LinkBERT attains more significant gains over BERT, compared to the full-resource regime in Table 1 (on NaturalQ, 5.4% vs 1.8% absolute in F1, or 15% vs 7% in relative error reduction). This result suggests that LinkBERT internalizes more knowledge than BERT during pretraining, which supports our core idea that document links can bring in new, useful knowledge for LMs. What linked documents to feed into LMs? We study the strategies discussed in 4.3 for obtaining linked documents: relevance, salience, and diversity. Effect of the DRP objective. Table 6 shows the ablation result on the DRP objective (4.2). Removing DRP in pretraining hurts downstream QA performance. The drop is large on tasks with multiple documents (HotpotQA, TriviaQA, and SQuAD with distracting documents). This suggests that DRP facilitates LMs to learn document relations. Pretraining LMs on biomedical text is shown to boost performance on biomedical NLP tasks (Beltagy et al., 2019; Lee et al., 2020; Lewis et al., 2020a; Gu et al., 2020). Biomedical LMs are typically trained on PubMed, which contains abstracts and citations of biomedical papers. While prior works only use their raw text for pretraining, academic papers have rich dependencies with each other via citations (references). We hypothesize that incorporating citation links can help LMs learn dependencies between papers and knowledge that spans across them. With this motivation, we pretrain LinkBERT on PubMed with citation links (6.1), which we term BioLinkBERT , and evaluate on biomedical downstream tasks (6.2). As our baseline, we follow and compare with the state-of-the-art biomedical LM, PubmedBERT (Gu et al., 2020), which has the same architecture as BERT and is trained on PubMed. Data. We use the same pretraining corpus used by PubmedBERT: PubMed abstracts (21GB). 4 We 4 https://pubmed.ncbi.nlm.nih.gov . We use papers published before Feb. 2020 as in PubmedBERT. use the Pubmed Parser 5 to extract citation links between articles. We then create training instances by sampling contiguous , random , or linked segments as described in 4, with the three options appearing uniformly (33%, 33%, 33%). In summary, our pretraining data is the same as PubmedBERT, except that we have citation links between PubMed articles. Implementation. We pretrain BioLinkBERT of -base size (110M params) from scratch, following the same hyperparamters as the PubmedBERT base (Gu et al., 2020). Specifically, we use a peak learning rate 6e-4, batch size 8,192, and train for 62,500 steps. We warm up the learning rate in the first 10% of steps and then linearly decay it. Training takes 7 days on eight A100 GPUs with fp16. Additionally, while the original PubmedBERT release did not include the -large size, we pretrain BioLinkBERT of the -large size (340M params) from scratch, following the same procedure as -base, except that we use a peak learning rate of 4e-4 and warm up steps of 20%. Training takes 21 days on eight A100 GPUs with fp16. Baselines. We compare BioLinkBERT with PubmedBERT released by Gu et al. (2020). For downstream tasks, we evaluate on the BLURB benchmark (Gu et al., 2020), a diverse set of biomedical NLP datasets, and MedQA-USMLE (Jin et al., 2021), a challenging biomedical QA dataset. BLURB consists of five named entity recognition tasks, a PICO (population, intervention, comparison, and outcome) extraction task, three relation extraction tasks, a sentence similarity task, a document classification task, and two question answering tasks, as summarized in Table 7. We follow the same fine-tuning method and evaluation metric used by PubmedBERT (Gu et al., 2020). MedQA-USMLE is a 4-way multi-choice QA task that tests biomedical and clinical knowledge. The questions are from practice tests for the US Medical License Exams (USMLE). The questions typically require multi-hop reasoning, e.g., given patient symptoms, infer the likely cause, and then answer the appropriate diagnosis procedure (Figure 4). We follow the fine-tuning method in Jin et al. (2021). More details are provided in Appendix B. MMLU-professional medicine is a multi-choice QA task that tests biomedical knowledge and reasoning, and is part of the popular MMLU benchmark 5 https://github.com/titipata/pubmed_parser PubMed-BERT base BioLink-BERT base BioLink-BERT large Namedentityrecognition BC5-chem (Lietal.,2016) 93.33 93.75 94.04 BC5-disease (Lietal.,2016) 85.62 86.10 86.39 NCBI-disease (Doganetal.,2014) 87.82 88.18 88.76 BC2GM (Smithetal.,2008) 84.52 84.90 85.18 JNLPBA (Kimetal.,2004) 80.06 79.03 80.06 PICOextraction EBM PICO (Nyeetal.,2018) 73.38 73.97 74.19 Relationextraction ChemProt (Krallingeretal.,2017) 77.24 77.57 79.98 DDI (Herrero-Zazoetal.,2013) 82.36 82.72 83.35 GAD (Bravoetal.,2015) 82.34 84.39 84.90 Sentencesimilarity BIOSSES (Sogancogluetal.,2017) 92.30 93.25 93.63 Documentclassification HoC (Bakeretal.,2016) 82.32 84.35 84.87 Questionanswering PubMedQA (Jinetal.,2019) 55.84 70.20 72.18 BioASQ (Nentidisetal.,2019) 87.56 91.43 94.82 BLURB score 81.10 83.39 84.30 Table 7: Performance on BLURB benchmark. (Hendrycks et al., 2021) that is used to evaluate massive language models. We take the BioLinkBERT fine-tuned on the above MedQA-USMLE task, and evaluate on this task without further adaptation. BLURB. Table 7 shows the results on BLURB. BioLinkBERT base outperforms PubmedBERT base on all task categories, attaining a performance boost of +2% absolute on average. Moreover, BioLinkBERT large provides a further boost of +1%. In total, BioLinkBERT outperforms the previous best by +3% absolute, establishing new state-of-the-art on the BLURB leaderboard. We see a trend that gains are notably large on document-level tasks such as question answering (+7% on BioASQ and 8010 Three days after undergoing a laparoscopic Whipple's procedure, a 43-year-old woman has swelling of her right leg . [Tidal Basin, Washington D.C.] The Tidal Basin is a man-made reservoir located between the LinkBERT prediction: Montreal (  ) BERT prediction: Toronto (  ) HotpotQA example LinkBERT predicts: Montreal (  ) BERT predicts: Toronto (  ) Figure 4: Case study of multi-hop reasoning on MedQA-USMLE. Answering the question (left) needs 2-hop reasoning (center): from the patient symptoms described in the question ( leg swelling , pancreatic cancer ), infer the cause ( deep vein thrombosis ), and then infer the appropriate diagnosis procedure ( compression ultrasonography ). While the existing PubmedBERT tends to simply predict a choice that contains a word appearing in the question (blood for choice D), BioLinkBERT correctly predicts the answer (B). Our intuition is that citation links bring relevant documents together in the same context in pretraining (right), which readily provides the multi-hop knowledge needed for the reasoning (center). Washington, D.C. It is part West Potomac Park, is near National Mall and is a focal of the National Cherry Blossom Festival held each spring. The Jeerson Memorial, the Martin Luther King Jr. Memorial, the Franklin Delano Roosevelt Memorial, and the George Mason Memorial are situated adjacent to the Tidal Basin. PubMedQA). This result is consistent with the general domain (5.3) and confirms that LinkBERT helps to learn document dependencies better. Doc A : Roden Brothers was founded June 1, 1891 in Toronto, Ontario, Canada by Thomas and Frank Roden. In the 1910s the firm became known as Roden Bros. Ltd. and were later taken over by Henry Birks and Sons in 1953. ... In 1974 Roden Bros. Ltd. published the book, "Rich Cut Glass" with Clock House Publications in Peterborough, Ontario, which was a reprint of the 1917 edition published by Roden Bros., Toronto. Doc B : Birks Group (formerly Birks & Mayors) is a designer, manufacturer and retailer of jewellery, timepieces, silverware and gifts, with stores and manufacturing facilities located in Canada and the United States. As of June 30, 2015, it operates stores under three dierent retail banners:  The company is headquartered in Montreal , Quebec, with American corporate oices located in Tamarac, Florida. LinkBERT prediction: Montreal (  ) BERT prediction: Toronto (  ) HotpotQA example Question : Roden Brothers were taken over in 1953 by a group headquartered in which Canadian city? MedQA-USMLE. Table 8 shows the results. BioLinkBERT base obtains a 2% accuracy boost over PubmedBERT base , and BioLinkBERT large provides an additional +5% boost. In total, BioLinkBERT outperforms the previous best by +7% absolute, attaining new state-of-the-art. To further gain qualitative insights, we studied in what QA examples BioLinkBERT succeeds but the baseline PubmedBERT fails. Figure 4 shows a representative example. Answering the question (left) needs 2-hop reasoning (center): from the patient symptoms described in the question ( leg swelling , pancreatic cancer ), infer the cause ( deep vein thrombosis ), and then infer the appropriate diagnosis procedure ( compression ultrasonography ). We find that while the existing PubmedBERT tends to simply predict a choice that contains a word appearing in the question (blood for choice D), BioLinkBERT correctly predicts the answer (B). Our intuition is that citation links bring relevant documents and concepts together in the same context in pretraining (right), 6 which readily provides the multi-hop knowledge needed for the reasoning (center). Combined with the analysis on HotpotQA (5.4), our results suggest that pretraining with document links consistently helps for multi-hop reasoning across domains (e.g., general documents with hyperlinks and biomedical articles with citation links). It is a spring celebration commemorating the March 27, 1912, gift of Japanese cherry trees from Mayor of Tokyo City to the city of Washington, D.C. ... Of the initial gift of 12 varieties of 3,020 trees, the Yoshino Cherry now dominates. ... ters, BioLinkBERT large achieves 50% accuracy on this QA task, significantly outperforming the largest general-domain LM or QA models such as GPT-3 175B params (39% accuracy) and UnifiedQA 11B params (43% accuracy). This result shows that with an effective pretraining approach, a small domain-specialized LM can outperform orders of magnitude larger language models on QA tasks. Canada by Thomas and Frank Roden. In the 1910s the firm became known as Roden Bros. Ltd. and were later taken over by Henry Birks and Sons in 1953. ... In 1974 Roden Bros. Ltd. published the book, "Rich Cut Glass" with Clock House Publications in Peterborough, Ontario, which was a reprint of the 1917 edition published by Roden Bros., Toronto. Doc B : Birks Group (formerly Birks & Mayors) is a designer, manufacturer and retailer of jewellery, timepieces, silverware and gifts, with stores and manufacturing facilities located in Canada and the United States. As of June 30, 2015, it operates stores under three different retail banners: ... The company is headquartered in Montreal , Quebec, with American corporate offices located in Tamarac, Florida. In both the general domain (pretrained on Wikipedia with hyperlinks) and biomedical domain (pretrained on PubMed with citation links), LinkBERT outperforms previous BERT models across a wide range of downstream tasks. The gains are notably large for multi-hop reasoning, multi-document understanding and few-shot question answering, suggesting that LinkBERT effectively internalizes salient knowledge through document links. Our results suggest that LinkBERT can be a strong pretrained LM to be applied to various knowledge-intensive tasks. MMLU-professional medicine. Table 9 shows the performance. Despite having just 340M parame-6 For instance, as in Figure 4 (right), Ansari et al. (2015) in PubMed mention that pancreatic cancer can induce deep vein thrombosis in leg , and it cites a paper in PubMed, Piovella et al. (2002), which mention that deep vein thrombosis is tested by compression ultrasonography . Placing these two documents in the same context yields the complete multi-hop knowledge needed to answer the question ( pancreatic cancer    deep vein thrombosis    compression ultrasonography ). Pretrained models, code and data are available at https://github.com/michiyasunaga/LinkBERT . Experiments are available at https://worksheets.codalab.org/worksheets/0x7a6ab9c8d06a41d191335b270da2902e . We thank Siddharth Karamcheti, members of the Stanford P-Lambda, SNAP and NLP groups, as well as our anonymous reviewers for valuable feedback. We gratefully acknowledge the support of NSFCAREER Award IIS-1552635; DARPA under Nos. 8011 HR00112190039 (TAMI), N660011924033 (MCS); Funai Foundation Fellowship; Microsoft Research PhD Fellowship; ARO under Nos. W911NF-16-1-0342 (MURI), W911NF-16-1-0171 (DURIP); NSF under Nos. OAC-1835598 (CINES), OAC-1934578 (HDR), CCF-1918940 (Expeditions), IIS-2030477 (RAPID), NIH under No. R56LM013365; Stanford Data Science Initiative, Wu Tsai Neurosciences Institute, Chan Zuckerberg Biohub, Amazon, JPMorgan Chase, Docomo, Hitachi, Intel, KDDI, Toshiba, NEC, Juniper, and UnitedHealth Group.
abstain abstain objective method objective method abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain method method method abstain abstain abstain abstain other other other other objective method other other method abstain objective other other other other objective other other objective other other other other method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain method method abstain method abstain abstain method abstain method method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method method method method method method abstain method method method method method abstain method method abstain abstain method abstain abstain method method abstain method abstain abstain other abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method other method abstain method method abstain method method method abstain method abstain abstain method method abstain method abstain abstain abstain method other abstain method abstain abstain abstain abstain abstain result abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other other other other other other other