Recently, the performance of Pre-trained Language Models (PLMs) has been significantly improved by injecting knowledge facts to enhance their abilities of language understanding. For medical domains, the background knowledge sources are especially useful, due to the massive medical terms and their complicated relations are difficult to understand in text. In this work, we introduce SMedBERT, a medical PLM trained on large-scale medical corpora, incorporating deep structured semantics knowledge from neighbours of linked-entity. In SMedBERT, the mention-neighbour hybrid attention is proposed to learn heterogeneous-entity information, which infuses the semantic representations of entity types into the homogeneous neighbouring entity structure. Apart from knowledge integration as external features, we propose to employ the neighbors of linked-entities in the knowledge graph as additional global contexts of text mentions, allowing them to communicate via shared neighbors, thus enrich their semantic representations. Experiments demonstrate that SMedBERT significantly outperforms strong baselines in various knowledge-intensive Chinese medical tasks. It also improves the performance of other tasks such as question answering, question matching and natural language inference. 1 1 Introduction Pre-trained Language Models (PLMs) learn effective context representations with self-supervised tasks, spotlighting in various NLP tasks (Wang et al., 2019a; Nan et al., 2020; Liu et al., 2020a). In addition, Knowledge-Enhanced PLMs (KEPLMs) (Zhang et al., 2019; Liu et al., 2020b; Wang et al., 2019b) further benefit language understanding by  Corresponding author. In the literatures, a majority of KEPLMs (Zhang et al., 2020a; Hayashi et al., 2020; Sun et al., 2020) inject information of entities corresponding to mention-spans from Knowledge Graphs (KGs) into contextual representations. However, those KEPLMs only utilize linked-entity in the KGs as auxiliary information, which pay little attention to the neighboring structured semantics information of the entity linked with text mentions. In the medical context, there exist complicated domain knowledge such as relations and medical facts among medical terms (Rotmensch et al., 2017; Li et al., 2020), which are difficult to model using previous approaches. To address this issue, we consider leveraging structured semantics knowledge in medical KGs from the two aspects. (1) Rich semantic information from neighboring structures of linked-entities, such as entity types and relations, are highly useful for medical text understanding. As in Figure 1,         (novel coronavirus) can be the cause of many diseases, such as     (pneumonia) and       (respiratory syndrome). 2 (2) Additionally, we leverage neighbors of linked-entity as global contexts to complement plain-text contexts used in (Mikolov et al., 2013a; Pennington et al., 2014). The structure knowledge contained in neighbouring entities can act as the knowledge bridge between mention-spans, facilitating the interaction of different mention representations. Hence, PLMs can learn better representations for rare medical terms. In this paper, we introduce SMedBERT, a KEPLM pre-trained over large-scale medical corpora and medical KGs. To the best of our knowledge, SMedBERT is the first PLM with structured semantics knowledge injected in the medical domain. Specifically, the contributions of SMedBERT mainly include two modules: Mention-neighbor Hybrid Attention: We fuse the embeddings of the node and type of linked-entity neighbors into contextual target mention representations. The type-level and node-level attentions help to learn the importance of entity types and the neighbors of linked-entity, respectively, in order to reduce the knowledge noise injected into the model. The type-level attention transforms the homogeneous node-level attention into a heterogeneous learning process of neighboring entities. Mention-neighbor Context Modeling: We propose two novel self-supervised learning tasks for promoting interaction between mention-span and corresponding global context, namely masked neighbor modeling and masked mention modeling. The former enriches the representations of con-text neighboring entities based on the well trained target word mention-span, while the latter focuses on gathering those information back from neighboring entities to the masked target like low-frequency mention-span which is poorly represented (Turian et al., 2010). In the experiments, we compare SMedBERT against various strong baselines, including mainstream KEPLMs pre-trained over our medical resources. The underlying medical NLP tasks include: named entity recognition, relation extraction, question answering, question matching and natural language inference. The results show that SMedBERT consistently outperforms all the baselines on these tasks. 2 Although we focus on Chinese medical PLMs here. The proposed method can be easily adapted to other languages, which is beyond the scope of this work. PLMs in the Open Domain. PLMs have gained much attention recently, proving successful for boosting the performance of various NLP tasks (Qiu et al., 2020). Early works on PLMs focus on feature-based approaches to transform words into distributed representations (Collobert and Weston, 2008; Mikolov et al., 2013b; Pennington et al., 2014; Peters et al., 2018). BERT (Devlin et al., 2019) (as well as its robustly optimized version RoBERTa (Liu et al., 2019b)) employs bidirectional transformer encoders (Vaswani et al., 2017) and self-supervised tasks to generate context-aware token representations. Further improvement of performances mostly based on the following three types of techniques, including self-supervised tasks (Joshi et al., 2020), transformer encoder architectures (Yang et al., 2019) and multi-task learning (Liu et al., 2019a). Knowledge-Enhanced PLMs. As existing BERT-like models only learn knowledge from plain corpora, various works have investigated how to incorporate knowledge facts to enhance the language understanding abilities of PLMs. KEPLMs are mainly divided into the following three types. (1) Knowledge-enhanced by Entity Embedding: ERNIE-THU (Zhang et al., 2019) and KnowBERT (Peters et al., 2019) inject linked-entity as heterogeneous features learned by KG embedding algorithms such as TransE (Bordes et al., 2013). (2) Knowledge-enhanced by Entity Description: E-BERT (Zhang et al., 2020a) and KEPLER (Wang et al., 2019b) add extra description text of entities to enhance semantic representation. (3) Knowledge-enhanced by Triplet Sentence: K-BERT (Liu et al., 2020b) and CoLAKE (Sun et al., 2020) convert triplets into sentences and insert them into the training corpora without pre-trained embedding. Previous studies on KG embedding (Nguyen et al., 2016; Schlichtkrull et al., 2018) have shown that utilizing the surrounding facts of entity can obtain more informative embedding, which is the focus of our work. PLMs in the Medical Domain. PLMs in the medical domain can be generally divided into three categories. (1) BioBERT (Lee et al., 2020), Blue-BERT (Peng et al., 2019), SCIBERT (Beltagy et al., 2019) and ClinicalBert (Huang et al., 2019) apply continual learning on medical domain texts, such as PubMed abstracts, PMC full-text articles and MIMIC-III clinical notes. (2) PubMedBERT (cid:68)(cid:437)(cid:367)(cid:410)(cid:349)(cid:882)(cid:44)(cid:286)(cid:258)(cid:282)(cid:3)(cid:94)(cid:286)(cid:367)(cid:296)(cid:882)(cid:4)(cid:410)(cid:410)(cid:286)(cid:374)(cid:410)(cid:349)(cid:381)(cid:374) (cid:38)(cid:286)(cid:286)(cid:282)(cid:3)(cid:38)(cid:381)(cid:396)(cid:449)(cid:258)(cid:396)(cid:282)(cid:3)(cid:62)(cid:258)(cid:455)(cid:286)(cid:396) (cid:100)(cid:381)(cid:364)(cid:286)(cid:374)(cid:3)(cid:47)(cid:374)(cid:393)(cid:437)(cid:410)(cid:3) (cid:68)(cid:454) (cid:100)(cid:882)(cid:28)(cid:374)(cid:272)(cid:381)(cid:282)(cid:286)(cid:396) (cid:60)(cid:882)(cid:28)(cid:374)(cid:272)(cid:381)(cid:282)(cid:286)(cid:396) (cid:69)(cid:454) (cid:68)(cid:437)(cid:367)(cid:410)(cid:349)(cid:882)(cid:44)(cid:286)(cid:258)(cid:282)(cid:3)(cid:94)(cid:286)(cid:367)(cid:296)(cid:882)(cid:4)(cid:410)(cid:410)(cid:286)(cid:374)(cid:410)(cid:349)(cid:381)(cid:374) (cid:44)(cid:455)(cid:271)(cid:396)(cid:349)(cid:282)(cid:3)(cid:4)(cid:410)(cid:410)(cid:286)(cid:374)(cid:410)(cid:349)(cid:381)(cid:374)(cid:3)(cid:69)(cid:286)(cid:410)(cid:449)(cid:381)(cid:396)(cid:364) (cid:44)(cid:286)(cid:410)(cid:286)(cid:396)(cid:381)(cid:336)(cid:286)(cid:374)(cid:286)(cid:381)(cid:437)(cid:400)(cid:3)(cid:47)(cid:374)(cid:296)(cid:381)(cid:396)(cid:373)(cid:258)(cid:410)(cid:349)(cid:381)(cid:374)(cid:3)(cid:38)(cid:437)(cid:400)(cid:349)(cid:381)(cid:374) (cid:87)(cid:396)(cid:286)(cid:882)(cid:410)(cid:396)(cid:258)(cid:349)(cid:374)(cid:349)(cid:374)(cid:336)(cid:3)(cid:100)(cid:258)(cid:400)(cid:364)(cid:400) (cid:68)(cid:62)(cid:68)(cid:3) (cid:68)(cid:286)(cid:374)(cid:410)(cid:349)(cid:381)(cid:374)(cid:882)(cid:374)(cid:286)(cid:349)(cid:336)(cid:346)(cid:271)(cid:381)(cid:396)(cid:3)(cid:18)(cid:381)(cid:374)(cid:410)(cid:286)(cid:454)(cid:410)(cid:3)(cid:68)(cid:381)(cid:282)(cid:286)(cid:367)(cid:349)(cid:374)(cid:336) (cid:894)(cid:258)(cid:895) (cid:94)(cid:68)(cid:286)(cid:282)(cid:17)(cid:28)(cid:90)(cid:100)(cid:3)(cid:4)(cid:396)(cid:272)(cid:346)(cid:349)(cid:410)(cid:286)(cid:272)(cid:410)(cid:437)(cid:396)(cid:286) (cid:60)(cid:374)(cid:381)(cid:449)(cid:367)(cid:286)(cid:282)(cid:336)(cid:286)(cid:3)(cid:47)(cid:374)(cid:393)(cid:437)(cid:410) (cid:68)(cid:286)(cid:282)(cid:349)(cid:272)(cid:258)(cid:367)(cid:3)(cid:60)(cid:39) (cid:7136) (cid:2000) (cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3) (cid:11253) (cid:8706) (cid:3) (cid:708) (cid:18)(cid:75)(cid:115)(cid:47)(cid:24)(cid:882)(cid:1005)(cid:1013) (cid:709) (cid:69)(cid:286)(cid:349)(cid:336)(cid:346)(cid:271)(cid:381)(cid:396)(cid:349)(cid:374)(cid:336)(cid:28)(cid:374)(cid:410)(cid:349)(cid:410)(cid:349)(cid:286)(cid:400) (cid:100)(cid:455)(cid:393)(cid:286) (cid:69)(cid:381)(cid:282)(cid:286) (cid:69)(cid:381)(cid:282)(cid:286) (cid:87)(cid:28)(cid:87)(cid:90) (cid:100) (cid:455) (cid:393) (cid:286) (cid:74) (cid:202) (cid:44)(cid:455)(cid:271)(cid:396)(cid:349)(cid:282)(cid:3)(cid:4)(cid:410)(cid:410)(cid:856)(cid:3)(cid:920)(cid:3)(cid:47)(cid:374)(cid:296)(cid:381)(cid:396)(cid:856)(cid:3)(cid:38)(cid:437)(cid:400)(cid:349)(cid:381)(cid:374)(cid:3)(cid:68)(cid:381)(cid:282)(cid:437)(cid:367)(cid:286)(cid:400) (cid:202) (cid:68)(cid:286)(cid:374)(cid:410)(cid:349)(cid:381)(cid:374)(cid:882)(cid:374)(cid:286)(cid:349)(cid:336)(cid:346)(cid:271)(cid:381)(cid:396)(cid:3)(cid:18)(cid:381)(cid:374)(cid:410)(cid:286)(cid:454)(cid:410)(cid:3)(cid:68)(cid:381)(cid:282)(cid:286)(cid:367)(cid:349)(cid:374)(cid:336) (cid:3843)(cid:68)(cid:258)(cid:400)(cid:364)(cid:286)(cid:282)(cid:3)(cid:69)(cid:286)(cid:349)(cid:336)(cid:346)(cid:271)(cid:381)(cid:396)(cid:3)(cid:68)(cid:381)(cid:282)(cid:286)(cid:367)(cid:349)(cid:374)(cid:336) (cid:20) (cid:85) (cid:22) (cid:85) (cid:21) (cid:85) (cid:364)(cid:374)(cid:381)(cid:449)(cid:367)(cid:286)(cid:282)(cid:336)(cid:286)(cid:3)(cid:272)(cid:381)(cid:374)(cid:410)(cid:286)(cid:454)(cid:410) (cid:364)(cid:374)(cid:381)(cid:449)(cid:367)(cid:286)(cid:282)(cid:336)(cid:286)(cid:3)(cid:272)(cid:381)(cid:374)(cid:410)(cid:286)(cid:454)(cid:410) (cid:23) (cid:85) (cid:896)(cid:68)(cid:4)(cid:94)(cid:60)(cid:897) (cid:312) (cid:68)(cid:258)(cid:400)(cid:364)(cid:286)(cid:282)(cid:3)(cid:68)(cid:286)(cid:374)(cid:410)(cid:349)(cid:381)(cid:374)(cid:3)(cid:68)(cid:381)(cid:282)(cid:286)(cid:367)(cid:349)(cid:374)(cid:336) (cid:87)(cid:62)(cid:68)(cid:400) (cid:410)(cid:381)(cid:364)(cid:286)(cid:374)(cid:400) (cid:20) (cid:80) (cid:79) (cid:22) (cid:80) (cid:79) (cid:21) (cid:80) (cid:79) Figure 2: Model overview of SMedBERT. (Gu et al., 2020) learns weights from scratch using PubMed data to obtain an in-domain vocabulary, alleviating the out-of-vocabulary (OOV) problem. This training paradigm needs the support of large-scale domain data and resources. (3) Some other PLMs use domain self-supervised tasks for pretraining. For example, MC-BERT (Zhang et al., 2020b) masks Chinese medical entities and phrases to learn complex structures and concepts. Disease-BERT (He et al., 2020) leverages the medical terms and its category as the labels to pre-train the model. In this paper, we utilize both domain corpora and neighboring entity triplets of mentions to enhance the learning of medical language representations. In the PLM, we denote the hidden feature of each token { w 1 , ..., w N } as { h 1 , h 2 , ..., h N } where N is the maximum input sequence length and the total number of pre-training samples as M . Let E be the set of mention-span e m in the training corpora. Furthermore, the medical KG consists of the entities set E and the relations set R . The triplet set is S = { ( h, r, t ) | h  E , r  R , t  E} , where h is the head entity with relation r to the tail entity t . The embeddings of entities and relations trained on KG by TransR (Lin et al., 2015) are represented as  ent and  rel , respectively. The neighboring entity set recalled from KG by e m is denoted as N e m = { e 1 m , e 2 m , ..., e Km } where K is the threshold of our PEPR algorithm. We denote the number of entities in the KG as Z . The dimensions of the hidden representation in PLM and the KG embeddings are d 1 and d 2 , respectively. The main architecture of the our model is shown in Figure 2. SMedBERT mainly includes three components: (1) Top-K entity sorting determine which K neighbour entities to use for each mention. (2) Mention-neighbor hybrid attention aims to infuse the structured semantics knowledge into encoder layers, which includes type attention, node attention and gated position infusion module. (3) Mention-neighbor context modeling includes masked neighbor modeling and masked mention modeling aims to promote mentions to leverage and interact with neighbour entities. Previous research shows that simple neighboring entity expansion may induce knowledge noises during PLM training (Wang et al., 2019a). In order to recall the most important neighboring entity set from the KG for each mention, we extend the Personalized PageRank (PPR) (Page et al., 1999) algorithm to filter out trivial entities. 3 Recall that the iterative process in PPR is V i = (1   ) A  V i  1 + P where A is the normalized adjacency matrix,  is the damping factor, P is uniformly distributed jump probability vector, and V is the iterative score vector for each entity. PEPR specifically focuses on learning the weight for the target mention span in each iteration. It 3 We name our algorithm to be Personalized Entity PageRank, abbreviated as PEPR. where T is the sum of frequencies of all entities. t e m is the frequency of e m in the corpora. After sorting, we select the topK entity set N e m . Besides the embeddings of neighboring entities, SMedBERT integrates the type information of medical entities to further enhance semantic representations of mention-span. Different types of neighboring entities may have different impacts. Given a specific mention-span e m , we compute the neighboring entity type attention. Concretely, we calculate hidden representation of each entity type  as h  = (cid:80) e im  E m h e im . E m are neighboring entities of e m with the same type  and h e im =  ent (cid:0) e im (cid:1)  R d 2 . where f sp is the self-attentive pooling (Lin et al., 2017) to generate the mention-span representation h e m  R d 1 and the ( h i , h i +1 , . . . , h j ) is the hidden representation of tokens ( w i , w i +1 , . . . , w j ) in mention-span e m trained by PLMs. h (cid:48) e m  R d 2 is obtained by  (  ) non-linear activation function GELU (Hendrycks and Gimpel, 2016) and the learnable projection matrix W be  R d 1  d 2 . LN is the LayerNorm function (Ba et al., 2016). Then, we calculate the each type attention weight using the type representation h   R d 2 and the transformed mention-span representation h (cid:48) e m :  (cid:48)  = tanh (cid:0) h (cid:48) e m W t + h  W t (cid:48) (cid:1) W a (3) where W t  R d 2  d 2 , W t (cid:48)  R d 2  d 2 and W a  R d 2  1 . Finally, the neighboring entity type attention weights   are obtained by normalizing the attention score  (cid:48)  among all entity types T . Apart from entity type information, different neighboring entities also have different influences. Specifically, we devise the neighboring entity node attention to capture the different semantic influ-ences from neighboring entities to the target mention span and reduce the effect of noises. We calculate the entity node attention using the mention-span representation h (cid:48) e m and neighboring entities representation h e im with entity type  as:  (cid:48) e m e im = (cid:0) h (cid:48) e m W q (cid:1) (cid:0) h e im W k (cid:1) T  d 2   (4)  e m e im = exp (cid:16)  (cid:48) e m e im (cid:17) (cid:80) e im N em exp (cid:16)  (cid:48) e m e im (cid:17) (5) where W q  R d 2  d 2 and W k  R d 2  d 2 are the attention weight matrices. The representations of all neighboring entities in N e m are aggregated to  h (cid:48) e m  R d 2 : (cid:98) h (cid:48) e m = (cid:88) e i m N em  e m e im (cid:0) h e im W v + b v (cid:1) (6)  h (cid:48) e m = LN (cid:16)(cid:98) h (cid:48) e m + (cid:16)  (cid:16)(cid:98) h (cid:48) e m W l 1 + b l 1 (cid:17) W l 2 (cid:17)(cid:17) (7) where W v  R d 2  d 2 , W l 1  R d 2  4 d 2 , W l 2  R 4 d 2  d 2 . b v  R d 2 and b l 1  R 4 d 2 are the bias vectors. h (cid:48) e m is the mention-neighbor representation from hybrid attention module. Knowledge-injected representations may divert the texts from its original meanings. We further reduce knowledge noises via gated position infusion: h (cid:48) e mf =  (cid:0)(cid:2)  h (cid:48) e m (cid:107) h (cid:48) e m (cid:3) W mf + b mf (cid:1) (8) (cid:101) h (cid:48) e mf = LN ( h (cid:48) e mf W bp + b bp ) (9) where W mf  R 2 d 2  2 d 2 , W bp  R 2 d 2  d 1 , b mf  R 2 d 2 , b bp  R d 1 . h (cid:48) e mf  R 2 d 2 is the span-level infusion representation. (cid:107)  means concatenation operation. (cid:101) h (cid:48) e mf  R d 1 is the final knowledge-injected representation for mention e m . We generate the output token representation h if by 4 : g i = tanh (cid:16)(cid:16)(cid:104) h i (cid:107) (cid:101) h (cid:48) e mf (cid:105)(cid:17) W ug + b ug (cid:17) (10) h if =  (cid:16)(cid:16)(cid:104) h i (cid:107) g i  (cid:101) h (cid:48) e mf (cid:105)(cid:17) W ex + b ex (cid:17) + h i (11) where W ug , W ex  R 2 d 1  d 1 . b ug , b ex  R d 1 . means element-wise multiplication. 4 We find that restricting the knowledge infusion position to tokens is helpful to improve performance. To fully exploit the structured semantics knowledge in KG, we further introduce two novel self-supervised pre-training tasks, namely Masked Neighbor Modeling (MNeM) and Masked Mention Modeling (MMeM). Formally, let r be the relation between the mention-span e m and a neighboring entity e im : where h mf is the mention-span hidden features based on the tokens hidden representation (cid:0) h if , h ( i +1) f , . . . , h jf (cid:1) . h r =  rel ( r )  R d 2 is the relation r representation and W sa  R d 1  d 2 is a learnable projection matrix. The goal of MNeM is leveraging the structured semantics in surrounding entities while reserving the knowledge of relations between entities. Considering the object functions of skip-gram with negative sampling (SGNS) (Mikolov et al., 2013a) and score function of TransR (Lin et al., 2015): LS = log f s ( w, c ) + k  E c n  PD [log f s ( w,  c n )] (13) f tr ( h, r, t ) = (cid:107) hM r + r  tM r (cid:107) (14) where the w in LS is the target word of context c . f s is the compatibility function measuring how well the target word is fitted into the context. Inspired by SGNS, following the general energy-based framework (LeCun et al., 2006), we treat mention-spans in corpora as target words, and neighbors of corresponding entities in KG as contexts to provide additional global contexts. We employ the Sampled-Softmax (Jean et al., 2015) as the criterion L MNeM for the mention-span e m : (cid:88) N em log exp( f s (  )) exp( f s (  )) + K  E e n  Q ( e n ) [exp( f s (  (cid:48) ))] (15) where  denotes the triplet ( e m , r, e im ) , e im  N e m . (cid:48) is the negative triplets ( e m , r, e n ) , and e n is negative entity sampled with Q ( e im ) detailed in Appendix B. To keep the knowledge of relations between entities, we define the compatibility function as: f s (cid:0) e m , r, e im (cid:1) = h mf M r + h r || h mf M r + h r ||  ( h e im M r ) T || h e im M r ||  (16) where  is a scale factor. Assuming the norms of both h mf M r + h r and h e im M r are 1,we have: f s (cid:0) e m , r, e im (cid:1) =   f tr ( h mf , h r , h e im ) = 0 (17) which indicates the proposed f s is equivalence with f tr . Because | h e n M r | needs to be calculated for each e n , the computation of the score function f s is costly. Hence, we transform part of the formula f s as follows: ( h mf M r + h r )  ( h e n M r ) T = (cid:2) h mf 1 (cid:3) (cid:20) M r h r (cid:21) (cid:20) M r h r (cid:21) T (cid:2) h e n 0 (cid:3) T = (cid:2) h mf 1 (cid:3) MP r (cid:2) h e n 0 (cid:3) T (18) In this way, we eliminate computation of transforming each h e n . Finally, to compensate the offset introduced by the negative sampling function Q ( e im ) (Jean et al., 2015), we complement f s ( e m , r, e i m ) as: (cid:2) h mf 1 (cid:3) MP r (cid:107) (cid:2) h mf 1 (cid:3) MP r (cid:107) (cid:2) h e im 0 (cid:3) (cid:107) h e im (cid:107)    log Q ( e im ) (19) 3.4.2 Masked Mention Modeling In contrast to MNeM, MMeM transfers the semantic information in neighboring entities back to the masked mention e m . where Y m is the ground-truth representation of e m and h ip =  p ( w i )  R d 2 . p is the pre-trained embedding of BERT in our medical corpora. The mention-span representation obtained by our model is h mf . For a sample s , the loss of MMeM L MMeM is calculated via Mean-Squared Error: L MMeM = M s (cid:88) m i (cid:107) h m i f  Y m i (cid:107) 2 (21) where M s is the set of mentions of sample s . In SMedBERT, the training objectives mainly consist of three parts, including the self-supervised loss proposed in previous works and the mention-neighbor context modeling loss proposed in our work. Our model can be applied to medical text pre-training directly in different languages as long as high-quality medical KGs can be obtained. The total loss is as follows: L total = LEX +  1 L MNeM +  2 L MMeM (22) where LEX is the sum of sentence-order prediction (SOP) (Lan et al., 2020) and masked language modeling. 1 and  2 are the hyperparameters. Pre-training Data. The pre-training corpora after pre-processing contains 5,937,695 text segments with 3,028,224,412 tokens (4.9 GB). The KGs embedding trained by TransR (Lin et al., 2015) on two trusted data sources, including the Symptom-In-Chinese from OpenKG 5 and DXY-KG 6 containing 139,572 and 152,508 entities, respectively. The number of triplets in the two KGs are 1,007,818 and 3,764,711. The pre-training corpora and the KGs are further described in Appendix A.1. Task Data. We use four large-scale datasets in ChineseBLUE (Zhang et al., 2020b) to evaluate our model, which are benchmark of Chinese medical NLP tasks. Additionally, we test models on four datasets from real application scenarios provided by DXY company 7 and CHIP 8 , i.e., Named Entity Recognition (DXY-NER), Relation Extraction (DXY-RE, CHIP-RE) and Question Answer (WebMedQA (He et al., 2019)). For other information of the downstream datasets, we refer readers to Appendix A.2. In this work, we compare SMedBERT with general PLMs, domain-specific PLMs and KEPLMs with knowledge embedding injected, pre-trained on our Chinese medical corpora: General PLMs: We use three Chinese BERT-style models, namely BERT-base (Devlin et al., 2019), BERT-wwm (Cui et al., 2019) and RoBERTa (Liu et al., 2019b). All the weights are initialized from (Cui et al., 2020). Domain-specific PLMs: As very few PLMs in the Chinese medical domain are available, we consider the following models. MC-BERT (Zhang et al., 5 http://www.openkg.cn/dataset/ symptom-in-chinese 6 https://portal.dxy.cn/ 7 https://auth.dxy.cn/accounts/login 8 http://www.cips-chip.org.cn:8088/home Model D1 D2 D3 SGNS-char-med 27.21% 27.16% 21.72% SGNS-word-med 24.64% 24.95% 20.37% GLOVE-char-med 27.24% 27.12% 21.91% GLOVE-word-med 24.41% 23.89% 20.56% BERT-open 29.79% 29.41% 21.83% BERT-wwm-open 29.75% 29.55% 21.97% RoBERTa-open 30.84% 30.56% 21.98% MC-BERT 30.63% 30.34% 22.65% BioBERT-zh 30.84% 30.69% 22.71% ERNIE-med 30.97% 30.78% 22.99% KnowBERT-med 30.95% 30.77% 23.07% SMedBERT 31.81% 32.14% 24.08 % Table 1: Results of unsupervised semantic similarity task. med refers to models continually pre-trained on medical corpora, and open means open-domain corpora. char' and word refer to the token granularity of input samples. 2020b) is pre-trained over a Chinese medical corpora via masking different granularity tokens. We also pre-train BERT using our corpora, denoted as BioBERT-zh. KEPLMs: We employ two SOTA KEPLMs continually pre-trained on our medical corpora as our baseline models, including ERNIE-THU (Zhang et al., 2019) and KnowBERT (Peters et al., 2019). For a fair comparison, KEPLMs use other additional resources rather than the KG embedding are excluded (See Section 2), and all the baseline KEPLMs are injected by the same KG embedding. The detailed parameter settings and training procedure are in Appendix B. 4.3 Intrinsic Evaluation To evaluate the semantic representation ability of SMedBERT, we design an unsupervised semantic similarity task. Specifically, we extract all entities pairs with equivalence relations in KGs as positive pairs. For each positive pair, we use one of the entity as query entity while the other as positive candidate, which is used to sample other entities as negative candidates. We denote this dataset as D1 . Besides, the entities in the same positive pair often have many neighbours in common. We select positive pairs with large proportions of common neighbours as D2 . Additionally, to verify the ability of SMedBERT of enhancing the low-frequency mention representation, we extract all positive pairs that with at least one low-frequency mention as D3 . There are totally 359,358, 272,320 and 41,583 samples for D1 , D2 , D3 respectively. We describe the Named Entity Recognition Relation Extraction Model cMedQANER DXY-NER Average CHIP-RE DXY-RE Average Dev Test Dev Test Test Test Dev Test Test BERT-open 80.69% 83.12% 79.12% 79.03% 81.08% 85.86% 94.18% 94.13% 90.00% BERT-wwm-open 80.52% 83.07% 79.48% 79.29% 81.18% 86.01% 94.35% 94.38% 90.20% RoBERT-open 80.92% 83.29% 79.27% 79.33% 81.31% 86.19% 94.64% 94.66% 90.43% BioBERT-zh 80.72% 83.38% 79.52% 79.45% 81.42% 86.12% 94.54% 94.64% 90.38% MC-BERT 81.02% 83.46% 79.79% 79.59% 81.53% 86.09% 94.74% 94.73% 90.41% KnowBERT-med 81.29% 83.75% 80.86% 80.44% 82.10% 86.27% 95.05% 94.97% 90.62% ERNIE-med 81.22% 83.87% 80.82% 80.87% 82.37% 86.25% 94.98% 94.91% 90.58% SMedBERT 82.23% 84.75% 83.06% 82.94% 83.85% 86.95% 95.73% 95.89% 91.42% Table 2: Performance of Named Entity Recognition (NER) and Relation Extraction (RE) tasks in terms of F1. details of collecting data and embedding words in Appendix C. In this experiments, we compare SMedBERT with three types of models: classical word embedding methods ( SGNS (Mikolov et al., 2013a), GLOVE (Pennington et al., 2014)), PLMs and KEPLMs. We compute the similarity between the representation of query entities and all the other entities, retrieving the most similar one. The evaluation metric is top-1 accuracy (Acc@1). Experiment results are shown in Table 1. From the results, we observe that: (1) SMedBERT greatly outperforms all baselines especially on the dataset D2 (+1.36%) , where most positive pairs have many shared neighbours, demonstrating that ability of SMedBERT to utilize semantic information from the global context. (2) In dataset D3 , SMedBERT improve the performance significantly (+1.01%) , indicating our model is effective to enhance the representation of low-frequency mentions. We first evaluate our model in NER and RE tasks that are closely related to entities in the input texts. Table 2 shows the performances on medical NER and RE tasks. In NER and RE tasks, we can observe from the results: (1) Compared with PLMs trained in open-domain corpora, KEPLMs with medical corpora and knowledge facts achieve better results. (2) The performance of SMedBERT is greatly improved compared with the strongest baseline in two NER datasets (+0.88%, +2.07%) , and (+0.68%, +0.92%) on RE tasks. We also evaluate SMedBERT on QA, QM and NLI tasks and the performance is shown in Table 3. We can observe that SMedBERT improve the performance consistently on these datasets (+0.90% on QA, +0.89% on QM and +0.63% on NLI) . In general, it can be seen from Table 2 and Table 3 that injecting the domain knowledge especially the structured semantics knowledge can improve the result greatly. In this experiment, we explore the model performance in NER and RE tasks with different entity hit ratios, which control the proportions of knowledge-enhanced mention-spans in the samples. The aver-Figure 3: Entity hit ratio results of SMedBERT and ERNIE in NER and RE tasks. age number of mention-spans in samples is about 40. Figure 3 illustrates the performance of SMedBERT and ERNIE-med (Zhang et al., 2019). From the result, we can observe that: (1) The performance improves significantly at the beginning and then keeps stable as the hit ratio increases, proving the heterogeneous knowledge is beneficial to improve the ability of language understanding and indicating too much knowledge facts are unhelpful to further improve model performance due to the knowledge noise (Liu et al., 2020b). (2) Compared with previous approaches, our SMedBERT model improves performance greatly and more stable. We further evaluate the model performance under different K over the test set of DXY-NER and DXY-RE. Figure 4 shows the the model result with K = { 5 , 10 , 20 , 30 } . In our settings, the SMedBERT can achieve the best performance in different tasks around K = 10 . The results of SMedBERT show that the model performance increasing first and then decreasing with the increasing of K . This phenomenon also indicates the knowledge noise problem that injecting too much knowledge of neighboring entities may hurt the performance. In Table 4, we choose three important model components for our ablation study and report the test set performance on four datasets of NER and RE tasks that are closely related to entities. Specifically, the three model components are neighboring entity type attention, the whole hybrid attention module, and mention-neighbor context modeling respectively, which includes two masked language model loss L MNeM and L MMeM . From the result, we can observe that: (1) Without any of the three mechanisms, our model performance can also perform competitively with the strong baseline ERNIE-med (Zhang et al., 2019). (2) Note that after removing the hybrid attention module, the performance of our model has the greatest decline, which indicates that injecting rich heterogeneous knowledge of neighboring entities is effective. In this work, we address medical text mining tasks with the structured semantics KEPLM proposed named SMedBERT. Accordingly, we inject entity type semantic information of neighboring entities into node attention mechanism via heterogeneous feature learning process. Moreover, we treat the neighboring entity structures as additional global contexts to predict the masked candidate entities based on mention-spans and vice versa. The experimental results show the significant improvement of our model on various medical NLP tasks and the intrinsic evaluation. There are two research directions that can be further explored: (1) Injecting deeper knowledge by using farther neighboring entities as contexts; (2) Further enhancing Chinese medical long-tail entity semantic representation. We would like to thank anonymous reviewers for their valuable comments. This work is supported by the National Key Research and Development Program of China under Grant No. 2016YFB1000904, and Alibaba Group through Alibaba Research Intern Program.
abstain abstain method abstain objective abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method objective objective abstain abstain objective abstain method abstain abstain method objective other other other other other other other other other other other abstain other other other other other other other other other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method method other abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other abstain method method method abstain abstain method other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain result abstain abstain result abstain result result abstain abstain abstain abstain abstain result result abstain abstain result abstain abstain abstain abstain abstain result abstain objective abstain method result abstain other other