Knowledge graph completion (KGC) aims to reason over known facts and infer the missing links. Text-based methods such as KG-BERT (Yao et al., 2019) learn entity representations from natural language descriptions, and have the potential for inductive KGC. However, the performance of text-based methods still largely lag behind graph embedding-based methods like TransE (Bordes et al., 2013) and RotatE (Sun et al., 2019b). In this paper, we identify that the key issue is efficient contrastive learning. To improve the learning efficiency, we introduce three types of negatives: in-batch negatives, pre-batch negatives, and self-negatives which act as a simple form of hard negatives. Combined with InfoNCE loss, our proposed model SimKGC can substantially outperform embedding-based methods on several benchmark datasets. In terms of mean reciprocal rank (MRR), we advance the state-of-the-art by +19% on WN18RR, +6.8% on the Wikidata5M transductive setting, and +22% on the Wikidata5M inductive setting. Thorough analyses are conducted to gain insights into each component. Our code is available at https://github.com/ intfloat/SimKGC . Large-scale knowledge graphs (KGs) are important components for knowledge-intensive applications, such as question answering (Sun et al., 2019a), recommender systems (Huang et al., 2018), and intelligent conversational agents (Dinan et al., 2019) etc. KGs usually consist of a set of triples ( h , r , t ), where h is the head entity, r is the relation, and t is the tail entity. Popular public KGs include Freebase (Bollacker et al., 2008), Wikidata (Vran-decic and Krtzsch, 2014), YAGO (Suchanek et al., 2007), ConceptNet (Speer et al., 2017), and WordNet (Miller, 1992) etc. Despite their usefulness  Work done while at Yuanfudao AI Lab. in practice, they are often incomplete. Knowledge graph completion (KGC) techniques are necessary for the automatic construction and verification of knowledge graphs. Existing KGC methods can be categorized into two families: embedding-based and text-based methods. Embedding-based methods map each entity and relation into a low-dimensional vector, without using any side information such as entity descriptions. This family includes TransE (Bor-des et al., 2013), TransH (Wang et al., 2014), RotatE (Sun et al., 2019b), and TuckER (Balaze-vic et al., 2019) etc. By comparison, text-based methods (Yao et al., 2019; Xie et al., 2016; Wang et al., 2021c) incorporate available texts for entity representation learning, as shown in Figure 1. Intuitively, text-based methods should outperform embedding-based counterparts since they have access to additional input signals. However, results on popular benchmarks (e.g., WN18RR, FB15k-237, Wikidata5M) tell a different story: text-based methods still lag behind even with pre-trained language models. We hypothesize that the key issue for such performance degradation is the inefficiency in contrastive learning. Embedding-based methods do not involve the expensive computation of text en-4281 coders and thus can be extremely efficient to train with a large negative sample size. For example, the default configuration of RotatE 1 trains 1000 epochs with a negative sample size of 64 on the Wikidata5M dataset. While the text-based method KEPLER (Wang et al., 2021c) can only train 30 epochs with a negative sample size of 1 due to the high computational cost incurred by RoBERTa. In this paper, inspired by the recent progress on contrastive learning, we introduce three types of negatives to improve the text-based KGC method: in-batch negatives, pre-batch negatives, and self-negatives. By adopting bi-encoder instead of cross-encoder (Yao et al., 2019) architecture, the number of in-batch negatives can be increased by using a larger batch size. Vectors from previous batches are cached and act as pre-batch negatives (Karpukhin et al., 2020). Additionally, mining hard negatives can be beneficial for improving contrastive learning. We find that the head entity itself can serve as hard negatives, which we call self-negatives. As a result, the negative sample size can be increased to the scale of thousands. We also propose to change the loss function from margin-based ranking loss to InfoNCE, which can make the model focus on hard negatives. One advantage of text-based methods is that they enable inductive entity representation learning. Entities that are not seen during training can still be appropriately modeled, while embedding-based methods like TransE can only reason under the transductive setting 2 . Inductive knowledge graph completion is important in the real world as new entities are coming out every day. Moreover, text-based methods can leverage state-of-the-art pre-trained language models to learn better representations. A line of recent work (Shin et al., 2020; Petroni et al., 2019) attempts to elicit the implicitly stored knowledge from BERT. The task of KGC can also be regarded as a way to retrieve such knowledge. Two entities are more likely to be related if connected by a short path in the graph. Empirically, we find that text-based models heavily rely on the semantic match and ignore such topological bias to some degree. We propose a simple re-ranking strategy by boosting the scores of the head entity's k -hop neighbors. conducting experiments on three popular benchmarks: WN18RR, FB15k-237, and Wikidata5M (both transductive and inductive settings). According to the automatic evaluation metrics (MRR, Hits@{1,3,10}), SimKGC outperforms state-of-the-art methods by a large margin on the WN18RR (MRR 47 . 6  66 . 6 ), Wikidata5M transductive setting (MRR 29 . 0  35 . 8 ), and inductive setting (MRR 49 . 3  71 . 4 ). On the FB15k-237 dataset, our results are also competitive. To help better understand our proposed method, we carry out a series of analyses and report human evaluation results. Hopefully, SimKGC will facilitate the future development of better KGC systems. Knowledge Graph Completion involves modeling multi-relational data to aid automatic construction of large-scale KGs. In translation-based methods such as TransE (Bordes et al., 2013) and TransH (Wang et al., 2014), a triple ( h , r , t ) is a relation-specific translation from the head entity h to tail entity t . Complex number embeddings are introduced by Trouillon et al. (2016) to increase the model's expressiveness. RotatE (Sun et al., 2019b) models a triple as relational rotation in complex space. Nickel et al. (2011); Balazevic et al. (2019) treat KGC as a 3-D binary tensor factorization problem and investigate the effectiveness of several factorization techniques. Some methods attempt to incorporate entity descriptions. DKRL (Xie et al., 2016) uses a CNN to encode texts, while KG-BERT (Yao et al., 2019), StAR (Wang et al., 2021a), and BLP (Daza et al., 2021) both adopt pre-trained language models to compute entity embeddings. GraIL (Teru et al., 2020) and BERTRL (Zha et al., 2021) conduct inductive relation prediction by utilizing subgraph or path information. In terms of benchmark performance (Wang et al., 2021c), text-based methods still underperform methods like RotatE. Pre-trained Language Models including BERT (Devlin et al., 2019), GPT (Radford et al., 2018), and T5 (Raffel et al., 2019) have led to a learning paradigm shift in NLP. Models are first pre-trained on large amounts of unlabeled text corpora with language modeling objectives, and then fine-tuned on downstream tasks. Considering their good performance in few-shot and even zero-shot 4282 scenarios (Brown et al., 2020), one interesting question is: Can pre-trained language models be used as knowledge bases? Petroni et al. (2019) proposed to probe language models with manually designed prompts. A series of following work (Shin et al., 2020; Zhong et al., 2021; Jiang et al., 2020) focus on finding better prompts to elicit the knowledge implicitly stored in the model parameters. Another line of work (Zhang et al., 2019; Liu et al., 2020; Wang et al., 2021c) injects symbolic knowledge into language model pre-training, and shows some performance boost on several knowledge-intensive tasks. Contrastive Learning learns useful representations by contrasting between positives and negatives (Le-Khac et al., 2020). The definitions of positives and negatives are task-specific. In self-supervised vision representation learning (Chen et al., 2020; He et al., 2020; Grill et al., 2020), a positive pair is two augmented views of the same image, while a negative pair is two augmented views of different images. Recently, contrastive learning paradigm has witnessed great successes in many different fields, including multi-modal pre-training (Radford et al., 2021), video-text retrieval (Liu et al., 2021), and natural language understanding (Gunel et al., 2021) etc. In the NLP community, by leveraging the supervision signals from natural language inference data (Gao et al., 2021), QA pairs (Ni et al., 2021), and parallel corpora (Wang et al., 2021b), these methods have surpassed non-contrastive methods (Reimers and Gurevych, 2019) on semantic similarity benchmarks. Karpukhin et al. (2020); Qu et al. (2021); Xiong et al. (2021) adopt contrastive learning to improve dense passage retrieval for open-domain question answering, where the positive passages are the ones containing the correct answer. A knowledge graph G is a directed graph, where the vertices are entities E , and each edge can be represented as a triple ( h , r , t ), where h , r , and t correspond to head entity, relation, and tail entity, respectively. The link prediction task of KGC is to infer the missing triples given an incomplete G . Under the widely adopted entity ranking evaluation protocol, tail entity prediction ( h , r , ? ) requires ranking all entities given h and r , similarly for head entity prediction ( ? , r , t ). In this paper, for each triple ( h , r , t ), we add an inverse triple ( t , r  1 , h ), where r  1 is the inverse relation of r . Based on such reformulation, we only need to deal with the tail entity prediction problem (Malaviya et al., 2020). Our proposed model SimKGC adopts a bi-encoder architecture. Two encoders are initialized with the same pre-trained language model but do not share parameters. Given a triple ( h , r , t ), the first encoder BERT hr is used to compute the relation-aware embedding for the head entity h . We first concatenate the textual descriptions of entity h and relation r with a special symbol [SEP] in between. BERT hr is applied to get the last-layer hidden states. Instead of directly using the hidden state of the first token, we use mean pooling followed by L 2 normalization to get the relation-aware embedding e hr , as mean pooling has been shown to result in better sentence embeddings (Gao et al., 2021; Reimers and Gurevych, 2019). e hr is relation-aware since different relations will have different inputs and thus have different embeddings, even though the head entity is the same. Similarly, the second encoder BERT t is used to compute the L 2 -normalized embedding e t for the tail entity t . The input for BERT t only consists of the textual description for entity t . Since the embeddings e hr and e t are both L 2 normalized, the cosine similarity cos( e hr , e t ) is simply the dot product between two embeddings: cos( e hr , e t ) = e hr  e t (cid:107) e hr (cid:107)(cid:107) e t (cid:107) = e hr  e t (1) For tail entity prediction ( h , r , ? ), we compute the cosine similarity between e hr and all entities in E , and predict the one with the largest score: argmax t i cos( e hr , e t i ) , t i  E (2) 3.3 Negative Sampling For knowledge graph completion, the training data only consists of positive triples. Given a positive triple ( h , r , t ), negative sampling needs to sample one or more negative triples to train discriminative models. Most existing methods randomly corrupt h or t and then filter out false negatives that appear in the training graph G . The 4283 negatives for different triples are not shared and therefore independent. The typical number of negatives are  64 for embedding-based methods (Sun et al., 2019b), and  5 for text-based methods (Wang et al., 2021a). We combine three types of negatives to improve the training efficiency without incurring significant computational and memory overhead. In-batch Negatives (IB) This is a widely adopted strategy in visual representation learning (Chen et al., 2020) and dense passage retrieval (Karpukhin et al., 2020) etc. Entities within the same batch can be used as negatives. Such in-batch negatives allow the efficient reuse of entity embeddings for bi-encoder models. Pre-batch Negatives (PB) The disadvantage of in-batch negatives is that the number of negatives is coupled with batch size. Pre-batch negatives (Lee et al., 2021) use entity embeddings from previous batches. Since these embeddings are computed with an earlier version of model parameters, they are not consistent with in-batch negatives. Usually, only 1 or 2 pre-batches are used. Other methods like MoCo (He et al., 2020) can also provide more negatives. We leave the investigation of MoCo as future work. Self-Negatives (SN) Besides increasing the number of negatives, mining hard negatives (Gao et al., 2021; Xiong et al., 2021) is also important for improving contrastive representation learning. For tail entity prediction ( h , r , ? ), text-based methods tend to assign a high score to the head entity h , likely due to the high text overlap. To mitigate this issue, we propose self-negatives that use the head entity h as hard negatives. Including self-negatives can make the model rely less on the spurious text match. We use NIB , NPB , and NSN to denote the aforementioned three types of negatives. During training, there may exist some false negatives. For example, the correct entity happens to appear in another triple within the same batch. We filter out such entities with a binary mask 3 . Combining them all, the collection of negatives N ( h, r ) is: { t (cid:48) | t (cid:48)  NIB  NPB  NSN , ( h, r, t (cid:48) ) /  G} (3) 3 False negatives that do not appear in the training data will not be filtered. Assume the batch size is 1024 , and 2 pre-batches are used, we would have |N IB | = 1024  1 , |N PB | = 2  1024 , |N SN | = 1 , and |N ( h, r ) | = 3072 negatives in total. Knowledge graphs often exhibit spatial locality. Nearby entities are more likely to be related than entities that are far apart. Text-based KGC methods are good at capturing semantic relatedness but may not fully capture such inductive bias. We propose a simple graph-based re-ranking strategy: increase the score of candidate tail entity t i by   0 if t i is in k -hop neighbors E k ( h ) of the head entity h based on the graph from training set: argmax t i cos( e hr , e t i ) +  1 ( t i  E k ( h )) (4) 3.5 Training and Inference During training, we use InfoNCE loss with additive margin (Chen et al., 2020; Yang et al., 2019): L =  log e (  ( h,r,t )   ) / e (  ( h,r,t )   ) / + (cid:80) |N| i =1 e  ( h,r,t (cid:48) i ) / (5) The additive margin  > 0 encourages the model to increase the score of the correct triple ( h , r , t ). ( h, r, t ) is the score function for a candidate triple, here we define  ( h, r, t ) = cos( e hr , e t )  [  1 , 1] as in Equation 1. The temperature  can adjust the relative importance of negatives, smaller  makes the loss put more emphasis on hard negatives, but also risks over-fitting label noise. To avoid tuning  as a hyperparameter, we re-parameterize log 1  as a learnable parameter. For inference, the most time-consuming part is O ( |E| ) BERT forward pass computation of entity embeddings. Assume there are |T | test triples. For each triple ( h , r , ? ) and ( t , r  1 , ? ), we need to compute the relation-aware head entity embedding and use a dot product to get the ranking score for all entities. In total, SimKGC needs |E| + 2  |T | BERT forward passes, while cross-encoder models like KG-BERT (Yao et al., 2019) needs |E|  2  |T | . Being able to scale to large datasets is important for practical usage. For bi-encoder models, we can precompute the entity embeddings and retrieve top-k entities efficiently with the help of fast similarity search tools like Faiss (Johnson et al., 2021). Datasets We use three datasets for evaluation: WN18RR, FB15k-237, and Wikidata5M (Wang et al., 2021c). The statistics are shown in Table 1. Bordes et al. (2013) proposed the WN18 and FB15k datasets. Later work (Toutanova et al., 2015; Dettmers et al., 2018) showed that these two datasets suffer from test set leakage and released WN18RR and FB15k-237 datasets by removing the inverse relations. The WN18RR dataset consists of  41 k synsets and 11 relations from WordNet (Miller, 1992), and the FB15k-237 dataset consists of  15 k entities and 237 relations from Freebase. The Wikidata5M dataset is much larger in scale with  5 million entities and  20 million triples. It provides two settings: transductive and inductive. For the transductive setting, all entities in the test set also appear in the training set, while for the inductive setting, there is no entity overlap between train and test set. We use Wikidata5M-Trans and Wikidata5M-Ind to indicate these two settings. For textual descriptions, we use the data provided by KG-BERT (Yao et al., 2019) for WN18RR and FB15k-237 datasets. The Wikidata5M dataset already contains descriptions for all entities and relations. Evaluation Metrics Following previous work, our proposed KGC model is evaluated with entity ranking task: for each test triple ( h, r, t ) , tail entity prediction ranks all entities to predict t given h and r , similarly for head entity prediction. We use four automatic evaluation metrics: mean reciprocal rank (MRR), and Hits@ k ( k  { 1 , 3 , 10 }) (H@ k for short). MRR is the average reciprocal rank of all test triples. H@ k calculates the proportion of correct entities ranked among the topk . MRR and H@ k are reported under the filtered setting (Bor-des et al., 2013), The filtered setting ignores the scores of all known true triples in the training, validation, and test set. All metrics are computed by averaging over two directions: head entity prediction and tail entity prediction. Hyperparameters The encoders are initialized with bert-base-uncased (English). Using better pre-trained language models is expected to improve performance further. Most hyperparameters except learning rate and training epochs are shared across all datasets to avoid dataset-specific tuning. We conduct grid search on learning rate with ranges { 10  5 , 3  10  5 , 5  10  5 }. Entity descriptions are truncated to a maximum of 50 tokens. Temperature  is initialized to 0 . 05 , and the additive margin for InfoNCE loss is 0 . 02 . For re-ranking, we set  = 0 . 05 . 2 pre-batches are used with logit weight 0 . 5 . We use AdamW optimizer with linear learning rate decay. Models are trained with batch size 1024 on 4 V100 GPUs. For the WN18RR, FB15k-237, and Wikidata5M (both settings) datasets, we train for 50 , 10 , and 1 epochs, respectively. Please see Appendix A for more details. We reuse the numbers reported by Wang et al. (2021c) for TransE and DKRL, and the results for RotatE are from the official GraphVite 4 benchmark. In Table 2 and 3, our proposed model SimKGC IB+PB+SN outperforms state-of-the-art methods by a large margin on the WN18RR, Wikidata5M-Trans, and Wikidata5M-Ind datasets, but slightly lags behind on the FB15k-237 dataset (MRR 33 . 6% vs 35 . 8% ). To the best of our knowledge, SimKGC is the first text-based KGC method that achieves better results than embedding-based counterparts. 4 https://graphvite.io/docs/latest/benchmark 4285 Method Wikidata5M-Trans Wikidata5M-Ind MRR H@1 H@3 H@10 MRR H@1 H@3 H@10 embedding-based methods TransE (Bordes et al., 2013) 25.3 17.0 31.1 39.2 --RotatE (Sun et al., 2019b) 29.0 23.4 32.2 39.0 --text-based methods DKRL (Xie et al., 2016) 16.0 12.0 18.1 22.9 23.1 5.9 32.0 54.6 KEPLER (Wang et al., 2021c) 21.0 17.3 22.4 27.7 40.2 22.2 51.4 73.0 BLP-ComplEx (Daza et al., 2021) --48.9 26.2 66.4 87.7 BLP-SimplE (Daza et al., 2021) --49.3 28.9 63.9 86.6 SimKGC IB 35.3 30.1 37.4 44.8 60.3 39.5 77.8 92.3 SimKGC IB+PB 35.4 30.2 37.3 44.8 60.2 39.4 77.7 92.4 SimKGC IB+SN 35.6 31.0 37.3 43.9 71.3 60.7 78.7 91.3 SimKGC IB+PB+SN 35.8 31.3 37.6 44.1 71.4 60.9 78.5 91.7 Table 2: Main results for the Wikidata5M dataset. We report results for various combinations of negatives. With in-batch negatives only, the performance of SimKGC IB is already quite strong thanks to the large batch size (1024) we use. Adding self-negatives tends to improve H@1 but hurt H@10. We hypothesize that self-negatives make the model rely less on simple text match. Thus they have negative impacts on metrics that emphasize recall, such as H@10. Combining all three types of negatives generally has the best results but not always. Compared to other datasets, the graph for the FB15k-237 dataset is much denser (average degree is  37 per entity), and contains fewer entities (  15 k ). To perform well, models need to learn generalizable inference rules instead of just modeling textual relatedness. Embedding-based methods are likely to hold an advantage for this scenario. It is possible to ensemble our method with embedding-based ones, as done by Wang et al. (2021a). Since this is not the main focus of this paper, we leave it as future work. Also, Cao et al. (2021) points out that many links in the FB15k-237 dataset are not predictable based on the available information. These two reasons help explain the unsatisfactory performance of SimKGC. Adding self-negatives is particularly helpful for the inductive setting of Wikidata5M dataset, with MRR rising from 60 . 3% to 71 . 3% . For inductive KGC, text-based models rely more heavily on text match than the transductive setting. Self negatives 4286 can prevent the model from simply predicting the given head entity. In terms of inference time, the most expensive part is the forward pass with BERT. For the Wikidata5M-Trans dataset, SimKGC requires  40 minutes to compute  4 . 6 million embeddings with 2 GPUs, while cross-encoder models such as KG-BERT (Yao et al., 2019) would require an estimated time of 3000 hours. We are not the first work that enables fast inference, models such as ConvE (Dettmers et al., 2018) and StAR (Wang et al., 2021a) also share similar advantages. Here we just want to re-emphasize the importance of inference efficiency and scalability when designing new models. We conduct a series of analyses to gain further insights into our proposed model and the KGC task. Compared to existing text-based methods, SimKGC makes two major changes: using more negatives, and switching from margin-based ranking loss to InfoNCE loss. To guide the future work on knowledge graph completion, it is crucial to understand which factor contributes most to the superior performance of SimKGC. In Table 4, we use SimKGC IB with batch size 256 as a baseline. By reducing the number of negatives from 255 to 5 , MRR drops from 64 . 4 to 48 . 8 . Changing the loss function from InfoNCE to the following margin loss makes MRR drop to 39 . 5 : 1 |N | |N| (cid:88) i =1 max(0 ,  +  ( h, r, t (cid:48) i )   ( h, r, t )) (6) Consistent with Equation 5,  ( h, r, t (cid:48) i ) is cosine similarity score for a candidate triple, and  = 0 . 8 . To summarize, both InfoNCE loss and a large number of negatives are important factors, while the loss function seems to have bigger impacts. For InfoNCE loss, the hard negatives naturally contribute larger gradients, and adding more negatives can lead to more robust representations. Wang and Liu (2021) also draws a similar conclusion: such hardness-aware property is vital for the success of contrastive loss. changing the weight in Equation 6 from |N| to exp( s ( t (cid:48) i ) / ) (cid:80) |N| j =1 exp( s ( t (cid:48) j ) / ) , where s ( t (cid:48) i ) = max(0 ,  +  ( h, r, t (cid:48) i )   ( h, r, t )) and  = 0 . 05 . Similar to InfoNCE loss, margin  loss makes the model pay more attention to hard negatives and leads to better performance as shown in Table 4. It is similar to the self-adversarial negative sampling proposed by Sun et al. (2019b). Most hyperparameters are tuned based on InfoNCE loss. We expect the margin loss to achieve better results with a bit more hyperparameter optimization. In Figure 2, we quantitatively illustrate how MRR changes as more negatives are added. There is a clear trend that the performance steadily improves from 48 . 8 to 67 . 1 . However, adding more negatives requires more GPU memory and may cause optimization difficulties (You et al., 2020; Chen et al., 2020). We do not experiment with batch size larger than 1024 . Our proposed re-ranking strategy is a simple way to incorporate topological information in the knowledge graph. For graphs whose connectivity patterns exhibit spatial locality, re-ranking is likely to help. In Table 6, we see a slight but stable increase for all metrics on the Wikidata5M-Trans dataset. Note that this re-ranking strategy does not apply to inductive KGC since entities in the test set never appear in the training data. Exploring more effective ways such as graph neural networks (Wu et al., 2019) instead of simple re-ranking would be a future direction. 5.3 Fine-grained Analysis 1-1 1-n spouse capital of lake inflows head of government child has part notable work side effect n-1 n-n instance of place of birth given name work location cast member member of influenced by nominated for Table 7: Examples for different categories of relations on the Wikidata5M-Trans dataset. We classify all relations into four categories based on the cardinality of head and tail arguments following the rules by Bordes et al. (2013): one-to-one(1-1), one-to-many(1-n), many-to-one(n-1), and many-to-many(n-n). Examples are shown in Dataset 1-1 1-n n-1 n-n Wikidata5M-Trans 30.4 8.3 71.1 10.6 Wikidata5M-Ind 83.5 71.1 80.0 54.7 Table 8: MRR for different kinds of relations on the Wikidata5M dataset with SimKGC IB+PB+SN . Table 7. As shown in Table 8, predicting the n side is generally more difficult, since there are many seemingly plausible answers that would confuse the model. Another main reason is the incompleteness of the knowledge graph. Some predicted triples might be correct based on human evaluation, especially for 1-n relations in head entity prediction, such as instance of, place of birth etc. In Table 5, for the first example, Marbletown, Ulster County, and New York are both correct answers. The second example illustrates the case for relation place of birth: a lot of people share the same place of birth, and some triples may not exist in the knowledge graph. This helps explain the low performance of 1-n relations for the Wikidata5M-Trans dataset. In the third example, SimKGC predicts a closely related but incorrect entity http server. The analyses above suggest that automatic evaluation metrics such as MRR tend to underestimate the model's performance. To have a more accurate estimation of the performance, we conduct human evaluation and list the results in Table 9. An average of 49% of the wrong predictions according to H@1 are correct according to human annotators. If we take this into account, the H@1 of our proposed model would be much higher. How to accurately 4288 correct wrong unknown ( h , r , ? ) 24% 54% 22% ( ? , r , t ) 74% 14% 12% Avg 49% 34% 17% Table 9: Human evaluation results on the Wikidata5M-Trans dataset. measure the performance of KGC systems is also an interesting future research direction. To examine our proposed model qualitatively, we visualize the entity embeddings from 8 largest categories 5 with 50 randomly selected entities per category. Entity embeddings are computed with BERT t in Section 3.2. In Figure 3, different categories are well separated, demonstrating the high quality of the learned embeddings. One interesting phenomenon is that the two categories Commu-nity and Village have some overlap. This is reasonable since these two concepts are not mutually exclusive. 5 We utilize the instance of relation to determine the entity category. efficient contrastive learning. Leveraging the recent progress in the field of contrastive learning, SimKGC adopts a bi-encoder architecture and combines three types of negatives. Experiments on the WN18RR, FB15k-237, and Wikidata5M datasets show that SimKGC substantially outperforms state-of-the-art methods. For future work, one direction is to improve the interpretability of SimKGC. In methods like RotatE (Sun et al., 2019b) and TransE (Bordes et al., 2013), a triple can be modeled as rotation in complex space or relational translation, while SimKGC does not enable such easy-to-understand interpretations. Another direction is to explore effective ways to deal with false negatives (Huynh et al., 2020) resulting from the incompleteness of knowledge graphs. Future work could use SimKGC as a solid baseline to keep improving text-based knowledge graph completion systems. Our experimental results and analyses also reveal several promising research directions. For example, how to incorporate global graph structure in a more principled way? Are there other loss functions that perform better than the InfoNCE loss? For knowledge-intensive tasks such as knowledge base question answering (KBQA), information retrieval, and knowledge-grounded response generation, etc., it would be interesting to explore the new opportunities brought by the improved knowledge graph completion systems. We would like to thank anonymous reviewers and area chairs for their valuable comments, and ACL Rolling Review organizers for their efforts.
abstain abstain abstain method result objective abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain result abstain abstain abstain result abstain objective abstain abstain abstain abstain abstain abstain abstain result objective abstain abstain result objective abstain other other other other other other other other other other other other other other other other other other other other other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain other method abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method method abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain result abstain abstain other result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other