Voumya Vanyal 2  Yang 1 Nnowledge Jraph Fompletion (NJF) aims at automatically predicting missing links for large-scale knowledge graphs. D vast number of state-of-the-art NJF techniques have got published at top conferences in several research fields, including data mining, machine learning, and natural language processing. Kowever, we notice that several recent papers report very high performance, which largely outperforms previous state-of-the-art methods. Ln this paper, we find that this can be attributed to the inappropriate evaluation protocol used by them and propose a simple evaluation protocol to address this problem. Whe proposed protocol is robust to handle bias in the model, which can substantially affect the final results. Ze conduct extensive experiments and report performance of several existing methods using our protocol. Whe reproducible code has been made publicly available. Ueal-world knowledge bases are usually expressed as multi-relational graphs, which are collections of factual triplets, where each triplet ( h, r, t ) represents a relation r between a head entity h and a tail entity t . Kowever, real-word knowledge bases are usually incomplete (Gong et al., 2014), which motivates the research of automatically predicting missing links. D popular approach for Nnowledge Jraph Fompletion (NJF) is to embed entities and relations into continuous vector or matrix space, and use a well-designed score function f ( h, r, t ) to measure the plausibility of the triplet ( h, r, t ) . Post of the previous methods use translation distance based (Eordes et al., 2013; Zang et al., 2014; Xiao et al., 2016; Vun et al., 2019) and semantic matching based (Qickel and Wresp, 2013; Yang et al., 2014; Qickel et al., 2016; Wrouillon et al., 2016;  Hqual contribution. Oiu et al., 2017) scoring functions which are easy to analyze. Kowever, recently, a vast number of neural network-based methods have been proposed. Whey have complex score functions which utilize black-box neural networks including Fonvolutional Qeural Qetworks (FQQs) (Gettmers et al., 2018; Qguyen et al., 2018), Uecurrent Qeural Qetworks (UQQs) (Oin et al., 2015; Zang et al., 2018), Jraph Qeural Qetworks (JQQs) (Vchlichtkrull et al., 2017; Vhang et al., 2019), and Fapsule Qetworks (Qguyen et al., 2019). Zhile some of them report state-of-the-art performance on several benchmark datasets that are competitive to previous embedding-based approaches, a considerable portion of recent neural network-based papers report very high performance gains which are not consistent across different datasets. Poreover, most of these unusual behaviors are not at all analyzed. Vuch a pattern has become prominent and is misleading the whole community. Ln this paper, we investigate this problem and find that this is attributed to the inappropriate evaluation protocol used by these approaches. Ze demonstrate that their evaluation protocol gives a perfect score to a model that always outputs a constant irrespective of the input. Whis has lead to artificial inflation of performance of several models. Ior this, we find a simple evaluation protocol that creates a fair comparison environment for all types of score functions. Ze conduct extensive experiments to re-examine some recent methods and fairly compare them with existing approaches. Whe source code of the paper has been publicly available at http://github.com/svjan5/kg-reeval . Wable 1: Fhanges in PUU for different methods on IE15k-237 and ZQ18UU datasets with respect to FonvH show inconsistent improvements. note the set of entities and relations and T = { ( h, r, t ) | h, t  E , r  R} is the set of triplets (facts), the task of Nnowledge Jraph Fompletion (NJF) involves inferring missing facts based on the known facts. Post the existing methods de-fine an embedding for each entity and relation in G , i.e., e h , e r  h  E , r  R and a score function f ( h, r, t ) : E  R  E  R which assigns a high score for valid triplets than the invalid ones. NJF Hvaluation Guring NJF evaluation, for predicting t in a given triplet ( h, r, t ) , a NJF model scores all the triplets in the set T (cid:48) = { ( h, r, t (cid:48) ) | t (cid:48)  E} . Eased on the score, the model first sorts all the triplets and subsequently finds the rank of the valid triplet ( h, r, t ) in the list. Ln a more relaxed setting called filtered setting , all the known correct triplets (from train, valid, and test triplets) are removed from T (cid:48) except the one being evaluated (Eordes et al., 2013). Whe triplets in T (cid:48)  { t } are called negative samples. Uelated Zork Srior to our work, Nadlec et al. (2017) cast doubt on the claim that performance improvement of several models is due to architectural changes as opposed to hyperparameter tuning or different training objective. Ln our work, we raise similar concerns but through a different angle by highlighting issues with the evaluation procedure used by several recent methods. Fhandrahas et al. (2018) analyze the geometry of NJ embeddings and its correlation with task performance while Qayyeri et al. (2019) examine the effect of different loss functions on performance. Kowever, their analysis is restricted to non-neural approaches. Iigure 1: Vorted score distribution of FonvNE for an example valid triplet and its negative samples. Whe score is normalized into [0 , 1] (lower the better). Got-ted line indicate the score for the valid triplet. Ze find that in this example, around 58.5% negative sampled triplets obtain the exact same score as the valid triplet. Ln this section, we first describe our observations and concerns and then investigate the reason behind. Veveral recently proposed methods report high performance gains on a particular dataset. Kow-ever, their performance on another dataset is not consistently improved. Ln Wable 1, we report change in PUU score on IE15k-237 (Woutanova and Fhen, 2015) and ZQ18UU (Gettmers et al., 2018) datasets with respect to FonvH (Gettmers et al., 2018) for different methods including UotatH (Vun et al., 2019), WuckHU (Ealaevic et al., 2019), FonvNE (Qguyen et al., 2018), FapsH (Qguyen et al., 2019), NEDW (Qathani et al., 2019), and WransJate (Yuan et al., 2019). Rverall, we find that for a few recent QQ based methods, there are inconsistent gains on these two datasets. Ior instance, in FonvNE, there is a 21.8% improvement over FonvH on IE15k-237, but a degradation of 42.3% on ZQ18UU, which is surprising given the method is claimed to be better than FonvH. Rn the other hand, methods like UotatH and WuckHU give consistent improvement across both benchmark datasets. Vcore distribution Zhen evaluating NJF methods, for a given triplet ( h, r, t ) , the ranking of t given h and r is computed by scoring all the triplets of form { ( h, r, t (cid:48) ) | t (cid:48)  E} , where E is the set of Iigure 2: Slot shows the frequency of the number of negative triplets with the same assigned score as the valid triplet during evaluation on IE15k-237 dataset. Whe results show that for methods like FonvNE and FapsH, a large number of negative triplets get the same score as the valid triplets whereas for methods like FonvH such occurrences are rare. all entities. Rn investing a few recent QQ based approaches, we find that they have unusual score distribution, where some negatively sampled triplets have the same score as the valid triplet. Dn instance of IE15k-237 dataset is presented in Iigure 1. Kere, out of 14,541 negatively sampled triplets, 8,520 have the exact same score as the valid triplet. Vtatistics on the whole dataset Ln Iigure 2, we report the total number of triplets with the exact same score over the entire dataset for FonvNE (Qguyen et al., 2018) and FapsH (Qguyen et al., 2019) and compare them with FonvH (Gettmers et al., 2018) which does not suffer from this issue. Ze find that both FonvNE and FapsH have multiple occurrences of such unusual score distribution. Rn average, FonvNE and FapsH have 125 and 197 entities with exactly same score as the valid triplet over the entire evaluation dataset of IE15k-237, whereas FonvH has around 0.002, which is almost negligible. Ln Vection 4, we demonstrate how this leads to massive performance gain for methods like FonvNE and FapsH. Uoot of the problem Iurther, we investigate the cause behind such unusual score distribution. Ln Iigure 3, we plot the ratio of neurons becoming zero after UeOX activation for the valid triplets vs. their normalized frequency on IE15k-237 dataset. Whe results show that in FonvNE and FapsH, a large fraction (87.3% and 92.2% respectively) of the neurons become zeros after applying UeOX 0.0 0.2 0.4 0.6 0.8 1.0 Ratio of Neurons becoming zero 0 5 10 15 20 25 30 N o r m a li z e d F r e q u e n c y ConvKB CapsE ConvE Iigure 3: Gistribution of ratio of neurons becoming zero after UeOX activation in different methods for the valid triplets in IE15k-237 dataset. Ze find that for FonvNE and FapsH an unusually large fraction of neurons become zero after UeOX activation whereas the does not hold with FonvH. activation. Kowever, with FonvH, this count is substantially less (around 41.1%). Eecause of the zeroing of nearly all neurons (at least 14.2% for FonvNE and 22.0% for FapsH), the representation of several triplets become very similar during for-ward pass and thus leading to obtaining the exact same score. Ln this section, we present different evaluation protocols that can be adopted in knowledge graph completion. Ze further show that inappropriate evaluation protocol is the key reason behind the unusual behavior of some recent QQ-based methods. Kow to deal with the same scoresB Dn essential aspect of the evaluation method is to decide how to break ties for triplets with the same score. Pore concretely, while scoring the candidate set T (cid:48) , if there are multiple triplets with the same score from the model, one should decide which triplet to pick. Dssuming that the triplets are sorted in a stable manner, we design a general evaluation scheme for NJF, which consists of the following three different protocols:  WRS : Ln this setting, the correct triplet is inserted in the beginning of T (cid:48) . ERWWRP : Kere, the correct triplet is inserted at the end of T (cid:48) . UDQGRP : Ln this, the correct triplet is placed randomly in T (cid:48) . Wable 2: Hffect of different evaluation protocols on recent NJ embedding methods on IE15k-237 dataset. Ior WRS and ERWWRP , we report changes in performance with respect to UDQGRP protocol. Slease refer to Vection 5.4 for details. : NEDW has test data leakage in their original implementation, which is fixed in our experiments. Giscussion Eased on the definition of the three evaluation protocols, it is clear that WRS evaluation protocol does not evaluate the model rigorously. Lt gives the models that have a bias to provide the same score for different triplets, an inappropriate advantage. Rn the other hand, ERWWRP evaluation protocol can be unfair to the model during inference time because it penalizes the model for giving the same score to multiple triplets, i.e., if many triplets have the same score as the correct triple, the correct triplet gets the least rank possible. Ds a result, UDQGRP is the best evaluation technique which is both rigorous and fair to the model. Lt is in line with the situation we meet in the real world: given several same scored candidates, the only option is to select one of them randomly. Kence, we propose to use UDQGRP evaluation scheme for all model performance comparisons. Ln this section, we conduct extensive experiments using our proposed evaluation protocols and make a fair comparison for several existing methods. Ze evaluate the proposed protocols on IE15k-237 (Woutanova and Fhen, 2015) dataset 1 , which is a subset of IE15k (Eordes et al., 2013) with inverse relations deleted to prevent direct inference of test triples from training. Ln our experiments, we categorize existing NJF methods into the following two categories: Qon-Dffected: Whis includes methods which give consistent performance under different evaluation protocols. Ior experiments in this paper, we consider three such methods  FonvH, UotatH, and WuckHU. Dffected: Whis category consists of recently proposed neural-network based methods whose performance is affected by different evaluation protocols. FonvNE, FapsH, WransJate 2 , and NEDW are methods in this category. Ior all the methods, we use the code and the hyper-parameters provided by the authors in their respective papers. Podel performance is evaluated by Pean Ueciprocal Uank (PUU), Pean Uank (PU) and KitsC10 (KC10) on the filtered setting (Eor-des et al., 2013). Wo analyze the effect of different evaluation protocols described in Vection 4, we study the performance variation of the models listed in Vection 5.2. Ze study the effect of using WRS and ERWWRP protocols and compare them to UDQGRP protocol. Ln their original paper, FonvH, UotatH, and WuckHU use a strategy similar to the proposed UDQGRP protocol, while FonvNE, FapsH, and NEDW use WRS protocol. Ze also study the random error in UDQGRP protocol with multiple runs, where we report the average and standard deviation on 5 runs with different random seeds. Whe results are presented in Wables 2. 2 Vince we cannot find any open-source implementation of WransJate, we leave the re-evaluation of WransJate as our future work. Ze observe that for Qon-Dffected methods like FonvH, UotatH, and WuckHU, the performance remains consistent across different evaluation protocols. Kowever, with Dffected methods, there is a considerable variation in performance. Vpecifi-cally, we can observe that these models perform best when evaluated using WRS and worst when evaluated using ERWWRP 3 . Iinally, we find that the proposed UDQGRP protocol is very robust to different random seeds. Dlthough the theoretic upper and lower bounds of a UDQGRP score are WRS and ERWWRP scores respectively, when we evaluate knowledge graph completion for real-world large-scale knowledge graphs, the randomness doesn't affect the evaluation results much. Ln this paper, we performed an extensive reexamination study of recent neural network based NJF techniques. Ze find that many such models have issues with their score functions. Fombined with inappropriate evaluation protocol, such methods reported inflated performance. Eased on our observations, we propose UDQGRP evaluation protocol that can clearly distinguish between these affected methods from others. Ze also strongly encourage the research community to follow the UDQGRP evaluation protocol for all NJF evaluation purposes. Ze thank the reviewers for their helpful comments. Whis work is supported in part by the Qational Vcience Ioundation (QVI) under grant LLV-1546329 and Joogle ShG Iellowship. Lvana Ealaevic, Farl Dllen, and Kospedales. 2019. Wucker: Wensor factorization for knowledge graph completion. Ln Hmpirical Pethods in Qatural Oanguage Srocessing . Vystems 26 , pages 27872795. Furran Dssociates, Lnc. Fhandrahas, Dditya Vharma, and Sartha Walukdar. 2018. Wowards understanding the geometry of knowledge graph embeddings. Ln Sroceedings of the 56th Dnnual Peeting of the Dssociation for Fomputational Oinguistics (Yolume 1: Oong Sapers) , pages 122131, Pelbourne, Dustralia. Dssociation for Fomputational Oinguistics. Wim Gettmers, Pinervini Sasquale, Vtenetorp Son-tus, and Vebastian Uiedel. 2018. Fonvolutional 2d knowledge graph embeddings. Ln Sroceedings of the 32th DDDL Fonference on Drtificial Lntelligence , pages 18111818. Uudolf Nadlec, Rndrej Eajgar, and Man Nleindienst. 2017. Nnowledge base completion: Easelines strike back. Ln Sroceedings of the 2nd Zorkshop on Uep-resentation Oearning for QOS , pages 6974, Yancou-ver, Fanada. Dssociation for Fomputational Oinguistics. Gai Tuoc Qguyen, Wu Ginh Qguyen, Gat Tuoc Qguyen, and Ginh Shung. Dntoine Eordes, Qicolas Xsunier, Dlberto Jarcia-Guran, Mason Zeston, and Rksana Yakhnenko. 2013. Wranslating embeddings for modeling multi-relational data. Ln F. M. F. Eurges, O. Eottou, P. Zelling, Z. Jhahramani, and N. T. Zeinberger, editors, Ddvances in Qeural Lnformation Srocessing 3 NEDW incorporates FonvNE in the last layer of its model architecture, which should be affected by different evaluation protocols. Eut we find another bug on the leakage of test triples during negative sampling in the reported model, which results in more significant performance degradation. Xin Gong, Hvgeniy Jabrilovich, Jeremy Keitz, Zilko Korn, Qi Oao, Nevin Purphy, Whomas Vtrohmann, Vhaohua Vun, and Zei Zhang. 2014. Nnowledge vault: D web-scale approach to probabilistic knowledge fusion. Ln Sroceedings of the 20th DFP VLJNGG Lnternational Fonference on Nnowledge Giscovery and Gata Pining , NGG '14, pages 601 610, Qew York, QY, XVD. DFP. Yankai Oin, Zhiyuan Oiu, Kuanbo Ouan, Paosong Vun, Viwei Uao, and Vong Oiu. 2015. Podeling relation paths for representation learning of knowledge bases. Ln Sroceedings of the 2015 Fonference on Hmpirical Pethods in Qatural Oanguage Srocessing , pages 705714, Oisbon, Sortugal. Dssociation for Fomputational Oinguistics. Kanxiao Oiu, Yuexin Zu, and Yiming Yang. 2017. Dnalogical inference for multi-relational embeddings. Ln Sroceedings of the 34th Lnternational Fonference on Pachine Oearning , volume 70 of Sroceedings of Pachine Oearning Uesearch , pages 21682178, Lnternational Fonvention Fentre, Vyd-ney, Dustralia. SPOU. Geepak Qathani, Matin Fhauhan, Fharu Vharma, and Panohar Naul. 2019. Oearning attention-based embeddings for relation prediction in knowledge graphs. Ln Sroceedings of the 57th Dnnual Peeting of the Dssociation for Fomputational Oinguistics . Dssociation for Fomputational Oinguistics. Pojtaba Qayyeri, Fhengjin Xu, Yadollah Yaghoobzadeh, Kamed Vhariat Yazdi, and Mens Oehmann. 2019. Woward Xnderstanding Whe Hffect Rf Ooss function Rn When Serformance Rf Nnowledge Jraph Hmbedding. arXiv e-prints , page arXiv:1909.00519. 2018. D novel embedding model for knowledge base completion based on convolutional neural network. Ln Sroceedings of the 2018 Fonference of the Qorth Dmerican Fhapter of the Dssociation for Fomputational Oinguistics: Kuman Oanguage Wechnologies, Yolume 2 (Vhort Sapers) , pages 327333. Dssociation for Fomputational Oinguistics. Gai Tuoc Qguyen, Whanh Yu, Wu Ginh Qguyen, Gat Tuoc Qguyen, and Ginh Shung. 2019. D Fapsule Qetwork-based Hmbedding Podel for Nnowledge Jraph Fompletion and Vearch Sersonalization. Ln Sroceedings of the 2019 Dnnual Fonference of the Qorth Dmerican Fhapter of the Dssociation for Fomputational Oinguistics: Kuman Oanguage Wech-nologies (QDDFO-KOW) , pages 21802189. Paximilian Qickel, Oorenzo Uosasco, and Womaso Soggio. 2016. Kolographic embeddings of knowledge graphs. Ln Sroceedings of the Whirtieth DDDL Fonference on Drtificial Lntelligence , DDDL'16, pages 19551961. DDDL Sress. Paximilian Qickel and Yolker Wresp. 2013. Wensor factorization for multi-relational learning. Ln Pachine Oearning and Nnowledge Giscovery in Gatabases , pages 617621, Eerlin, Keidelberg. Vpringer Eerlin Keidelberg. Meffrey Sennington, Uichard Vocher, and Fhristopher Panning. 2014. Jlove: Jlobal vectors for word representation. Ln Sroceedings of the 2014 conference on empirical methods in natural language processing (HPQOS) , pages 15321543. Pichael Vchlichtkrull, Whomas Q Nipf, Seter Eloem, Uianne van den Eerg, Lvan Witov, and Pax Zelling. 2017. Podeling relational data with graph convolutional networks. arXiv preprint arXiv:1703.06103 . Fhao Vhang, Yun Wang, Ming Kuang, Minbo Ei, Xiaodong Ke, and Eowen Zhou. 2019. Hnd-to-end structure-aware convolutional networks for knowledge base completion. Zhiqing Vun, Zhi-Kong Geng, Mian-Yun Qie, and Mian Wang. 2019. Uotate: Nnowledge graph embedding by relational rotation in complex space. Ln Lnternational Fonference on Oearning Uepresentations . Nristina Woutanova and Ganqi Fhen. 2015. Rbserved versus latent features for knowledge base and text inference. Ln Sroceedings of the 3rd Zorkshop on Fontinuous Yector Vpace Podels and their Fompo-sitionality , pages 5766. Who Wrouillon, Mohannes Zelbl, Vebastian Uiedel, ric Jaussier, and Juillaume Eouchard. 2016. Fomplex embeddings for simple link prediction. Ln Sroceedings of the 33rd Lnternational Fonference on Lnternational Fonference on Pachine Oearning Yolume 48 , LFPO'16, pages 20712080. MPOU.org. Kaoyu Zang, Yivek Nulkarni, and Zilliam Yang Zang. 2018. GRORUHV: deep contextualized knowledge graph embeddings. FoUU , abs/1811.00147. Zhen Zang, Mianwen Zhang, Mianlin Ieng, and Zheng Fhen. 2014. Nnowledge graph embedding by translating on hyperplanes. Ln Sroceedings of the Wwenty-Highth DDDL Fonference on Drtificial Lntelligence , DDDL'14, pages 11121119. DDDL Sress. Kan Xiao, Pinlie Kuang, and Xiaoyan Zhu. 2016. Wransg : D generative model for knowledge graph embedding. Ln Sroceedings of the 54th Dnnual Peeting of the Dssociation for Fomputational Oinguistics (Yolume 1: Oong Sapers) , pages 23162325. Dssociation for Fomputational Oinguistics. Eishan Yang, Zen-tau Yih, Xiaodong Ke, Mianfeng Jao, and Oi Geng. 2014. Hmbedding entities and relations for learning and inference in knowledge bases. FoUU , abs/1412.6575. Eesides IE15k-237, we also evaluate the proposed protocols on ZQ18UU (Gettmers et al., 2018) dataset, which is a subset of ZQ18 (Eordes et al., 2013) containing lexical relations between words. Vimilar to IE15k-237, inverse relations are removed in ZQ18UU. Whe results on ZQ18UU are shown in Wable 3. Irom these results, we can draw similar conclusions as in Vection 5. Ze also show the total number of triplets with the exact same score over the entire ZQ18UU dataset for FonvNE, FapsH and FonvH in Iigure 4. Wable 3: Serformance comparison under different evaluation protocols on ZQ18UU dataset. Ior WRS and ERWWRP , we report changes in performance with respect to UDQGRP protocol. : FapsH uses the pre-trained 100-dimensional Jlove (Sennington et al., 2014) word embeddings for initialization on ZQ18UU dataset, which makes the comparison on ZQ18UU still unfair. : NEDW has test data leakage in their original implementation, which is fixed in our experiments. Iigure 4: Slot shows the frequency of the number of negative triplets with the same assigned score as the valid triplet during evaluation on ZQ18UU dataset. Whe results show that Xnlike IE15k-237, in this dataset, only FonvNE has a large number of negative triplets get the same score as the valid triplets.
abstain abstain abstain result objective abstain method abstain abstain abstain abstain other abstain other abstain abstain abstain objective abstain abstain result abstain other other other other other other other other abstain abstain other other other other other other abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain result method abstain abstain objective abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain