We present InferWiki, a Knowledge Graph Completion (KGC) dataset that improves upon existing benchmarks in inferential ability, assumptions, and patterns. First, each testing sample is predictable with supportive data in the training set. To ensure it, we propose to utilize rule-guided train/test generation, instead of conventional random split. Second, InferWiki initiates the evaluation following the open-world assumption and improves the inferential difficulty of the closed-world assumption, by providing manually annotated negative and unknown triples. Third, we include various inference patterns (e.g., reasoning path length and types) for comprehensive evaluation. In experiments, we curate two settings of InferWiki varying in sizes and structures, and apply the construction process on CoDEx as comparative datasets. The results and empirical analyses demonstrate the necessity and high-quality of InferWiki. Nevertheless, the performance gap among various inferential assumptions and patterns presents the difficulty and inspires future research direction. Our datasets can be found in https://github. com/TaoMiner/inferwiki . Knowledge Graph Completion (KGC) aims to predict missing links in KG by inferring new knowledge from existing ones. Attributed to its reasoning ability, KGC models are crucial in alleviating the KG's incompleteness issue and benefiting many downstream applications, such as recommendation (Cao et al., 2019b) and information extraction (Hu et al., 2021; Cao et al., 2020a). However, the KGC performance on existing benchmarks are still unsatisfactory  0.51 Hit Ratio@1 and 187 Mean Rank of the top-ranked model (Wang et al., 2019) on the widely used FB15k237 (Toutanova and Chen, 2015). Do we have a slow progress of Head Predicate Tail Test David location ? models (Akrami et al., 2020)? Or should we blame for the low-quality of benchmarks? In this paper, we re-think the task of KGC and construct a new benchmark dubbed InferWiki that highlights three fundamental objectives: Test triples should be inferential : this is the essential requirement of KGC. Each test triple should have supportive samples in the train set. However, we observe two major issues of current KGC datasets: unpredictable and meaningless test triples, which may hinder evaluating and advancing state-of-the-arts. As shown in Table 1, the first example of inferring the location for David (i.e., Florida) is even impossible for humans  not to mention machines  merely based on his birthplace and nationality (i.e., Atlanta and USA). In contrast, the second one is predictable but meaningless to find the missing month from a list of months within a year. The above cases are very common in existing datasets, e.g., YAGO3-10 (Dettmers et al., 2018) and CoDEx (Safavi and Koutra, 2020), mainly due to their construction process: first collecting a high-frequency subset of entities and then randomly splitting their triples into train/test. In this setting, KGC models may be overor under-estimated, as we are even unsure if a human can perform better. Test triples may be inferred positive, negative, or unknown . Following open-world assumption: what is not observed in KG is not necessar-FB15k237 WN18RR YAGO3-10 CoDEx-m Kinship Country InferWiki16k/64k Source FreeBase WordNet YAGO Wikidata Artificial Wikidata Inferential (cid:55) (cid:55) (cid:55) (cid:55) (cid:51) (cid:51) (cid:51) (cid:51) #Entity 14,541 40,943 123,182 17,050 104 272 16,288 64,718 #Relation 237 11 37 51 26 2 197 239 #train 272,115 86,835 1,079,040 185,584 8,548 1,111 162,424 782,243 #valid (+) 17,535 3,034 5,000 10,310 1,069 24 3,398 7,747 (\ UNK) \ -\ -\ -10,310 \ -\ -\ -1,910 \ 1,456 6,125 \ 1,605 #test (+) 20,466 3,134 5,000 10,311 1,069 24 3,398 7,747 (\ UNK) \ -\ -\ -10,311 \ -\ -\ -1,868 \ 1,501 6,062 \ 1,685 Table 2: Statistics of KGC datasets. However, existing benchmarks generate unseen triples as negatives (i.e., the closed-world assump-tion), because KG contains only positive triples. They usually randomly corrupt the head or tail entity in a triple, sometimes with type constraints (Li et al., 2019a). This leads to trivial evaluation (al-most 100% accuracy in triple classification (Safavi and Koutra, 2020)). Besides, the lack of unknown test ignores a critical inference capacity and may cause false negative errors in knowledge-driven tasks (Kotnis and Nastase, 2017). Inference has various patterns . Concentrating on limited patterns in evaluation may bring in severe bias. Domain-specific datasets Kinship (Kemp et al., 2006) and Country (Bouchard et al., 2015) only focus on a few relations and are nearly solved (Das et al., 2017). General-domain WN18RR (Dettmers et al., 2018) contains prevalent symmetry relation types, which incorrectly boosts the performance of RotatE (Abboud et al., 2020). Clearly, limited patterns leads to unfair comparisons among KGC models. To this end, we curated an Infer ential KGC dataset extracted from Wiki data and establish the benchmark with two settings of varying in sizes and structures: InferWiki64k and InferWiki16k . Instead of random split, we mine rules via AnyBURL (Meilicke et al., 2019) to guide train/test generation. All test triples are thus guaranteed inferential from training data. To avoid the rule leakage, we utilize two sets of triples: a large set for high-quality rule extraction and a small set for train/test split. Moreover, we infer unseen triples and manually annotate them with positive, negative and unknown labels to improve the difficulty of evaluation following both closed-world and openworld assumptions. For inference patterns, we include and balance triples with different reasoning path length, relation types and patterns (e.g., symmetry and composition). Our contributions can be summarized as follows:  We summarize three principles of KGC: inferential ability, assumptions and patterns, and construct a rule-guided dataset. We highlight the importance of negatives and unknowns, and initiate open-world evaluation. We conduct extensive experiments to establish the benchmark. The results and deep analyses verify the necessity and challenge of InferWiki, providing insights for future research. We can roughly classify current KGC datasets into two groups: inferential and non-inferential datasets. The first group is usually manually curated to ensure each testing sample can be inferred from training data through reasoning paths, while they only focus on specific relations, such as Families (Garcia-Duran et al., 2015), Kinship (Kemp et al., 2006), and Country (Bouchard et al., 2015). The limited scale and inference patterns make them not challenging. HOLE (Nickel et al., 2016) achieves 99.7% ACU-PR on the dataset of Country. The second group of datasets are automatically derived from public KGs and randomly split positive triples into train/test, leading to a risk of testing samples non-inferential from training data. Popular datasets include FB15k-237 (Toutanova and Chen, 2015), WN18RR (Dettmers et al., 2018), and YAGO3-10 (Dettmers et al., 2018). CoDEx (Safavi and Koutra, 2020) argues the scope and difficulty of the above datasets, thus propose a comprehensive dataset with manually verified hard negatives. In fact, inference is an important ability for intelligence. Various fields study how inference is done in practice, ranging from logic to cognitive psychology. Inference helps people make reliable predictions, which is also an expected ability for AI models. Indeed, once deployed, a model may have to make a prediction when there is no evidence in the training set. But, instead of an unreliable guess, we highlight the ability to know unknown, a.k.a. open-world assumption. Therefore, we aim to curate an large-scale inferential benchmark InferWiki including various inference patterns and testing samples (i.e., positive, negative, and unknown), for better evaluation. We list the statistics in Table 2. We describe our dataset construction that comprises four steps: data preprocessing, rule mining, rule-guided train/test generation, and inferred test labeling. We then give a detailed analysis. More and more studies utilize Wikidata 1 as a knowledge resource due to its high quality and large quantity. We utilize the September 2019 English dump in experiments. Data preprocessing aims to define relation vocabulary and extract two sets of triples from Wikidata: a large one for rule mining T r and a relatively small one for dataset generation T d . The reason for using two sets is to avoid the leakage of rules. In other words, some frequent rules on the large set may be very few on the small set. The different distributions shall avoid that rule mining methods will easily achieve high performance. Besides, more triples can improve the quality of mined rules. In contrast, the relatively small set is enough for efficient KGC training and evaluation. In specific, we first extract all triples that consist of two entity items and one relation with English labels. We then remove the repeated triples and obtain 40,199,175 triples with 7,734,841 entities and 1,170 different relation types. Considering rule mining efficiency, we reduce the relation vocabulary by (1) manually filtering out meaningless relations, such as movie ID or film rating, (2) removing relations of InstanceOf and subClassOf following existing benchmarks (Toutanova and Chen, 2015), (3) select the most frequent 500 relation types. We focus on the most frequent 800,000 entities, which result in 8,632,777 triples as the large set for rule mining. To obtain the small set for dataset construction, we further select the most frequent 120,000 entities and 300 relations, which result in 1,283,246 triples. Note that we also infer new triples and label them as positive, negative, or unknown later. Since developing advanced rule mining models is not the focus of this paper and several mature tools are available online, such as AMIE+ (Galarraga et al., 2015) and AnyBURL (Meilicke et al., 2019). We utilize AnyBURL 2 in experiments due to its efficiency and effectiveness. Given a set of triples (i.e., the large set T r ), this step aims to automatically learn rules F = { ( f p ,  p ) } Pp =1 , where f p denotes a horn rule, e.g., spouse ( x, y )  father ( x, z )  mother ( y, z ) , and  p  [0 , 1] denotes the confidence of f p . For each rule f p , the left side of  is called the premise, and the right side is called the conclusion, where the conclusion contains a single atom and the premise is a conjunction of several atoms in the Horn rule scheme. We can ground specific entities to replace x, y, z in f p , which shall denote an inferential relationship between premise and conclusion triples. For example, given spouse (LeBron James, Savannah Brinson) and father (LeBron James, Bronny James), we may infer a new triple mother (Savannah Brinson, Bronny James). Of course, not all of the mined rules are reasonable. To alleviate the negative impacts of unreasonable rules, we rely on more data (a large set of triples) and keep high-confidence rules only. Particularly, we follow the suggested configuration of AnyBURL. We run it for 500 seconds to ensure that all triples can be traversed at least once and obtain 251,317 rules, where 168,996 out of them whose confidence meets  p > 0 . 1 have been selected as the rule set to guide dataset construction. Different from existing benchmarks, InferWiki provides inferential testing triples with supportive data in the training set. Moreover, it aims to include as many inference patterns as possible and these patterns are better evenly distributed to avoid biased evaluation. Thus, this step has four objectives: rule-guided split, path extension, negative supplement, and inference pattern balance. Rule-guided Split grounds the mined rules F on triples T d to obtain premise triples and corresponding conclusion triples. All premise triples form a training set, and all conclusion triples form a test set. Thus, they are naturally guaranteed to be inferential. For correctness, all of premise triples 2 http://web.informatik.uni-mannheim. must exist in the given triple set T d , while conclusion triples are not necessarily in T d and may be generated for further annotation (i.e., Section 3.4). For example, given a rule spouse ( x, y )  father ( x, z )  mother ( y, z ) , we traverse all of the given triples and find entities LeBron James , Savannah Brinson , and Bronny James that meet the premise. We then add the premise triples spouse (LeBron James, Savannah Brinson) and father (LeBron James, Bronny James) into the training set, and generate the conclusion triple mother (Savannah Brinson, Bronny James) for testing, no matter it is given or not. Path Extension aims to increase the inference path patterns by (1) adding more reasoning paths for the same testing triple, and (2) elongating paths by replacing those premise triples that have reasoning paths. For example, we replace father (LeBron James, Bronny James) with two triples that can infer it: father (LeBron James, Bryce James) and brother (Bronny James, Bryce James). The original path is then extended by one hop. Correspondingly, we define the confidence of extended paths as the multiplication of all involved rules. Longer paths will challenge long-distance reasoning ability. Negative Supplement is to generate negative triples if we cannot annotate the same number of negatives with positive triples. Otherwise, we will face an imbalance issue. Following conventions, we randomly corrupt the head or tail entities in a positive triple with the following constraints: (1) the relation of the positive triple is exclusive, e.g., placeOfBirth , if the ratio from head to tail entities is smaller than a threshold (we choose 1 . 2 heuristically in experiments); otherwise, the corrupted negative triple may be actually positive, leading to false negative errors. (2) We choose positive triples from the test set for corruption to improve the difficulty  the model has to correctly infer the corresponding positive triple from training data, then classify the corrupted triple as negative through the confliction. Particularly, for non-exclusive relation types, most of their corrupted results should be unknown following open-world assumption. The inferred test set covers such cases, which will be discussed in Section 3.4. Inference Pattern Balance aims to balance various inference patterns, including path length, relation types, and relation patterns (i.e., symmetry, inversion, hierarchy, composition, and others). This is because concentrating on some patterns may lead to severe bias and unfair comparison between KGC models (Zhang et al., 2020). We first count the frequency of testing triples according to the path lengths, relation types and patterns, respectively. For each of them, we rank their counting and choose highest ranked groups of triples as frequent ones, instead of setting a threshold. We then carefully remove some frequent triples randomly, until the new distributions reach an accepted range (checked by humans). Different from existing datasets, InferWiki aims to include positive, negative, and unknown testing triples, to evaluate the model under two types of assumptions: open-world assumption and closedworld assumption. The main difference between them is whether unknown triples are regarded as negatives. That is, the open-world evaluation is a three-class classification problem (i.e., positive, negative, and unknown). The closed-world evaluation targets only positive and negative triples, and we can simply relabel unknown triples as negatives without changing the test set. So far, we have two test sets: one is generated via rule guidance, and the other contains the supplemented negatives. This section aims to label the generated triples. First, we automatically label the triples with positive if they exist in Wikidata. Then, we manually annotate the remaining 4,053 triples. The annotation guideline can be found in Appendix B. Note that all of the unknowns are factually incorrect but not inferential. To assess the quality of annotations, we verify a random selection of 300 test triples ( 100 for each label). The annotators agree with our labels 84 . 3 % of the time. We further investigate the disagreements by relabeling 100 samples. 85% of the time, humans prefer an unknown, while automatic labeling tends to assign them with positive or negative labels. This suggests the inferential difference between humans and machines  the capacity of knowing unknown. Finally, we remove the entities that are not in any of the grounded paths and their triples. We randomly select half of the test set as valid. This forms InferWiki64k . We further extract a dense subset InferWiki16k by filtering out the positive triples whose confidence is smaller than 0 . 6 . Correspondingly, negative/unknown triples are reduced to keep balance. The statistics is listed in Table 2. Table 3 shows positive, negative, and unknown examples of InferWiki and their (possible) supportive training data. For positives, their paths seem reasonable and vary in length, relation types, and patterns. The 7-hop path of the sibling example is even difficult for a human. For negatives and unknowns, they are indeed incorrect and more challenging. There are no direct contradicted triples in the train set  the model is encouraged to reason related triples and justify if there is a confliction (i.e., negative) or not (i.e., unknown). Nevertheless, there are two minor issues. First, some unreasonable paths may corrupt the predictability. We thus increase the rule confidence threshold  > 0 . 6 for InferWiki16k and manually annotate uncertain test triples for the correctness of labels. More advanced rule mining models can improve the construction pipeline. We leave it in the future. Second, does unknown triples have a bias on certain relation types? The answer is yes but not exactly. As shown in Table 3, the relation connectsW ith is involved in both positive and unknown triples, which is also determined by the paths. Next, we analyze the relation patterns and path length distribution through comparisons with existing KGC datasets. Due to the different construction pipelines, existing datasets are difficult to offer quantitative statistics. We thus apply our pipeline on CoDEx (Safavi and Koutra, 2020). Only inferential test triples remain, and the training set keeps unchanged, namely CoDEx-m-infer, which reduces the test and valid positives from 20,622 Figure 1: Distribution of paths in relation patterns. to 7,050. This agree with the original paper that reports 20.56% triples are symmetry or compositional through AMIE+ analysis. We find more paths due to more extensive rules extracted from a large set of triples. This also demonstrates the necessity of rule-guided train/test generation  most test triples are not guaranteed inferential when using random split. Relation Pattern Following convention, we count reasoning paths for various patterns: symmetry, inversion, hierarchy, composition, and others, whose detailed explanations and examples can be found in Appendix C. If a triple has multiple paths, we count all of them. As Figure 1 shows, we can see that (1) there are no inversion and only a few symmetry and hierarchy patterns in CoDEx-m, as most current datasets remove them to avoid train/test leakage. But, we argue that learning and remembering such patterns are also an essential capacity of inference. It just needs to control their numbers for a fair comparison. (2) The patterns of InferWiki is more evenly distributed. Note that the patterns Figure 2: Comparison of paths in different lengths. of symmetry, inversion, and hierarchy refer to 1-hop paths, while composition and others refer to multi-hop paths. So, the total number of the former three is almost the same as that of the latter two, to balance paths with varying lengths, which will be discussed next. Path Length Distribution The reasoning paths can ensure test triples' predictability but may not be the shortest ones, as there may be undiscovered paths connecting two entities. Thus, our statistics concerning path length offer a conservative analysis and give an upper bound. For a test triple with multiple paths, we count the shortest one. As shown in Figure 2, we can see that InferWiki has more long-distance paths, while CoDEx-m-infer normally concentrates on maximum 3-hop reasoning paths. In specific, the maximum path length of InferWiki is 9 (4 before path extension) and the average length is 2 . 9 ( 1 . 5 before path extension). Further analysis of relation, entity and neighbor distributions can be found in Appendix D&E. Although we carefully design the construction of inferWiki, there are still two types of limitations: rule biases and dataset errors, that can to be addressed along with the development of KG techniques in the future. In terms of rule biases, AnyBURL may be over-estimated due to its role in the construction. Although we utilize two triple sets to avoid rule leakage, their overlap may still bring unfair performance gain to AnyBURL. We consider synthesize several rule mining results to improve InferWiki in the next version. In terms of dataset errors, first, to balance positive and negative triples in the larger InferWiki64k, we follow conventions to randomly sample a portion of negatives. These negatives may be unknown if following open-world assumption. We manually assess the randomly sampled negatives and find a 15.7% error rate. Therefore, we conduct open-world experiments on the smaller InferWiki16k, all of whose testing negatives are verified by humans. The second type of errors is due to unreasonable rules for dataset split, which is caused by prediction errors of existing rule mining models. However, there is no suitable evaluation in this field to provide quantitative analysis. Our ongoing work aims to develop an automatic evaluation for path rationality to improve the mining quality, and thus facilitate our inferential pipeline. We benchmark performance on InferWiki for the tasks: (1) Link Prediction , the task of predicting the missing head/tail entity for a given query triple (?, r, t) or (h, r, ?). Models are encouraged to rank correct entities higher than others in the vocabulary. We adopt the filtering setting (Bor-des et al., 2013) that excludes those entities, if the predicted triples have been seen in the train set. Mean reciprocal rank (MRR) and hits@k are standard metrics for evaluation. (2) Triple Classification aims to predict a label for each given triple (h, r, t). The label following open-world assumption is trinary y  { 1 , 0 , 1 } and becomes binary y  { 1 , 1 } when adopting closed-world assumption  all 0 -label triples are re-labeled with  1 , since our unknown triples are factually negative yet non-inferential from training data. Since KGC models output real-value scores for triples, we classify scores into labels by choosing one or two thresholds per relation type on valid. Accuracy, precision, recall, and F1 are measurements. For comprehensive comparison, we choose three types of representative models as baselines: (1) Knowledge Graph Embedding models, including TransE (Bordes et al., 2013), ComplEx (Trouil-lon et al., 2016), RotatE (Sun et al., 2019), ConvE (Dettmers et al., 2018), and TuckER (Bal-azevic et al., 2019), (2) multihop reasoning model Multihop (Lin et al., 2018), and (3) rule-based AnyBURL (Meilicke et al., 2019). Note that the latter two are specially designed for link prediction. The detailed implementation including parameters and thresholds can be found in Appendix F. 4.3 Triple Classification Results Table 4 shows micro scores for triple classification. well  around 90% F1 scores. This is consistent with recent findings that triple classification is a nearly solved task (around 98% F1 scores) (Safavi and Koutra, 2020). Nevertheless, the lower performance demonstrates the difficulty of our curated datasets, mainly due to the manually annotated hard negatives of InferWiki (and CoDEx). Impacts of Hard Negatives Figure 3 presents the accuracy on InferWiki16k regarding various types of triples: positive, random supplemented negatives, and annotated negatives (including relabeled unknowns). We can see that (1) random negative triples are indeed trivial for all of baseline models, which motivates the necessity of harder negative triples to push this research direction forward, (2) positive triples are slightly difficult to judge than random negatives, and (3) the accuracy significantly drops on annotation negatives. This is mainly because most annotated triples are actually unknown  they are factually incorrect, but there are no obvious abnormal patterns. Such non-inferential cases may underestimate KGC models. Since most baselines fail in judging unknown as negative, we now investigate them following open-world assumption to see their ability in recog-Acc nizing unknown triples. Table 5 shows the macro performance 3 on InferWiki16k. We can see that all of the baseline models perform worse than those under the closed-world assumption. On one hand, the trinary classification is intuitively more difficult than binary classification. On the other hand, it is a rather straightforward method to search two decision thresholds  one between positive and unknown and the other between unknown and negative. This motivates us future works on advanced models to represent KG, which should also be able to detect the limitation and boundaries of given KG. It is a fundamental capacity of inference to respond I do not know, to avoid false negatives in downstream applications. Figure 4 presents a detailed analysis of each model regarding their search thresholds. We can see that although their best performance seems not bad, the worst scores are only around 10%. That is, they are very sensitive to thresholds. Besides, most of the time, the average F1 scores of ComplEx, RotatE, and TuckER are around 20%, while transE achieves higher scores. Maybe that is the reason why it is still the most widely used KGC method. ConvE stably outperforms other baselines, no matter in terms of best, worst, or average performance. Table 6 shows the average scores for head and tail prediction. We can see that (1) AnyBURL performs the best most of the time, but the performance gap is not significant. This is mainly due to its role in 3 Micro performance is only applicable to binary classification, while open-world evaluation is trinary. dataset construction, although we utilize two sets of triples to minimize rule leakage. Actually, inference of rules may be more important than we thought to improve the reliability and interpretability of knowledge-driven models. This also motivates us to incorporate rule knowledge into KGC training for advanced reasoning ability (Guo et al., 2018; Li et al., 2019b). (2) KGC models perform better on InferWiki16k than InferWiki64k, due to the higher structure density and rule confidence. (3) Models have higher hit@10 and lower hit@1 on InferWiki than other datasets (e.g., CoDEx). This agrees with an intuition that most entities are irrelevant, making it trivial to judge these corrupted triples as in triple classification. And, only a small portion of entities is difficult to predict, which requires strong inference ability. Besides, hit@1 varies a lot, so that we can better compare among models. Figure 5 presents Hit@1 curves for tail prediction regarding varying path length on InferWiki64k 4 . We can see an overall downwards trend along with the increasing path length. Meanwhile, the large fluctuation may be due to two possible reasons: (1) as discussed in Section 3.5, the inferential paths ensure the predictability, but may not be the shortest ones. This thus offers a conservative 4 Multihop is designed for tail prediction, and Hit@1 on InferWiki64k is more distinct for following ablation study. analysis and give an upper bound of the performance concerning k-hop paths. Our paths are of high coverage and quality compared with existing datasets, which either conduct case study or postprocess datasets via rule mining. (2) Relation types and patterns also have significant impacts. Shorter paths contain more long-tail relations, and longer paths tend to cover many common relations. This improves the difficulty of shorter paths and makes longer paths easier. We present the Hit@1 tail prediction on InferWiki64k regarding relation patterns in Table 7. We can see that symmetry and inversion are not well-solved, which should be considered into evaluation but limited in scale. TransE performs worse on symmetry and inversion relations, consistent with the analysis in Abboud et al. (2020). Even if ComplEx and RotatE can capture such patterns, they fail to rank corresponding entities at the top. Embedding-based models perform well on hierarchy relations, even outperforms AnyBURL. For compositional relations, it is still quite challenging and worthwhile further investigation. (a) Triple classification (F1). (b) Link Prediction (MRR). CoDEx-m. The two datasets share the same training set. The only difference lies in how we obtain the test triples, either using our proposed pipeline (CoDEx-m-infer) or randomly (CoDEx-m). Thus, the results reflect the impacts of inferential guarantee for dataset construction and demonstrate the necessity to avoid over-estimation or underestimation of the inferential ability of KGC models. We report the performance on CoDEx-m from the original paper (Safavi and Koutra, 2020). We can see that all of models perform better with inferential path guarantee on CoDEx-m-infer than CoDEx-m, except ComplEx for link prediction. This is because rule guidance elimites those noninferential testing triples, making the task easier. Nevertheless, the scores on hard cases are actually decreased (as discussed in Figure 3 and Table 7). Models are excepted a stronger reasoning ability among several related entities, instead of trivially filtering out massive irrelevant entities. This also demonstrates the necessity of InferWiki to avoid overor underestimation of the inferential ability of KGC models  learning new knowledge from existing ones. We illustrate the most frequent relation types and their distribution of InferWiki64k and InferWiki16k in Figure 8. We can see that InferWiki has a diverse relation types that are not limited to specific domains. Besides, the triples of each relation type are well balanced. We established a benchmark with three types of seven KGC models on two tasks of triple classification and link prediction. The results present a detailed analysis regarding various inference patterns, which demonstrates the necessity of an inferential guarantee for better evaluation and the difficulty of new open-world triple classification. In the future, we are interested in cross-KGs inference and transfer (Cao et al., 2019a), and investigating how to inject knowledge into deep learning architectures, such as for information extraction (Tong et al., 2020) or text generation (Cao et al., 2020b). This research was conducted in collaboration with SenseTime. This work is partially supported by A*STAR through the Industry Alignment Fund Industry Collaboration Projects Grant, by NTU (NTUACE2020-01) and Ministry of Education (RG96/20), and by the National Research Foundation, Prime Minister's Office, Singapore under its Energy Programme (EP Award No. NRF2017EWT-EP003-023) administrated by the Energy Market Authority of Singapore. This work is partially supported by Singapore MOE AcRF T1.
result abstain objective abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain objective abstain result abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method result abstain objective result method objective abstain other other other other other other other other other other abstain objective abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method other abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other abstain result method abstain method abstain method abstain method abstain method result abstain abstain abstain abstain objective method objective abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other other other