  Recent work on Graph Neural Networks has demonstrated that self-supervised
pretraining can further enhance performance on downstream graph, link, and node
classification tasks. However, the efficacy of pretraining tasks has not been
fully investigated for downstream large knowledge graph completion tasks. Using
a contextualized knowledge graph embedding approach, we investigate five
different pretraining signals, constructed using several graph algorithms and
no external data, as well as their combination. We leverage the versatility of
our Transformer-based model to explore graph structure generation pretraining
tasks (i.e. path and k-hop neighborhood generation), typically inapplicable to
most graph embedding methods. We further propose a new path-finding algorithm
guided by information gain and find that it is the best-performing pretraining
task across three downstream knowledge graph completion datasets. While using
our new path-finding algorithm as a pretraining signal provides 2-3% MRR
improvements, we show that pretraining on all signals together gives the best
knowledge graph completion results. In a multitask setting that combines all
pretraining tasks, our method surpasses the latest and strong performing
knowledge graph embedding methods on all metrics for FB15K-237, on MRR and
Hit@1 for WN18RRand on MRR and hit@10 for JF17K (a knowledge hypergraph
dataset).

Machine Learning