Knowledge graph embedding (KGE) models represent each entity and relation of a knowledge graph (KG) with low-dimensional embedding vectors. These methods have recently been applied to KG link prediction and question answering over incomplete KGs (KGQA). KGEs typically create an embedding for each entity in the graph, which results in large model sizes on real-world graphs with millions of entities. For downstream tasks these atomic entity representations often need to be integrated into a multi stage pipeline, limiting their utility. We show that an off-the-shelf encoder-decoder Transformer model can serve as a scalable and versatile KGE model obtaining state-of-the-art results for KG link prediction and incomplete KG question answering. We achieve this by posing KG link prediction as a sequence-to-sequence task and exchange the triple scoring approach taken by prior KGE methods with autoregressive decoding. Such a simple but powerful method reduces the model size up to 98% compared to conventional KGE models while keeping inference time tractable. After finetuning this model on the task of KGQA over incomplete KGs, our approach outperforms baselines on multiple large-scale datasets without extensive hyperparameter tuning. 1 1 Introduction A knowledge graph (KG) is a multi-relational graph where the nodes are entities from the real world (e.g. Barack Obama, United States ) and the named edges represent the relationships between them (e.g. Barack Obama born in United States ). KGs can be either domain-specific such as WikiMovies (Miller et al., 2016) or public, cross-domain KGs encoding common knowledge such as Wikidata and DBpedia (Heist et al., 2020). These graph-structured databases play an important role 1 Resources are available at https://github.com/ apoorvumang/kgt5 in knowledge-intensive applications including web search, question answering and recommendation systems (Ji et al., 2020). Most real-world knowledge graphs are incomplete. However, some missing facts can be inferred using existing facts in the KG (Bordes et al., 2013). This task termed knowledge graph completion (KGC) 2 has become a popular area of research in recent years (Wang et al., 2017) and is often approached using knowledge graph embedding (KGE) models. KGE models represent each entity and relation of the KG by a dense vector embedding. Using these embeddings the model is trained to distinguish correct from incorrect facts. One of the main downstream applications of KGEs is question answering over incomplete KGs (KGQA) (Choudhary et al., 2021). Taking into account the large size of real world KGs (Wikidata contains  90M entities) and the applicability to downstream tasks, KGE models should fulfill the following desiderata: (i) scalability  i.e. have model size and inference time independent of the number of entities (ii) quality  reach good empirical performance (iii) versatility  be applicable for multiple tasks such as KGC and QA, and (iv) simplicity  consist of a single module with a standard architecture and training pipeline. Traditional KGE models fulfill quality and simplicity. They build upon a simple architecture and reach a high quality in terms of KGC. However, as they create a unique embedding per entity/relation, they scale linearly with the number of entities in the graph, both in model size and inference time, and offer limited versatility. Methods such as DKRL (Xie et al., 2016a) and KEPLER (Wang et al., 2021) attempt to tackle the scalability issue using compositional embeddings. However, they fail to achieve quality comparable to conventional KGEs. KG-BERT (Yao et al., 2019) utilizes pretrained BERT for link prediction and holds po-2 We use the term KGC for the task of KG link prediction. tential in terms of versatility as it is applicable to downstream NLP tasks. However, it is not scalable due to its underlying cross-encoder. 3 QA methods which leverage KGEs outperform traditional KGQA approaches on incomplete KGs, but combining KGEs with the QA pipeline is a non-trivial task; models that attempt to do this often work on only limited query types (Huang et al. 2019; Sun et al. 2021; Saxena et al. 2020) or require multistage training and inference pipelines (Ren et al., 2021). Here, in order to achieve quality, these models have sacrificed versatility and simplicity. A comparison of approaches in terms of desiderata is summarized in Tab. 9 in the appendix. Our paper shows that all of these desiderata can be fulfilled by a simple sequence-to-sequence (seq2seq) model. To this end, we pose KG link prediction as a seq2seq task and train an encoder-decoder Transformer model (Vaswani et al., 2017) on this task. We then use this model pretrained for link prediction and further finetune it for question answering; while finetuning for QA, we regularize with the link prediction objective. This simple but powerful approach, which we call KGT5, is visualised in Fig. 1. With such a unified seq2seq approach we achieve (i) scalability  by using compositional entity representations and autoregressive decoding (rather than scoring all entities) for inference (ii) quality  we obtain state-of-the-art performance on two tasks (iii) versatility  the same model can be used for both KGC and KGQA on multiple datasets, and (iv) simplicity  we obtain all results using an off-the-shelf model with no task or dataset-specific hyperparameter tuning. In summary, we make the following contributions:  We show that KG link prediction and question answering can be treated as sequence-to-sequence tasks and tackled successfully with a single encoder-decoder Transformer (with the same architecture as T5-small (Raffel et al., 2020)). With this simple but powerful approach called 3 Shen et al. (2020) estimate it would take KG-BERT 3 days for an evaluation run on a KG with just 40k entities. KGT5, we reduce model size for KG link prediction up to 98% while outperforming conventional KGEs on a dataset with 90M entities. We show the versatility of this approach through the task of KGQA over incomplete graphs. By pretraining on KG link prediction and finetuning on QA, KGT5 performs similar to or better than much more complex methods on multiple large-scale KGQA benchmarks. Given a set of entities E and a set of relations R , a knowledge graph K  E  R  E is a collection of subject-predicate-object ( s, p, o ) triples. Link prediction is the task of predicting missing triples in K by answering queries of the form of ( s, p, ?) and (? , p, o ) . This is typically accomplished using knowledge graph embedding (KGE) models. Conventional KGEs assign an embedding vector to each entity and relation in the KG. They model the plausibility of ( s, p, o ) triples via model specific scoring functions f ( e s , e p , e o ) using the subject ( e s ), predicate ( e p ) and object ( e o ) specific embeddings. Once trained, these embeddings are used for downstream tasks such as question answering. Knowledge graph question answering (KGQA) is the task of answering a natural language question using a KG as source of knowledge. The questions can be either simple factual questions that require single fact retrieval (e.g. Which languages are spoken in India? ), or they can be complex questions that require reasoning over multiple facts in the KG (e.g. What are the genres of movies, in which Leonardo DiCaprio was leading actor? ). KGEs can be utilized to perform KGQA when the background KGs are incomplete. the form of their scoring function f ( e s , e p , e o ) . A comprehensive survey of these models, their scoring functions, training regime and link prediction performance can be found in Wang et al. (2017) and Ruffinelli et al. (2020). It is important to note that although these models obtain superior performance in the link prediction task, they suffer from a linear scaling in model size with the number of entities in the KG, and applying them to question answering necessitates separate KGE and QA modules. Compositional KGE models. To combat the linear scaling of the model size with the number of entities in a KG, entity embeddings can be composed of token embeddings. DKRL (Xie et al., 2016b) embeds entities by combining word embeddings of entity descriptions with a CNN encoder, followed by the TransE scoring function. KEPLER (Wang et al., 2021) uses a Transformer-based encoder and combines the typical KGE training objective with a masked language modeling objective. Both of these approaches encode entities and relations separately which limits the transferability of these models to downstream tasks such as question answering. MLMLM (Clouatre et al., 2021) encodes the whole query with a RoBERTa-based model and uses [MASK] tokens to generate predictions. However, it performs significantly worse than atomic KGE models on link prediction on large KGs, and is yet to be applied to downstream text-based tasks. Knowledge Graph Question Answering (KGQA) has been traditionally solved using semantic parsing (Berant et al. 2013; Bast and Haussmann 2015; Das et al. 2021a) where a natural language (NL) question is converted to a symbolic query over the KG. This is problematic for incomplete KGs, where a single missing link can cause the query to fail. Recent work has focused on KGQA over incomplete KGs, which is also the focus of our work. These methods attempt to overcome KG incompleteness using KG embeddings (Huang et al. 2019; Saxena et al. 2020; Sun et al. 2021; Ren et al. 2021). In order to use KGEs for KGQA, these methods first train a KGE model on the background KG, and then integrate the learned entity and relation embeddings into the QA pipeline. This fragmented approach brings several disadvantages; for example Huang et al. (2019)'s method only works for single fact question answering, while EmQL (Sun et al., 2021) requires prior knowledge of the NL question's query structure. EmbedKGQA (Saxena et al., 2020) is capable of multi-hop question answering but is unable to deal with questions involving more than one entity. Hence, these methods are lacking in versatility. LEGO (Ren et al., 2021) can theoretically answer all first order logic based questions but requires multiple dataset dependent components including entity linking, relation pruning and branch pruning modules; here, to obtain versatility, LEGO has sacrificed simplicity. We pose both knowledge graph link prediction and question answering as sequence-to-sequence (seq2seq) tasks. We then train a simple encoder-decoder Transformer  that has the same architecture as T5-small (Raffel et al., 2020) but without the pretrained weights  on these tasks. While training for question answering, we regularize with the link prediction objective. This method, which we call KGT5, results in a scalable KG link prediction model with vastly fewer parameters than conventional KGE models for large KGs. This approach also confers simplicity and versatility to the model, whereby it can be easily adapted to KGQA on any dataset regardless of question complexity. Posing KG link prediction as a seq2seq task requires textual representations of entities and relations, and a verbalization scheme to convert link prediction queries to textual queries; these are detailed in 3.1. The link prediction training procedure is explained in 3.2 and inference in 3.3. The KGQA finetuning and inference pipeline is explained in 3.4. Text mapping. For link prediction we require a one-to-one mapping between an entity/relation and its textual representation. For Wikidata-based KGs, we use canonical mentions of entities and relations as their textual representation, followed by a disambiguation scheme that appends descriptions and unique ids to the name. 4 For datasets used for QA only we do not enforce a one-to-one mapping as, in this case, unnecessary disambiguation can even harm model performance. 5 4 Please see appendix A for details on textual representations. 5 This is because QA systems consider surface forms during evaluation, not entity IDs. For example, it will be better to have the same mention for both the single and album version of a song rather than append a unique number to their mentions. Verbalization. We convert ( s, p, ?) query answering to a sequence-to-sequence task by verbalizing the query ( s, p, ?) to a textual representation. This is similar to the verbalization performed by Petroni et al. (2019), except there is no relation-specific template. For example, given a query (barack obama, born in, ?) , we first obtain the textual mentions of the entity and relation and then verbalize it as 'predict tail: barack obama | born in' . This sequence is input to the model, and output sequence is expected to be the answer to this query, 'united states' , which is the unique mention of entity United States . To train KGT5, we need a set of (input, output) sequences. For each triple ( s, p, o ) in the training graph, we verbalize the queries ( s, p, ?) and (? , p, o ) according to 3.1 to obtain two input sequences. The corresponding output sequences are the text mentions of o and s respectively. KGT5 is trained with teacher forcing (Williams and Zipser, 1989) and cross entropy loss. 6 One thing to note is that unlike standard KGE models, we train without explicit negative sampling . At each step of decoding, the model produces a probability distribution over possible next tokens. While training, this distribution is penalised for 6 More details about training are available in Appendix B Dataset Entities Rels Edges Token. being different from the true' distribution (i.e. a probability of 1 for the true next token, 0 for all other tokens) using cross entropy loss. Hence, this training procedure is most similar to the 1vsAll + CE loss in Ruffinelli et al. (2020), except instead of scoring the true entity against all other entities, we are scoring the true token against all other tokens at each step, and the process is repeated as many times as the length of the tokenized true entity. This avoids the need for many negatives, and is independent of the number of entities. In conventional KGE models, we answer a query ( s, p, ?) by finding the score f ( s, p, o )  o  E , where f is the model-specific scoring function. The entities o are then ranked according to the scores. In our approach, given query ( s, p, ?) , we first 2817 verbalize it (3.1) before feeding it to KGT5. We then sample a fixed number of sequences from the decoder, 7 which are then mapped to their entity ids. 8 By using such a generative model, we are able to approximate (with high confidence) topm model predictions without having to score all entities in the KG, as is done by conventional KGE models. For each decoded entity we assign a score equal to the (log) probability of decoding its sequence. This gives us a set of (entity, score) pairs. To calculate the final ranking metrics comparable to traditional KGE models, we assign a score of  for all entities not encountered during the sampling procedure. A comparison of inference strategy of conventional KGE models and KGT5 is shown in Figure 2. 3.4 KGQA Training and Inference For KGQA, we pretrain the model on the background KG using the link prediction task (3.2). This pretraining strategy is analogous to KGE module training' used in other KGQA works (Sun et al. 2021; Ren et al. 2021). The same model is then finetuned for question answering. Hereby, we employ the same strategy as Roberts et al. (2020): we concatenate a new task prefix ( predict answer: ) with the input question and define the mention string of the answer entity as output. This unified approach allows us to apply KGT5 to any KGQA dataset regardless of question complexity, and without the need for sub-modules such as entity linking. To combat overfitting during QA finetuning (es-pecially on tasks with small KGs) we devise a regularisation scheme: we add link prediction sequences sampled randomly from the background KG to each batch such that a batch consists of an equal number of QA and link prediction sequences. For inference, we use beam search followed by neighbourhood-based reranking (4.3) to obtain the model's prediction which is a single answer. We investigate whether KGT5i.e. a simple seq2seq Transformer modelcan be jointly trained 8 The decoded sequence may or may not be an entity mention. We experimented with constrained decoding (Cao et al., 2021) to force the decoder to output only entity mentions; however, we found this unnecessary since the model almost always outputs an entity mention, and increasing the number of samples was enough to solve the issue. to perform both knowledge graph link prediction as well as question answering. Hereby, we first describe the used datasets (4.1), the baselines we compared to (4.2) and the experimental setup (4.3). The results of our experiments are analysed in 4.4-4.8. Before going into detail, we summarize our key findings: 1. For link prediction on large KGs, the text-based approach of KGT5 reduces model size to comparable KGE models by up to 98% and reaches or outperforms current state-of-the-art. 2. On the task of KGQA over incomplete KGs, our simple seq2seq approach obtains better results than the current state-of-the-art across multiple datasets. 3. KG link prediction training might be more beneficial than language modeling pretraining on knowledge intensive tasks such as KGQA. 4. Although KGT5 is good at generalizing to unseen facts, it is rather poor at memorizing facts. This problem can be alleviated, if needed, by using an ensemble of KGT5 and conventional link prediction or KGQA systems. We evaluate the link prediction capability of KGT5 on Wikidata5M (Wang et al., 2021) and WikiKG90Mv2 (Hu et al., 2021), two of the largest publicly available benchmark KGs. Although KGT5 is designed for large problems, we evaluate on the smaller benchmark KGs FB15k-237 (Toutanova and Chen, 2015), WN18RR (Dettmers et al., 2018) and YAGO3-10 (Dettmers et al., 2018) for comparability. We evaluate the QA capabilities of KGT5 on three large-scale KGQA benchmark datasets: MetaQA (Zhang et al., 2018), WebQuestionsSP (WQSP) (Yih et al., 2016) and ComplexWebQuestions (CWQ) (Talmor and Berant, 2018). Questions in MetaQA span from 1-hop to 3-hop questions requiring path-based reasoning on a KG based on WikiMovies (Miller et al., 2016). WQSP contains both 1-hop and 2-hop path based questions while CWQ contains questions requiring steps such as compositional, conjunctive, comparative and superlative reasoning. Both WQSP and CWQ can be answered using Freebase (Google, 2015) as the background KG. We create subsets of Freebase using the scheme proposed by Ren et al. (2021) which results in KGs that are much smaller than Freebase but can still be used to answer all ques-2818 Model MRR Hits@1 Hits@3 Hits@10 Params TransE (Bordes et al., 2013)  0.253 0.170 0.311 0.392 2,400M DistMult (Yang et al., 2015)  0.253 0.209 0.278 0.334 2,400M SimplE (Kazemi and Poole, 2018)  0.296 0.252 0.317 0.377 2,400M RotatE (Sun et al., 2019b)  0.290 0.234 0.322 0.390 2,400M QuatE (Zhang et al., 2019)  0.276 0.227 0.301 0.359 2,400M ComplEx (Trouillon et al., 2016) $ 0.308 0.255 -0.398 614M KGT5 (Our method) 0.300 0.267 0.318 0.365 60M ComplEx 14-dim  0.201 0.161 0.211 0.275 67M ComplEx 26-dim  0.239 0.187 0.261 0.342 125M KEPLER (Wang et al., 2021)  0.210 0.173 0.224 0.277 125M DKRL (Xie et al., 2016a)  0.160 0.120 0.181 0.229 20M MLMLM (Clouatre et al., 2021)  0.223 0.201 0.232 0.264 355M KGT5-ComplEx Ensemble 0.336 0.286 0.362 0.426 674M Table 2: Link prediction results on Wikidata5M . Following prior work (Sun et al., 2019a) we randomly drop 50% of edges from all KGs to simulate KG incompleteness. This stochasticity causes different works to have different KGs, making it hard to compare results without re-implementing methods. Ren et al. (2021) implemented all comparison methods using their own KG splits which they have not yet published. 9 Our KG split is available along with our implementation 1 and we encourage further studies to use it. We do not re-implement comparison methods but instead report the numbers for our methods and baselines separately. We also report the accuracy obtained by executing the 9 Through private communication with the authors we were able to obtain the same KG split for WQSP. ground truth SPARQL queries (GT query) for test questions. GT query serves as an estimate of the hardness of a KG split and helps us compare model performance across KG splits. Note that for training all models, we only use (NL question, answer entity) pairs no ground truth query information is used for training . Statistics of the KGs used in our experiments are shown in Tab. 1. Statistics of the QA datasets are shown in Tab. 11. For KG completion on Wikidata5M, we compared with several standard KGE models that have been shown to achieve good performance across multiple datasets (Ruffinelli et al., 2020) but with a large number of parameters. Among low-parameter models, we compared to the text based approaches KEPLER (Wang et al., 2021), DKRL (Xie et al., 2016a) and MLMLM (Clouatre et al., 2021). We also consider low-dimensional versions of the state-of-the-art method ComplEx. For the small benchmark KGs we compared with the currently best performing model NBFNet (Zhu et al., 2021). For KGQA, we compared against several methods that have been shown to achieve SOTA on QA over incomplete KGs. These include PullNet (Sun et al., 2019a), EmQL (Sun et al., 2021), EmbedKGQA (Saxena et al., 2020) and LEGO (Ren et al., 2021). Additionally, for the MetaQA datasets, we compared with a relation-path finding baseline, which we call PathPred. This simple 2819 Model CWQ WQSP GT query 25.2 56.9 Pullnet 26.8 (+1.6) 47.4 (-9.5) EmbedKGQA -42.5 (-14.4) LEGO 29.4 (+4.2) 48.5 (-8.4) GT query 24.5 56.9 KGT5 34.5 (+10.0) 50.5 (-6.4) Table 4: Hits@1 (gain vs GT query) on ComplexWebQuestions (CWQ) and WebQuestionsSP (WQSP) datasets in the 50% KG setting. method maps a NL question to a relation path using distantly supervised data obtained from QA pairs in the training set. 10 4.3 Experimental Setup In all our main experiments we used a model with the same architecture as T5-small (  60M parameters) but without the pretrained weights. For to-kenizing sequences, we trained a BPE tokenizer using the SentencePiece (Kudo and Richardson, 2018) library on the verbalised KGs (see Tab. 1 for tokenizer statistics). We used AdaFactor (Shazeer and Stern, 2018) with a learning rate warmup schedule for link prediction training, batch size 320 and 10% dropout. We adopted the same procedure as Roberts et al. (2020) for QA finetuning we halved the batch size and fixed the learning rate to 0.001. All experiments were performed using 4 Nvidia 1080Ti GPUs and models were implemented using the HuggingFace library (Wolf et al., 2019). We performed no dataset-specific hyperparameter tuning for KGT5 and used the same architecture, batch size, dropout and learning rate schedule throughout all experiments. 11 All models were trained until validation accuracy did not significantly increase for 10k steps. 12 For inference, we used sampling size = 500 for link prediction and beam size = 4 for KGQA. We further performed a neighbourhood-based reranking for KGQA: given question q , topic entity from 10 Please see Appendix D for details of PathPred. 11 The vocabulary size for MetaQA is 10k, compared to  30k for other datasets. This was necessary in order to train a BPE tokenizer on such a small KG. 12  5M steps for large KGs (WD5M, W90M),  500k steps for smaller KGs and  30k steps for QA finetuning Model 1-hop 2-hop 3-hop GT query 63.3 45.8 45.3 PullNet 65.1 (+1.8) 52.1 (+6.3) 59.7 (+14.4) EmbedKGQA 70.6 (+7.3) 54.3 (+8.5) 53.5 (+8.2) EmQL 63.8 (+0.5) 47.6 (+1.8) 48.1 (+2.8) LEGO 69.3 (+6.0) 57.8 (+12.0) 63.8 (+18.5) GT query 67.7 48.7 44.4 PathPred 67.7 (+0.0) 48.7 (+0.0) 44.4 (+0.0) KGT5 75.0 (+7.3) 36.2 (-8.2) 64.4 (+20.0) KGT5-PP-Ens. where  is a constant hyperparameter and N ( e ) is the n -hop neighbourhood of the topic entity ( n = 1, 2 or 3). Re-ranking was only done on datasets where topic entity annotation is available as part of test questions. Tab. 3 shows link prediction performance on WikiKG90Mv2, one of the largest benchmark KGs available. Here we compare against TransE, ComplEx and their variants. *-MPNet and *-concat methods use text embeddings as part of entity representations, and operate on the same textual data as KGT5. KGT5 achieves the highest MRR on validation set while having 98% fewer parameters than the next best performing model on the leaderboard. 13 Tab. 2 shows link prediction performance on Wikidata5M, a smaller but better studied KG. We see that KGT5 outperformed all low-parameter count models on all metrics. When compared to the large ComplEx model, there is a drop of 0.008 points in MRR and a gain of 0.012 points in hits@1. We performed a more fine-grained analysis of 13 The authors of OGB-LSC did not provide us with scores on the hidden test set because we used the entity mentions that were provided with the dataset. These entity mentions have now been removed; we provide them for reproducibility on our resource website. model predictions according to the type of query for Wikidata5M (Tab. 13 in the appendix). We found that KGT5 excelled at answering queries which have none or only a few correct answers in the train set; performance dropped when several entities can be correct for a query. This could be due to the nature of sampling: low probability sequences are harder to sample and also harder to rank correctly. Additionally, the limited sampling (3.3) may not even provide the correct answer if there exist more known positives than sampled answers. Based on these observations we created an ensemble of ComplEx and KGT5 which answers queries as follows: if the query does not have answers in the train KG, use KGT5; otherwise use ComplEx (614M). As shown in Tab. 2, the ensemble created by this simple rule outperformed all other single models and achieved the state-of-the-art on Wikidata5M. 14 , 15 Such an ensemble neither achieves the goal of scalability nor versatility but instead serves as an ablation to point out weak spots of KGT5. Tab. 10 in the appendix shows link prediction performance on KGs with  150k entities. Here KGT5 sometimes falls behind the baselines; Transformer models are known to struggle when data is scarce, and this could be the reason for poor performance on these small datasets. Due to the lack of public KG splits, we compared KGQA methods using gain over ground truth query model , which is available for both the comparison methods (from Ren et al. 2021) as well as our methods. 16 Tab. 4 shows hits@1 performance on Freebase-based datasets ComplexWebQuestions and WebQuestionsSP. On both datasets, KGT5 outperformed all baselines. The gains were the largest on ComplexWebQuestions which is the hardest dataset in terms of complexity and KG size. Tab. 5 shows hits@1 performance on the MetaQA datasets. On MetaQA 1and 3-hop, KGT5 was either equal or better than all baselines (in terms of gain). On MetaQA 2-hop however, the performance was significantly worse compared to 14 In this ensemble KGT5 was used to answer 42% of the queries; the rest were answered by ComplEx 15 To the best of our knowledge current state-of-the-art on Wikidata5M is ComplEx published with Kochsiek and Gemulla (2021) presented in Tab. the baselines, and even worse than ground truth querying. We did a more fine-grained analysis of the performance of KGT5 on different question types (Tab. 15-16 in the appendix). We found that KGT5 performance suffered most on questions where the head and answer entity were of the same type (for e.g. actor  movie  actor questions). These question types are absent in the 1-hop and 3-hop datasets. When head and answer entities had different types (for e.g. director  movie  language questions), KGT5 was able to answer them better than GT query. To remedy this issue and create a model more faithful towards the knowledge present in the incomplete KG, we devised an ensemble of KGT5 with the PathPred baseline. The ensemble works as follows: Given a question q , try to answer it using PathPred. If this returns an empty set, use KGT5. This ensemble outperformed all single models on all MetaQA datasets, often by large margins (Tab. 5). Additionally, we performed an ablation to study the effect of neighbourhood reranking on KGQA performance (Tab. 6). We found that reranking gave small but consistent gains on all datasets. Knowledge probing works such as LAMA (Petroni et al., 2019) aim to answer the following question: can models (e.g. BERT) which are pretrained on generic text corpora with a language modeling objective be used as knowledge bases? In our case, the model has been explicitly trained with the link prediction objective , and a knowledge probing experiment would be akin to checking train set performance of link prediction (which is discussed in 4.8). Furthermore, we do not claim that KGT5 is as general purpose as large LMs, or that it contains generic world knowledge. Hence we do not perform knowledge probing experiments on datasets such as T-Rex or Google-RE (Petroni et al., 2019). We analyzed how generic corpora pretraining performed compared to KG link prediction training for the task of KGQA. We compared with T5-small (Raffel et al., 2020), which has the same architecture as KGT5 but pretrained on a mixture of tasks, most notably language modeling on web text. From Tab. 7 we see that KGT5 vastly outperformed T5-small. This is not surprising: the data for KGT5 pretraining was tailored towards the task performedKGQAwhich was not the case for T5-small. However, this shows that it is the link prediction pretraining that is responsible for the excellent KGQA performance of KGT5. Full-KG Question Answering. Tab. 7 shows hits@1 performance in the full KG setting. KGT5 performance only marginally improves when pretrained on full KG compared to 50% KG, and lags far behind both EmbedKGQA (a ComplEx-based method) as well as CBR-KGQA (a semantic parsing method that uses (NL-query, SPARQL-query) parallel data). This indicates that although KGT5 excels at generalizing to unseen facts, it may not be good at memorizing facts. This is further supported by the train set link prediction performance of KGT5 (Tab. 8); although both ComplEx and KGT5 have comparable test MRR, train MRR of ComplEx is significantly better. One possible explanation could be that the reduced model capacity of KGT5  which has only 60M parameters  does not allow it to memorize facts seen during pretraining, leading to poor train MRR and full-KG KGQA performance. Hence we recommend against using KGT5 as a standalone KGQA method, and it should be used only when query-parsing does not yield good results. Use of textual mentions. Since KGT5 requires textual representations for every entity, it cannot be directly applied to all KGs, and is especially unsuitable for KGs that contain CVT nodes as entities (e.g. full Freebase). Also, care must be taken when comparing models that make use of entity names/descriptions with those that do not. In our experiments, we noticed a significant proportion of validation triples in WikiKG90Mv2 required just text processing (eg. <Giovanni Bensi, family name, Bensi> ) and we found a few cases of potential data leakage when definitions are used in WN18RR (eg. <hylidae the amphibian family of tree frogs, hypernym, amphibian family> ). However, from a practical perspective, models which can leverage text data could be more advantageous, and one must assess the pros and cons of a technique before applying it. We have shown that KG link prediction and question answering can be treated as seq2seq tasks and tackled successfully with a single encoder-decoder Transformer model. We did this by training a Transformer model with the same architecture as T5-small on the link prediction task, and then finetuning it on the QA task. This simple but powerful approach, which we call KGT5, performed competitively with the state-of-the-art methods for KG completion on large KGs while using upto 98% fewer parameters. On the task of KGQA on incomplete KGs, we found that our unified approach outperformed baselines on multiple large-scale benchmark datasets. Additionally, we compared language modeling pretraining with KG link prediction training and found that for knowledge-intensive tasks such as KGQA, link prediction training could be more beneficial. One promising direction for future exploration would be to see whether KG link prediction training could be considered as an additional pretraining objective when training large seq2seq models. Furthermore, the impact of model size, and whether larger Transformer models can indeed store more relational information should be investigated.
abstain abstain abstain abstain result result abstain result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain result method objective result abstain result abstain result abstain abstain result result objective other other other other other other other other other other other other other other other other other other other other other method other other other other other other abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain method method abstain abstain method method abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain result method method result result abstain abstain