Knowledge graph embedding (KGE) models learn algebraic representations of the entities and relations in a knowledge graph. A vast number of KGE techniques for multi-relational link prediction has been proposed in the recent literature, often with state-of-the-art performance. These approaches differ along a number of dimensions, including different model architectures, different training strategies, and different approaches to hyperparameter optimization. In this paper, we take a step back and aim to summarize and quantify empirically the impact of each of these dimensions on model performance. We report on the results of an extensive experimental study with popular model architectures and training strategies across a wide range of hyperparameter settings. We found that when trained appropriately, the relative performance differences between various model architectures often shrinks and sometimes even reverses when compared to prior results. For example, RESCAL (Nickel et al., 2011), one of the first KGE models, showed strong performance when trained with state-of-the-art techniques; it was competitive to or outperformed more recent architectures. We also found that good (and often superior to prior studies) model configurations can be found by exploring relatively few random samples from a large hyperparameter space. Our results suggest that many of the more advanced architectures and techniques proposed in the literature should be revisited to reassess their individual benefits.
Summary
========
The paper conducts a thorough analysis of existing models for constructing knowledge graph embeddings. It focuses on attempting to remove confounding aspects of model features and training regime, in order to better assess the merits of KGE models. The paper describes the reimplementation of five different KGE models, re-trained with a common training framework which conducts hyperparameter exploration. The results show surprising insights, e.g., demonstrating that a system from 2011, despite being the earliest of the KGE models analyzed, demonstrates competitive results over a more recent (2017) published model.
Overall Comments
===============
The paper, and the described software release specifically, represent a solid contribution to the area of knowledge graph embeddings. I agree with the basic premise of this paper’s analysis: in order to accelerate research in a maturing field (like knowledge graphs), it is important to be able to properly compare with older systems, removing artifacts that are due to general improvements in training and optimization techniques, from modeling specific changes. The report of the strong results from the RESCAL system, along with others, drive the point through. Furthermore, the paper is well-written and easy to follow, and should become a good reference for future works on KGEs.
Detailed comments
===============
Below are some detailed comments about specific parts of the paper, in order of importance:
1. The paper mentions disregarding “monolithic” models in the current analysis, primarily due to the expensive training of these models. It may, however, be the case that the future state-of-the-art models will be larger and slower to train (and, perhaps, of the monolithic type). Are there any limitations to the proposed experimental framework that would prevent running monolithic/large models?
2. Regarding the item above, if one were to look at the training curves for the exploration of the current 5 KGE models, is it possible that verify winning hyperparameter configurations earlier than the full training is complete. In my experience, it is often the case that with fewer than 1/10th steps of full training (well before convergence), it is possible to compare model configurations (relatively). For example, “Population-base training” (https://arxiv.org/abs/1711.09846, https://arxiv.org/abs/1902.01894) is one framework where fewer training steps are used to quickly learn good hyperparameter configurations. I’m wondering whether the KGE hyperparameter exploration training curves display similar early trends. Could a shortened training procedure produce sufficient information for learning good parameters, and potentially deal with larger/slower models?  In addition: would adopting population-based training be applicable to the proposed framework?
3. In Section 3.2, “Limitations”, there is a surprising comment that performance can be improved with further hyperparameter tuning. It is not clear how the authors found the configurations that produced the improved results. It would be helpful to clarify why the hyperparameter exploration proposed in the paper did not discover these improved configurations. Were the improved configurations outside of the range of considered values? Or would the exploration require more points to find the improved configuration?
4. In Section 3.3 “Best configurations (quasi-random search)”, specifically Table 3, the paper presents an ablation of independent hyperparameters, over the best configuration for each of the 5 models. This is a very interesting section. One further suggestion, however, is whether the paper could include the performance of each of the models on the _average_ best configuration. Although the paper describes losses for switching individual parameters to their second best values, it is unlikely that the losses are cumulative. So, for example, if we can take the average/majority best value for each parameter (embedding size = 512, batch size = 1024, training type = 1vsall, loss = CE, etc.), and collect results for that configuration. I think it would be interesting to know the difference between a model trained on a “collectively known good” set of parameters vs. a model and task specific tuned set of parameters.
5. In Section 2, “Evaluation”, HITS@k is not formally defined. Unfortunately, I have encountered slight variants of this metrics (e.g: (1) given a SINGLE correct label, HITS@k is the average rate of the label being present in the top k scored results, or (2) given ALL possible correct labels, HITS@k is the percentage of correct labels present within the top k scored results, etc.). It would be nice to precisely describe HITS@k in this work.
6. Caption for Table 2 does not contain a description for the “Recent” super-column.
7. In Section 3.3 “Best configuration (quasi-random search)” Space missing at “... Tables 6 and 7(in …”, between 7 and (.