We develop a probabilistic extension of state-of-the-art embedding models for link prediction in relational knowledge graphs. Knowledge graphs are collections of relational facts, where each fact states that a certain relation holds between two entities, such as people, places, or objects. We argue that knowledge graphs should be treated within a Bayesian framework because even large knowledge graphs typically contain only few facts per entity, leading effectively to a small data problem where parameter uncertainty matters. We introduce a probabilistic reinterpretation of the DistMult  and ComplEx (Trouillon et al., 2016) models and employ variational inference to estimate a lower bound on the marginal likelihood of the data. We find that the main benefit of the Bayesian approach is that it allows for efficient, gradient based optimization over hyperparameters, which would lead to divergences in a non-Bayesian treatment. Models with such learned hyperparameters improve over the state-of-the-art by a significant margin, as we demonstrate on several benchmarks.
Summary:
The paper presents a probabilistic treatment of knowledge graph embeddings, motivating it in parameter uncertainty estimation and easier hyperparameter optimisation. The authors present density-based DistMult and ComplEx variants, where the posterior parameter distributions for entity and relation embeddings are approximated by diagonal Gaussians q_\gamma. Variational EM is used to infer the variational parameters \gamma as well as the per-entity/per-relation precision (\lambda) hyperparameters. The training process proposed by the authors consists of three phases: (1) pretraining a MAP estimate that’s used as initial means of the posterior approximating Gaussians, (2) variational EM (see above) to find better hyperparameters and (3) another MAP training phase that uses the updated per-entity/per-relation hyperparameters. Finally, experimental results indicate a slight improvement in MRR and HITS@10 across FB and WN datasets.
Originality:
To the best of the reviewer’s knowledge, the presented approach is novel for knowledge graph embeddings.
Discussion:
While the task is relevant, it is unclear how significant the improvements are. While overall, the proposed method seems to indicate small improvement upon a very strong baseline, in some cases it’s very close (96.2 vs 96.4 HITS@10 on WN18, 36.4 vs 36
.5 MRR on FB15K237), or worse (85.8 vs 85.4 MRR on FB15K). 
It is unclear how adequate some details in the experimental setup are for verifying the main hyperparameter optimization claim. In particular, what is “a reasonable choice of hyperparameters” in the first training phase? From figure 3b it seems the initial lambda’s are set proportionately to the frequency, as in the baseline. Are the initial hyperparameter values in EM set the same as the hyperparameter values used for MAP in the reported results? If the claim is to optimize hyperparameters, shouldn’t their initial values be set as uninformed as possible? How do different initial hyperparameter values affect final performance?
The authors claim that the improvement is most notable for entities with fewer training points, however, this is only investigated by using a balanced MRR, where the results are again very close, the same (WN18) or worse (FB15K) for ComplEx. Wouldn’t it be clearer to perform a separate evaluation only considering low-frequency entities to verify this claim?
Parameter uncertainty is not further handled in the paper, the final approach is a point estimate, which discards the uncertainties obtained by VI. Authors mention (last paragraph of Sec. 4) that for a large embedding dimension, bayesian predictions are worse, while for small dimension, they are better. The author’s hypothesis is that a more flexible posterior approximation could solve this issue. No concrete numbers or further analysis are provided.
Clarity and presentation:
The result tables should be merged and formatted better. 
Figures need some work (Fig. 2 looks poorly scaled, all figures should be in vector format for scalability, typos in Fig. 1)
Questions:
- How much additional computation is needed to achieve the reported results?
- Would it be possible to group the entities in bins by frequencies (say 6-10 bins) and assign each bin a hyperparameter, and run grid search over just 6-10 hyperparameters, and then interpolate between the bins to set hyperparameters per entity as a function of its frequency?