SimplE Embedding for Link Prediction in Knowledge Graphs
Knowledge graphs contain knowledge about the world and provide a structured representation of this knowledge. Current knowledge graphs contain only a small subset of what is true in the world. Link prediction approaches aim at predicting new links for a knowledge graph given the existing links among the entities. Tensor factorization approaches have proved promising for such link prediction problems. Proposed in 1927, Canonical Polyadic (CP) decomposition is among the first tensor factorization approaches. CP generally performs poorly for link prediction as it learns two independent embedding vectors for each entity, whereas they are really tied. We present a simple enhancement of CP (which we call SimplE) to allow the two embeddings of each entity to be learned dependently. The complexity of SimplE grows linearly with the size of embeddings. The embeddings learned through SimplE are interpretable, and certain types of background knowledge can be incorporated into these embeddings through weight tying. We prove SimplE is fully expressive and derive a bound on the size of its embeddings for full expressivity. We show empirically that, despite its simplicity, SimplE outperforms several state-of-the-art tensor factorization techniques. SimplE's code is available on GitHub at https://github.com/Mehran-k/SimplE.
# Summary  This paper considers link prediction in knowledge graphs: give a triple (head, relation, tail) ((h,r,t)) the task is to determine whether the triple represents a correct relation. The contribution is a tensor factorization method that represents each object and relation as some vector. The main novelty relative to previous approaches is that each relation r is represented by two embedding vectors: one for r, and one for r^-1. The motivation is that should allow objects that appear as both heads and tails to more easily learn jointly from both roles.    # Strengths  The individual components of the paper are mostly well written and easy to understand. The core idea is simple and intuitive, and the authors do a reasonable job of motivating it.  # Criticisms  1. It appears that some of the statements made in the paper are incorrect.  (i) the authors claim (comparing TransE and FTransE) that a model which assigns relations based on v(t) - v(r) is less expressive than one that assigns relations based only on (v(t)-v(r))/||v(t) - v(r)||. Consider the case where the only possible embeddings are (0,0), (1,1) and (2,2). The reversal of this inclusion makes it unclear whether Corollary 1 holds---it's at least not obvious to me.  (ii) At line 135, the sentence beginning "Models based on..." is not obviously true. For neural net universality to (trivially) hold all of the (h,r,t) would need to be input to some neural net---but that doesn't match the architectures described in "Deep Learning Approaches"  (iii) at lines 284-285, the loss function on minibatches is given and described as a negative log likelihood. Although the expression could be considered a negative llhd for some probabilistic model, such a model has not been previously introduced. (I'm actually unsure whether the expression literally corresponds to any distribution since there's no obvious normalization term)  2. I'm not sure about the significance. In particular, does it matter that SimplE is moderately more computationally efficient than ComplEx? What does that mean for training time? For prediction time?  3. The structuring of the paper is a rather odd. My two biggest complaints here are (i) the main idea of the paper isn't given until lines 200-205 on page 5, but this could have easily been explained in the introduction (along with an overview of the full method) (ii) the loss function and training procedure are described in "Learning SimplE Models" (starting at line 275) as a subsection of "Experiments and Results". This material deals with specifying the model and should appear earlier, more prominently, and certainly not as a subsection of experiments.  4. I find the footnote on page 7 promising to release the code after the paper is submitted to be somewhat troubling. Why not include it with the submission? Although this is not strictly required, it's certainly better practice.  5. The major motivation for SimplE is that it allows the head and tail vectors for each object to be learned dependently. I don't see how this is reflected in the experiments.    ### Update  The authors mostly did a good job addressing my concerns (especially given the very limited space!). I still think 1.i is a problem---the fact that TransE takes embedding vectors to be points and FTransE takes embedding vectors to be lines means that TransE is more expressive.  I'm also still not convinced that 'fewer multiplications' is a meaningful advantage, but the paper is nice even if this is meaningless, so whatever.