Translating Embeddings for Modeling Multi-relational Data
We consider the problem of embedding entities and relationships of multirelational data in low-dimensional vector spaces. Our objective is to propose a canonical model which is easy to train, contains a reduced number of parameters and can scale up to very large databases. Hence, we propose TransE, a method which models relationships by interpreting them as translations operating on the low-dimensional embeddings of the entities. Despite its simplicity, this assumption proves to be powerful since extensive experiments show that TransE significantly outperforms state-of-the-art methods in link prediction on two knowledge bases. Besides, it can be successfully trained on a large scale data set with 1M entities, 25k relationships and more than 17M training samples.
This paper presents a model of multi-relational data, 
      such as the set of facts of the form (subject, predicate, object) that 
      comprise the Freebase database. Perhaps the most common use case of such a 
      model is the following: given a triple with either the subject or object 
      missing, rank all possible completions of that triple. The proposed model 
      is energy-based, and thus assigns a score to each possible completion. The 
      lower the score, the more likely that triple represents a true fact that 
      is typically not already present in the database. In addition to a 
      description of the model, the paper presents a training algorithm and 
      reports large-scale experimental results on Freebase and other datasets on 
      the aforementioned completion task. The essense of the model is to 
      embed each entity and relation into a continuous vector space. The 
      dimensions of this space have no direct interpretable meaning except that 
      entities which are semantically similar (in the sense of participating in 
      the same set of facts) will be nearby in the latent space (according to L1 
      or L2), as is standard in matrix factorization. The latent representation 
      of the tail of a triple is modeled as a translation of the latent 
      representation of the head of the triple, with a different learned 
      translation vector for each relation. The quality of a proposed latent 
      embedding is given by an objective function which is essentially the total 
      score of all observed true facts minus the total score of all triples not 
      known to be true, plus a margin term. The parameters of the model (a 
      latent vector for each entity and relation) are learned via a 
      straightforward stochastic gradient descent algorithm from a training set 
      of known facts. Many similar latent-embedding models of 
      multi-relational data exist in the literature. The main novelty in this 
      paper is the restriction that the embedding of tail of a triple be a 
      relationship-dependent translation of the embedding of the subject, rather 
      than an arbitrary linear transformation. I believe this boils down to the 
      assumption that the contribution of each of the features represented by 
      each dimension of the latent space to the energy function are independent 
      of each other, for a given relationship. As the authors points out, this 
      is strictly a special case of of the 'Structured Embeddings' model. SE 
      embeds each relationship into a matrix rather than a vector, allowing for 
      the possiblity that the transformation represented by the relationship 
      between head and tail embeddings is an arbitrary linear transformation. 
      The experimental results use the established metrics for this kind 
      of model and show impressive state-of-the-art performance on standard 
      benchmarks. It seems that the expressivity reduction of this model allows 
      for more efficient training and more resilience to overfitting. As these 
      benchmarks are standard, I have no serious complaint with the methodology 
      here. Some minor comments: It would have been interesting to also see 
      results for a task where a missing relationship has to be predicted given 
      a head and tail. The 'filtered' dataset concept, while a good innovation 
      over the 'raw' dataset, seems superflous to this paper given how close the 
      results are between them and since it was not even used for the FB1M 
      dataset, which is the largest and perhaps most important dataset the 
      authors tested on. The authors claim all experimental results are 
      significant (line 302), but do not name the statistical test being used or 
      justify the assumptions behind the test. They also do not say how the 
      latent dimensionality constant was chosen (268). My main concern 
      with this paper is that is fairly light on novelty and is rather more 
      incremental than a top-tier conference demands. The model is a simple 
      restriction of Structued Embedding, an existing, well-known model: 
      relationship embeddings are replaced with vectors instead of matrices, and 
      so encode translations in latent space rather than arbitrary linear 
      transformations. The learning algorithm is entirely standard stochastic 
      gradient descent. The empirical results are standard benchmarks on 
      standard datasets. While the restriction is shown to be benefial on these 
      tasks, there is somewhat of a lack of in-depth analysis of why. Vague 
      claims in lines 71-93 and on 304 ('appropriate design of the model 
      according to the data') are made that this model is well-suited to the 
      data, but the experiments do not make it clear what is actually helping 
      over SE: Is it that the model is more efficiently trainable? Is it the 
      margin term in the optimization function? Is it that it inherently 
      generalizes better? What exactly is it about these datasets that make 
      translation a good model of relations? The authors present each of these 
      as hypotheses but do not elaborate. As it stands, I feel this paper is 
      presenting a useful idea which could become the core of a good paper, but 
      it needs a deeper analysis of why and when this restriction works. 
      Some minor comments: * The handling of negative data and the role 
      the closed-world assumption is making to the training procedure was 
      somewhat unclear to me in 134-148. What exactly is being assumed about the 
      truth of facts missing from the database? * On 195, the authors state that 
      experiments with the Neural Tensor Model could not be run, but do not give 
      a reason. * On 236, they state that a corrupted triple 'would' be ranked 
      about the test triplet if the corrupted triple is in fact true, but I 
      would think it may or may not be. * The paper only has a single figure, 
      which comes at the end and displays results. I think a figure earlier in 
      the paper presenting a graphical depiction of the model and its output 
      would be useful.