Reasoning With Neural Tensor Networks for Knowledge Base Completion
Knowledge bases are an important resource for question answering and other tasks but often suffer from incompleteness and lack of ability to reason over their discrete entities and relationships. In this paper we introduce an expressive neural tensor network suitable for reasoning over relationships between two entities. Previous work represented entities as either discrete atomic units or with a single entity vector representation. We show that performance can be improved when entities are represented as an average of their constituting word vectors. This allows sharing of statistical strength between, for instance, facts involving the "Sumatran tiger" and "Bengal tiger." Lastly, we demonstrate that all models improve when these word vectors are initialized with vectors learned from unsupervised large corpora. We assess the model by considering the problem of predicting additional true relations between entities given a subset of the knowledge base. Our model outperforms previous models and can classify unseen relationships in WordNet and FreeBase with an accuracy of 86.2% and 90.0%, respectively.
This paper describes a model for multi-relational data 
      specialized to knowledge bases and other datasets with entities and 
      relationships labeled by words. The model is a straightforward 
      generalization of other models from the literature. The model uses entity 
      vectors that are the average of word vectors in the entity name. The score 
      of a triple (e1, R, e2) is based on three vector-valued terms fed through 
      an elementwise nonlinearity and then dotted with a relation-dependent 
      vector. One term is a bias vector that is a function of R. One term has a 
      weight matrix dependent on R applied to the concatenation of the vectors 
      for e1 and e2. The final term is a bilinear tensor product using a 
      relation-dependent set of weights and the two entity vectors. To justify 
      the use of this model, the paper presents results on classifying (entity, 
      relation, entity) triples as positive (from a knowledge base) or negative 
      (synthetic triples unlikely to exist in the knowledge base). I 
      don't see the advantage of the non-linearity in equation 1. I expect the 
      model would be mostly unchanged in practice if f was removed from equation 
      1 (and u_R was folded into other parameters where appropriate). When all 
      the embedding vectors are learned and unconstrained and there are so many 
      arbitrary relation-dependent weights the nonlinearity adds very little 
      power to the model since the mapping from entity id to entity vector is 
      arbitrary. Although the model in the paper is powerful and 
      potentially interesting, the experimental evaluation has a few serious 
      weaknesses. The full train/dev/test splitting protocal needs to be 
      described precisely. If (x,r,y) is in the test set, can (x,r, z != y) be 
      in the training set? Or can only (u!=x, r, v!=y) and (x, q != r, y) be in 
      the training set? The most powerful models should do best when they have 
      lots of data and have seen entities in the test set many times and seen 
      relations in the test set many times. Therefore the most interesting 
      experiments for the model described in the paper are ones that test the 
      generalization ability of the model from limited training data. Along 
      these lines, why are results on all relations in Freebase relegated to the 
      supplementary material and so few relations used in the main paper? A 
      dozen relations is simply not enough to do any meaningful evaluation of 
      this model, especially since it has so many relation-specific parameters. 
      And how many entities are there for the full experiments? If anything, 
      remove the experiments that focus on the small subset of Freebase and if 
      necessary break down some of the results based on how much relevant 
      training data was used for the particular test cases. The model needs to 
      be tested in the regime where it has very little data for the particular 
      relation being tested and in the regime where it has very little data for 
      the entities; that is where simpler special cases of the model may shine. 
      My own view is that a simpler model will probably work better in most 
      situations although it is possible we as a community have not yet settled 
      on the best simple model. There are many obvious extensions to models in 
      the literature that add modeling power and when a paper introduces one it 
      must demonstrate that the overfitting risks can be controlled and that the 
      new model is actually really useful. Figure 3 demonstrates what is 
      probably an overfitting effect for the NTN model on WordNet since the NTN 
      does so poorly when trained without word vectors. This supports my concern 
      that overfitting is a larger than usual risk with this model and that the 
      best evaluation would employ data with relations that have few training 
      cases associated with them. The explanation of how negative cases 
      were generated for the classification task needs more detail. Can the 
      negative triple (Pablo Picasso, nationality, George Washington) get 
      generated? I don't understand why a classification task was used 
      instead of a ranking or information retrieval style task as in [7]. Using 
      a classification task seems non-standard and makes it harder to compare to 
      other papers. Also for open domain knowledge base completion, a ranking 
      task makes more sense. I recommend removing the average column of 
      table 2 since it adds nothing and averaging results across tasks should be 
      avoided.