Expanding Holographic Embeddings for Knowledge Completion
Neural models operating over structured spaces such as knowledge graphs require a continuous embedding of the discrete elements of this space (such as entities) as well as the relationships between them. Relational embeddings with high expressivity, however, have high model complexity, making them computationally difficult to train. We propose a new family of embeddings for knowledge graphs that interpolate between a method with high model complexity and one, namely Holographic embeddings (HOLE), with low dimensionality and high training efficiency. This interpolation, termed HOLEX, is achieved by concatenating several linearly perturbed copies of original HOLE. We formally characterize the number of perturbed copies needed to provably recover the full entity-entity or entity-relation interaction matrix, leveraging ideas from Haar wavelets and compressed sensing. In practice, using just a handful of Haar-based or random perturbation vectors results in a much stronger knowledge completion system. On the Freebase FB15K dataset, HOLEX outperforms originally reported HOLE by 14.7% on the HITS@10 metric, and the current path-based state-of-the-art method, PTransE, by 4% (absolute).
must balance computational complexity against model complexity. Not surprisingly, embedding models capable of capturing rich correlations in relational data often have limited computational scalability. On the other hand, models that can be trained efficiently are often less expressive.
We focus on two compositional operators. The first is the full tensor product [18], which captures correlations between every pair of dimensions of two embedding vectors in R d , by considering their outer product. The resulting quadratic (d 2 ) parameter space makes this impractical even for modestly sized KGs. The second is the circular correlation underlying holographic embedding or HOLE [19], which is inspired by holographic models of associated memory. Notably, HOLE keeps the parameter space linear in d by capturing only the sum along each diagonal of the full tensor product matrix.
Our main contribution is a new compositional operator that combines the strengths of these two models, resulting in much stronger knowledge completion system. Specifically, we propose expanded holographic embeddings or HOLEX, which is a collection of models that interpolates between holographic embeddings and the full tensor product.
The idea is to concatenate l ≥ 1 copies of the HOLE model, each perturbed by a linear vector, allowing various copies to focus on different subspaces of the embedding. HOLEX forms a complete spectrum connecting HOLE with the full tensor product model: it falls back to HOLE when l = 1 and all entries in the perturbation vector are non-zero, and is equivalent to the full tensor product model when l = d, the embedding dimension, and all perturbation vectors are linearly independent.
We consider two families of perturbation vectors, low frequency Haar wavelets [6,10] and random 0/1 vectors. We show that using the former corresponds to considering sums of multiple subsequences of each diagonal line of the full product matrix, in contrast to the original holographic embedding, which sums up the entire diagonal. We find that even just a few low frequency vectors in the Haar matrix are quite effective in practice for HOLEX. When using the complete Haar matrix, the length of the subsequences becomes one, thereby recovering the full tensor product case. Our second family of perturbation vectors, namely random 0/1 vectors, corresponds to randomly sub-selecting half the rows of the tensor product matrix in each copy. This is valuable when the full product matrix is sparse. Specifically, using techniques from compressed sensing, if each diagonal line is dominated by a few large entries (in terms of absolute values), we show that a logarithmic number of random vectors suffice to recover information from these large entries.
To assess its efficacy, we implement HOLEX using the framework of ProjE [23], a recent neural method developed for the Freebase FB15K knowledge completion dataset [3,5], where the 95% confidence interval for statistical significance is 0.3%. In terms of the standard HITS@10 metric, HOLEX using 16 random 0/1 vectors outperforms the original HOLE by 14.7% (absolute), ProjE by 5.7%, and a path-based state-of-the-art method by 4%.
In this paper, the authors introduce a method to interpolate between RESCAL and HolE in the setting of knowledge base completion. Whereas RESCAL models the score of a triple (o, r, s) as   (with x the tensor product), HolE models the score of a triple (o, r, s) as  where conv is a circular convolution.  The authors show that by adding a hadamard product with e_o and a vector c_j, with k independent c_j, we obtain a rank-k linear transform of the tensor product e_o x x_s. Thus, by increasing the rank, the authors interpolate in expressivity between RESCAL and HolE. The next part of the paper introduces 2 specific bases, based on haar wavelets and bernoulli vectors. Finally the authors test their algorithms on FB15K.  I like the main idea to interpolate between HolE and RESCAL, but didn't understand section 3.1 and 3.2. It seems to me that the only criteria for expressivity is the rank of C_d. In this case, why not use a truncated Identity matrix for C_d? Why is there a difference in performance between the Haar basis and the bernoulli basis ?  Regarding the Haar basis, why are we happy to get partial sums of the diagonals ?   As currently explained, I didn't understand the value of theorem 2 and 3. Why would we want to "pin-point" (and what does it mean) non-zero values? is it realistic to have a sparse tensor product matrix ? (embeddings are not asked to be sparse, I don't see where the sparsity in this matrix would come from)  I think this part should be re-written, maybe using figures, and theorem 2/3 should be written in a more formal way. The same comment on formality applies to the proofs of theorem 1 and 2, which are elementary but hard to follow because objects are not properly defined before they are used.   It seems natural to introduce what would happen with a truncated identity matrix. This would amount to only using the k first values in the left-hand side vector, which would lead to a RESCAL with a matrix of size (k,d) pre-multiplied by a projector on the first k vectors. This begs the question, is this method very different from a low-rank RESCAL ? I think this comparison would be much more informative than the current section 3.  Regarding experiments, I find the state of the art badly represented. Notably "Knowledge Base Completion: Baselines Strike Back" by Rudolf Kadlec et al. obtains a hit@10 of 89.3 with DistMult, which is higher than the results reported here for HolEx. Thus, the claim in the abstract about beating the state of the art seems wrong.   ---------------------------- AFTER REBUTTAL ----------------------------  Thank you for your feedback, and for discussing the truncated identity matrix solution. Since this doesn't work as well as the Haar or Bernoulli "basis", I would think the real justification for what this work does lies in fourier space, rather than in a low-rank approximation of the tensor product.  After discussion, I change my decision to a 6. I believe that the theory is not currently satisfying and / or supported by experiments.  - low rank approximation of tensor product doesn't seem to be what makes this work. - no experimental evidence that the tensor product estimated are sparse. (which kills justification based on sensing) (regarding the author's feedback, maybe the type "person" won't be encoded on one dimension, but on a linear combination of dimensions. Since everything is initialized at random, I find it hard to believe that the canonical basis will have any meaning in the final embeddings) - no experimental evidence of a full "interpolation" between HolE and RESCAL (i.e., an experiment that shows HolE and RESCAL performances and how this method's performance get close to either HolE or RESCAL depending on the size of the basis used) With these, this paper would easily go from a 6 to a 9.