UNDERSTANDING COMPOSITION OF WORD EMBED- DINGS VIA TENSOR DECOMPOSITION
Word embedding is a powerful tool in natural language processing. In this paper we consider the problem of word embedding composition -given vector representations of two words, compute a vector for the entire phrase. We give a generative model that can capture specific syntactic relations between words. Under our model, we prove that the correlations between three words (measured by their PMI) form a tensor that has an approximate low rank Tucker decomposition. The result of the Tucker decomposition gives the word embeddings as well as a core tensor, which can be used to produce better compositions of the word embeddings. We also complement our theoretical results with experiments that verify our assumptions, and demonstrate the effectiveness of the new composition method.
The authors consider the use of tensor approximations to more accurately capture syntactical aspects of compositionality for word embeddings. Given two words a and b, when your goal is to find a word whose meaning is roughly that of the phrase (a,b), a standard approach to to find the word whose embedding is close to the sum of the embeddings, a + b. The authors point out that others have observed that this form of compositionality does not leverage any information on the syntax of the pair (a,b), and the propose using a tensor contraction to model an additional multiplicative interaction between a and b, so they propose finding the word whose embedding is closest to a + b + T*a*b, where T is a tensor, and T*a*b denotes the vector obtained by contracting a and b with T. They test this idea specifically on the use-case where (a,b) is an adjective,noun pair, and show that their form of compositionality outperforms weighted versions of additive compositionality in terms of spearman and pearson correlation with human judgements. In their model, the word embeddings are learned separately, then the tensor T is learned by minimizing an objective whose goal is to minimize the error in predicting observed trigram statistics. The specific objective comes from a nontrivial tensorial extension of the original matricial RAND-WALK model for learning word embeddings.
The topic is fitting with ICLR, and some attendees will find the results interesting. As in the original RAND-WALK paper, the theory is interesting, but not the main attraction, as it relies on strong generative modeling assumptions that essentially bake in the desired results. The main appeal is the idea of using T to model syntactic interactions, and the algorithm for learning T. Given that the main attraction of the paper is the potential for more performant word embeddings, I do not believe the work will have wide appeal to ICLR attendees, because no evidence is provided that the features from the learned tensor, say [a, b, T*a*b], are more useful in downstream applications than [a,b] (one experiment in sentiment analysis is tried in the supplementary material with no compelling difference shown).
Pros:
- theoretical justification is given for their assumption that the higher-order interactions can be modeled by a tensor
- the tensor model does deliver some improvement over linear composition on noun-adjective pairs when measured against human judgement
Cons:
- no downstream applications are given which show that these higher order interactions can be useful for downstream tasks.
- the higher-order features T*a*b are useful only when a is noun and b is an adjective: why not investigate using T to model higher-order interaction for all (a,b) pairs regardless of the syntactic relationships between a and b?
- comparison should be made to the linear composition method in the Arora, Liang, Ma ICLR 2017 paper 
Some additional citations: 
- the above-mentioned ICLR paper provides a performant alternative to unweighted linear composition
- the 2017 Gittens, Achlioptas, Drineas ACL paper provides theory on the linear composition of some word embeddings