Understanding Composition of Word Embeddings via Tensor Decomposition
Word embedding is a powerful tool in natural language processing. In this paper we consider the problem of word embedding composition --- given vector representations of two words, compute a vector for the entire phrase. We give a generative model that can capture specific syntactic relations between words. Under our model, we prove that the correlations between three words (measured by their PMI) form a tensor that has an approximate low rank Tucker decomposition. The result of the Tucker decomposition gives the word embeddings as well as a core tensor, which can be used to produce better compositions of the word embeddings. We also complement our theoretical results with experiments that verify our assumptions, and demonstrate the effectiveness of the new composition method.
 The authors suggest a method to create combined low-dimensional representations for combinations of pairs of words which have a specific syntactic relationship (e.g. adjective - noun). Building on the generative word embedding model provided by Arora et al. (2015), their solution uses the core tensor from the Tucker decomposition of a 3-way PMI tensor to generate an additive term, used in the composition of two word embedding vectors. Although the method the authors suggest is a plausible way to explicitly model the relationship between syntactic pairs and to create a combined embedding for them, their presentation does not make this obvious and it takes effort to reach the conclusion above. Unlike Arora*s original work, the assumptions they make on their subject material are not supported enough, as in their lack of explanation of why linear addition of two word embeddings should be a bad idea for composition of the embedding vectors of two syntactically related words, and why the corrective term produced by their method makes this a good idea. Though the title promises a contribution to an understanding of word embedding compositions in general, they barely expound on the broader implications of their idea in representing elements of language through vectors. Their lack of willingness to ground their claims or decisions is even more apparent in two other cases. The authors claim that the Arora*s RAND-WALK model does not capture any syntactic information. This is not true. The results presented by Arora et al. indeed show that RAND-WALK captures syntactic information, albeit to a lesser extent than other popular methods for word embedding (Table 1, Arora et al. 2015). Another unjustified choice by the authors is their choice of weighing the Tensor term (when it is being added to two base embedding vectors) in the phrase similarity experiment. The reason the authors provide for weighing the composition Tensor is the fact that in the unweighted version their model produced a worse performance than the additive composition. One would at least expect an after-the-fact interpretation for the weighted tensor term and what this implies with regard to their method and syntactic embedding compositions in general. Arora*s generative model for word embeddings, on which the current paper is largely based upon, not only make the mathematical relationship among different popular word embedding methods explicit, but also by making and verifying explicit assumptions with regard to properties of the word embeddings created by their model, they are able to explain why low-dimensional embeddings provide superior performance in tasks that implicate semantic relationships as linear algebraic relations. Present work, however interesting with regard to its potential implications, strays away from providing such theoretical insights and suffices with demonstrating limited improvements in empirical tasks.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Understanding Composition of Word Embeddings via Tensor Decomposition
Word embedding is a powerful tool in natural language processing. In this paper we consider the problem of word embedding composition --- given vector representations of two words, compute a vector for the entire phrase. We give a generative model that can capture specific syntactic relations between words. Under our model, we prove that the correlations between three words (measured by their PMI) form a tensor that has an approximate low rank Tucker decomposition. The result of the Tucker decomposition gives the word embeddings as well as a core tensor, which can be used to produce better compositions of the word embeddings. We also complement our theoretical results with experiments that verify our assumptions, and demonstrate the effectiveness of the new composition method.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The authors suggest a method to create combined low-dimensional representations for combinations of pairs of words which have a specific syntactic relationship (e.g. adjective - noun). Building on the generative word embedding model provided by Arora et al. (2015), their solution uses the core tensor from the Tucker decomposition of a 3-way PMI tensor to generate an additive term, used in the composition of two word embedding vectors. Although the method the authors suggest is a plausible way to explicitly model the relationship between syntactic pairs and to create a combined embedding for them, their presentation does not make this obvious and it takes effort to reach the conclusion above. Unlike Arora*s original work, the assumptions they make on their subject material are not supported enough, as in their lack of explanation of why linear addition of two word embeddings should be a bad idea for composition of the embedding vectors of two syntactically related words, and why the corrective term produced by their method makes this a good idea. Though the title promises a contribution to an understanding of word embedding compositions in general, they barely expound on the broader implications of their idea in representing elements of language through vectors. Their lack of willingness to ground their claims or decisions is even more apparent in two other cases. The authors claim that the Arora*s RAND-WALK model does not capture any syntactic information. This is not true. The results presented by Arora et al. indeed show that RAND-WALK captures syntactic information, albeit to a lesser extent than other popular methods for word embedding (Table 1, Arora et al. 2015). Another unjustified choice by the authors is their choice of weighing the Tensor term (when it is being added to two base embedding vectors) in the phrase similarity experiment. The reason the authors provide for weighing the composition Tensor is the fact that in the unweighted version their model produced a worse performance than the additive composition. One would at least expect an after-the-fact interpretation for the weighted tensor term and what this implies with regard to their method and syntactic embedding compositions in general. Arora*s generative model for word embeddings, on which the current paper is largely based upon, not only make the mathematical relationship among different popular word embedding methods explicit, but also by making and verifying explicit assumptions with regard to properties of the word embeddings created by their model, they are able to explain why low-dimensional embeddings provide superior performance in tasks that implicate semantic relationships as linear algebraic relations. Present work, however interesting with regard to its potential implications, strays away from providing such theoretical insights and suffices with demonstrating limited improvements in empirical tasks.
assistant
 The authors suggest a method to create combined low-dimensional representations for combinations of pairs of words which have a specific syntactic relationship (e.g. adjective - noun). Building on the generative word embedding model provided by Arora et al. (2015), their solution uses the core tensor from the Tucker decomposition of a 3-way PMI tensor to generate an additive term, used in the composition of two word embedding vectors. Although the method the authors suggest is a plausible way to explicitly model the relationship between syntactic pairs and to create a combined embedding for them, their presentation does not make this obvious and it takes effort to reach the conclusion above. Unlike Arora*s original work, the assumptions they make on their subject material are not supported enough, as in their lack of explanation of why linear addition of two word embeddings should be a bad idea for composition of the embedding vectors of two syntactically related words, and why the corrective term produced by their method makes this a good idea. Though the title promises a contribution to an understanding of word embedding compositions in general, they barely expound on the broader implications of their idea in representing elements of language through vectors. Their lack of willingness to ground their claims or decisions is even more apparent in two other cases. The authors claim that the Arora*s RAND-WALK model does not capture any syntactic information. This is not true. The results presented by Arora et al. indeed show that RAND-WALK captures syntactic information, albeit to a lesser extent than other popular methods for word embedding (Table 1, Arora et al. 2015). Another unjustified choice by the authors is their choice of weighing the Tensor term (when it is being added to two base embedding vectors) in the phrase similarity experiment. The reason the authors provide for weighing the composition Tensor is the fact that in the unweighted version their model produced a worse performance than the additive composition. One would at least expect an after-the-fact interpretation for the weighted tensor term and what this implies with regard to their method and syntactic embedding compositions in general. Arora*s generative model for word embeddings, on which the current paper is largely based upon, not only make the mathematical relationship among different popular word embedding methods explicit, but also by making and verifying explicit assumptions with regard to properties of the word embeddings created by their model, they are able to explain why low-dimensional embeddings provide superior performance in tasks that implicate semantic relationships as linear algebraic relations. Present work, however interesting with regard to its potential implications, strays away from providing such theoretical insights and suffices with demonstrating limited improvements in empirical tasks.