A closer look at the word analogy problem
Although word analogy problems have become a standard tool for evaluating word vectors, little is known about why word vectors are so good at solving these problems. In this paper, I attempt to further our understanding of the subject, by developing a simple, but highly accurate generative approach to solve the word analogy problem for the case when all terms involved in the problem are nouns. My results demonstrate the ambiguities associated with learning the relationship between a word pair, and the role of the training dataset in determining the relationship which gets most highlighted. Furthermore, my results show that the ability of a model to accurately solve the word analogy problem may not be indicative of a model’s ability to learn the relationship between a word pair the way a human does.
 This paper presents, and analyzes, a method for learning word relationships based on co-occurrence. In the method, relationships between pairs of words (A, B) are represented by the terms that tend to occur around co-mentions of A and B in text. The paper shows the start of some interesting ideas, but needs revisions and much more extensive experiments. On the plus side, the method proposed here does perform relatively well (Table 1) and probably merits further investigation. The experiments in Table 1 can only be considered preliminary, however. They only evaluate over a small number of relationships (three) -- looking at 20 or so different relationships would greatly improve confidence in the conclusions. Beyond Table 1 the paper makes a number of claims that are not supported or weakly supported (the paper uses only a handful of examples as evidence). An attempt to explain what Word2Vec is doing should be made with careful experiments over many relations and hundreds of examples, whereas this paper presents only a handful of examples for most of its claims. Further, whether the behavior of the proposed algorithm actually reflects what word2vec is doing is left as a significant open question. I appreciate the clarity of Assumption 1 and Proposition 1, but ultimately this formalism is not used and because Assumption 1 about which nouns are "semantically related" to which other nouns attempts to trivialize a complex notion (semantics) and is clearly way too strong -- the paper would be better off without it. Also Assumption 1 does not actually claim what the text says it claims (the text says words outside the window are *not* semantically related, but the assumption does not actually say this) and furthermore is soon discarded and only the frequency of noun occurrences around co-mentions is used. I think the description of the algorithm could be retained without including Assumption 1. minor: References to numbered algorithms or assumptions should be capitalized in the text. what the introduction means about the "dynamics" of the vector equation is a little unclear A submission shouldn*t have acknowledgments, and in particular with names that undermine anonymity MLE has a particular technical meaning that is not utilized here, I would just refer to the most frequent words as "most related nouns" or similar In Table 1, are the "same dataset" results with w2v for the nouns-only corpus, or with all the other words? The argument made assuming a perfect Zipf distribution (with exponent equal to one) should be made with data. will likely by observed -> will likely be observed lions:dolphins probably ends up that way because of "sea lions" Table 4 caption: frequencies -> currencies Table 2 -- claim is that improvements from k=10 to k=20 are *nominal* but they look non-negligible to me I did not understand how POS lying in the same subspace means that Vec(D) has to be in the span of Vecs A-C.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
A closer look at the word analogy problem
Although word analogy problems have become a standard tool for evaluating word vectors, little is known about why word vectors are so good at solving these problems. In this paper, I attempt to further our understanding of the subject, by developing a simple, but highly accurate generative approach to solve the word analogy problem for the case when all terms involved in the problem are nouns. My results demonstrate the ambiguities associated with learning the relationship between a word pair, and the role of the training dataset in determining the relationship which gets most highlighted. Furthermore, my results show that the ability of a model to accurately solve the word analogy problem may not be indicative of a model’s ability to learn the relationship between a word pair the way a human does.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper presents, and analyzes, a method for learning word relationships based on co-occurrence. In the method, relationships between pairs of words (A, B) are represented by the terms that tend to occur around co-mentions of A and B in text. The paper shows the start of some interesting ideas, but needs revisions and much more extensive experiments. On the plus side, the method proposed here does perform relatively well (Table 1) and probably merits further investigation. The experiments in Table 1 can only be considered preliminary, however. They only evaluate over a small number of relationships (three) -- looking at 20 or so different relationships would greatly improve confidence in the conclusions. Beyond Table 1 the paper makes a number of claims that are not supported or weakly supported (the paper uses only a handful of examples as evidence). An attempt to explain what Word2Vec is doing should be made with careful experiments over many relations and hundreds of examples, whereas this paper presents only a handful of examples for most of its claims. Further, whether the behavior of the proposed algorithm actually reflects what word2vec is doing is left as a significant open question. I appreciate the clarity of Assumption 1 and Proposition 1, but ultimately this formalism is not used and because Assumption 1 about which nouns are "semantically related" to which other nouns attempts to trivialize a complex notion (semantics) and is clearly way too strong -- the paper would be better off without it. Also Assumption 1 does not actually claim what the text says it claims (the text says words outside the window are *not* semantically related, but the assumption does not actually say this) and furthermore is soon discarded and only the frequency of noun occurrences around co-mentions is used. I think the description of the algorithm could be retained without including Assumption 1. minor: References to numbered algorithms or assumptions should be capitalized in the text. what the introduction means about the "dynamics" of the vector equation is a little unclear A submission shouldn*t have acknowledgments, and in particular with names that undermine anonymity MLE has a particular technical meaning that is not utilized here, I would just refer to the most frequent words as "most related nouns" or similar In Table 1, are the "same dataset" results with w2v for the nouns-only corpus, or with all the other words? The argument made assuming a perfect Zipf distribution (with exponent equal to one) should be made with data. will likely by observed -> will likely be observed lions:dolphins probably ends up that way because of "sea lions" Table 4 caption: frequencies -> currencies Table 2 -- claim is that improvements from k=10 to k=20 are *nominal* but they look non-negligible to me I did not understand how POS lying in the same subspace means that Vec(D) has to be in the span of Vecs A-C.
assistant
 This paper presents, and analyzes, a method for learning word relationships based on co-occurrence. In the method, relationships between pairs of words (A, B) are represented by the terms that tend to occur around co-mentions of A and B in text. The paper shows the start of some interesting ideas, but needs revisions and much more extensive experiments. On the plus side, the method proposed here does perform relatively well (Table 1) and probably merits further investigation. The experiments in Table 1 can only be considered preliminary, however. They only evaluate over a small number of relationships (three) -- looking at 20 or so different relationships would greatly improve confidence in the conclusions. Beyond Table 1 the paper makes a number of claims that are not supported or weakly supported (the paper uses only a handful of examples as evidence). An attempt to explain what Word2Vec is doing should be made with careful experiments over many relations and hundreds of examples, whereas this paper presents only a handful of examples for most of its claims. Further, whether the behavior of the proposed algorithm actually reflects what word2vec is doing is left as a significant open question. I appreciate the clarity of Assumption 1 and Proposition 1, but ultimately this formalism is not used and because Assumption 1 about which nouns are "semantically related" to which other nouns attempts to trivialize a complex notion (semantics) and is clearly way too strong -- the paper would be better off without it. Also Assumption 1 does not actually claim what the text says it claims (the text says words outside the window are *not* semantically related, but the assumption does not actually say this) and furthermore is soon discarded and only the frequency of noun occurrences around co-mentions is used. I think the description of the algorithm could be retained without including Assumption 1. minor: References to numbered algorithms or assumptions should be capitalized in the text. what the introduction means about the "dynamics" of the vector equation is a little unclear A submission shouldn*t have acknowledgments, and in particular with names that undermine anonymity MLE has a particular technical meaning that is not utilized here, I would just refer to the most frequent words as "most related nouns" or similar In Table 1, are the "same dataset" results with w2v for the nouns-only corpus, or with all the other words? The argument made assuming a perfect Zipf distribution (with exponent equal to one) should be made with data. will likely by observed -> will likely be observed lions:dolphins probably ends up that way because of "sea lions" Table 4 caption: frequencies -> currencies Table 2 -- claim is that improvements from k=10 to k=20 are *nominal* but they look non-negligible to me I did not understand how POS lying in the same subspace means that Vec(D) has to be in the span of Vecs A-C.