Although word analogy problems have become a standard tool for evaluating word vectors, little is known about why word vectors are so good at solving these problems. In this paper, I attempt to further our understanding of the subject, by developing a simple, but highly accurate generative approach to solve the word analogy problem for the case when all terms involved in the problem are nouns. My approach solves the word analogy problem using a small fraction of the data that is typically used to train word vectors. My results demonstrate the ambiguities associated with learning the relationship between a word pair, and the role of the training dataset in determining the relationship which gets most highlighted. Furthermore, my results show that the ability of a model to accurately solve the word analogy problem may not be indicative of a model's ability to learn the relationship between a word pair the way a human does.
This paper proposes a new method for solving the analogy task, which can potentially provide some insight as to why word2vec recovers word analogies.
In my view, there are three main issues with this paper: (1) the assumptions it makes about our understanding of the analogy phenomenon; (2) the authors' understanding of the proposed method, what it models, and its relation to prior art; (3) the very selective subset of analogies that the author used for evaluation.
ASSUMPTIONS
The author assumes that there the community does not understand why word embedding methods such as word2vec recover analogies. I believe that, in fact, we do have a good understanding of this phenomena. Levy & Goldberg [1] showed that optimizing x for
argmax(cos(x, A - B + C))
is equivalent to optimizing x for
argmax(cos(x, A) - cos(x, B) + cos(x, C))
which means that one can interpret this objective as searching for a word x that is similar to A, similar to C, and dis-similar to B. Linzen [2] cemented this explanation by removing the negative term (-cos(x, B)) and showing that for a wide variety of analogies, the method still works. Drozd et al [3] and Rogers et al [4] also argue that the original datasets used by Mikolov et al were too easy because they focused on encyclopedic facts, and expand these datasets to other non-encyclopedic relations, which are significantly more difficult to solve using simple vector arithmetic.
In other words, we know why word2vec is able to solve analogies via vector arithmetic: because many analogies (like those in Mikolov et al's original dataset) are "gameable", and can be solved by finding a term that is similar to A and similar to C at the same time. For example, if A="woman" and C="king", then x="queen" fits the bill.
METHOD
From what I can understand, the proposed method models the 3-way co-occurrence between A, B, and a context noun (let's call it R). Leveraging the distribution of (X, R, Y) for solving problems in lexical semantics has been studied quite a bit in the past, e.g. Latent Relational Analysis [5] and even Hearst patterns. I think the current description overlooks this major deviation from word2vec and other distributional methods, which only model the 2-way co-occurrence (X, R). This is a much more profound difference than just filtering non-nouns. I think the proposed method should be redescribed in these terms, and compared to other work that modeled 3-way co-occurrences.
EVALUATION DATA
The author evaluates their method on a subset of the original analogy task, which is very limited. I would like to see an evaluation on (A) the original two datasets of Mikolov et al (without non-nouns), and (B) the larger datasets provided by Drozd et al [3] and Rogers et al [4].
In addition, since I think the analogy phenomenon is well understood, I would like to see some demonstration that this method has added value beyond the analogy benchmark.
MISCELLANEOUS COMMENTS
* The author does not state the important fact that when searching for the closest x to A - B + C, the search omits A, B, and C. It is often the case that the result is A or C without this omission.
* The paper is partially de-anonymized (e.g. links and acknowledgements).
* One of the problems with modeling 3-way co-occurrences (as opposed to 2-way co-occurrences) is that they are much sparser. I think this is a more precise explanation for why the currency relation is particularly hard to capture with this method.
[1] http://www.aclweb.org/anthology/W14-1618 
[2] http://anthology.aclweb.org/W16-2503 
[3] http://aclweb.org/anthology/C/C16/C16-1332.pdf 
[4] http://www.aclweb.org/anthology/S17-1017 
[5] https://arxiv.org/pdf/cs/0508053.pdf