Neural Word Embedding as Implicit Matrix Factorization
We analyze skip-gram with negative-sampling (SGNS), a word embedding method introduced by Mikolov et al., and show that it is implicitly factorizing a word-context matrix, whose cells are the pointwise mutual information (PMI) of the respective word and context pairs, shifted by a global constant. We find that another embedding method, NCE, is implicitly factorizing a similar matrix, where each cell is the (shifted) log conditional probability of a word given its context. We show that using a sparse Shifted Positive PMI word-context matrix to represent words improves results on two word similarity tasks and one of two analogy tasks. When dense low-dimensional vectors are preferred, exact factorization with SVD can achieve solutions that are at least as good as SGNS's solutions for word similarity tasks. On analogy questions SGNS remains superior to SVD. We conjecture that this stems from the weighted nature of SGNS's factorization.
The paper investigates the skip-gram model of Mikolov et al., more specifically, the more successful 'negative sampling' version.  The submission draws a connection between the word2vec objective and a form of PMI (pointwise mutual information) measure, and, guided by these analyses, proposes a simple modification of PMI (shifting it by a constant, and replacing negative values with zero). The paper is generally well-written and interesting in many aspects, however I have a few reservations:1)  The connection between the skip gram model and factorizing log P(w | c ) / P(w) is  unsurprising, especially for the soft-max version of the model (also recall the fairly established practice of applying LSA in log space). Still, it is nice to see a more formal look into this connection for the negative-sampling version.2) The authors may miss another role  of the constant k in the context of word2vec (and perhaps more important one?). Unlike, their theoretical analyses, word2vec does not perform marginalization over the entire vocabulary and multiplication of this expectation by k, but rather draws k samples. Thus, another role of k is to produce a more reasonable approximation of the corresponding expectation (~ the softmax partition function). This may be especially important given that word2vec is normally run only for a very small number of iterations over the training set.  It may be interesting to see if what matters is the 'shift' role of k (as argued by the authors) or the approximation / exploration role of k.  This is actually fairly easy to do by introducing 2 constants (m and l) instead of k. Instead of considering k negative samples and summing over them,  one could consider m samples, normalize them (i.e. divide the sum by m) and multiply the result by l.  This would help to disentangle the 2 roles: increasing m would improve the quality of the approximation (without affecting the shift), and increasing l would increase the 'shift' (without affecting the quality of the approximation).  3) Overall, this "shifting-by-log k" extension seems a little ad-hoc (and not so helpful in practice). I am not sure I can really buy its motivation — it seems to be motivated by the fact that the negative sampling objective is ‘using’ the ‘shift’. But it is not really clear that this is really desirable, and it is not clear whether this is really that useful (see the point (2) above), especially in the context of PMI.  On the one hand it results in ignoring low ‘correlations’ but on the other it is doing more than that as the cosine similarity is not shift invariant.  Some more discussion / analyses would be helpful.24 The authors look only into the local objective (i.e. partial derivative of the objective on a specific word-context pair). It may make more sense to look into the global objective, as this objective explicitly incorporates weighting for the individual terms in the factorization problem.  This is mentioned in section 3.2 but I was surprised that it has not been done in 3.1.5) I am really not sure what these small values for SPPMI in Table 1 are telling to us.  SPPMI is different from the objective only for negative terms, so this is probably where the difference is coming from. Is this delta so small because negative terms are infrequent? I am slightly unsure if the results are also affected by the quality of the objective approximation you are using (here I refer to the approximations used to compute the objective, as described in section 5.1). Some additional discussion may be helpful. 6) Overall, section 5.1 is somewhat questionable as SPPMI does not really optimize the objective as no embeddings are really induced. For any objective function, optimizing terms independently (i.e. ignoring any interaction between them) would result in a better 'solution'.  Not really sure what this tells us though. 7) Though it is interesting to see that using PPMI results in comparable or better results on many tasks, this is not a new result as  similar observations were made in Levy and Goldberg (CoNLL 2014, cited).