Distributed Representations of Words and Phrases and their Compositionality
The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.
This paper has 3 main contributions: (i) it extends 
      the skip-gram model of (Mikolov et al, 2013) to speed up training by 
      adopting an objective close to Noise Contrastive Estimation and 
      subsampling frequent words, (ii) it proposes to learn phrase 
      representations and introduces a dataset to evaluate those, (iii) it 
      introduces the concept of additive compositionality. This is a 
      good paper. It is clear and reads well. It is technically correct. The 
      part on speeding up learning (i) justifies the paper. The additive 
      compositionality (iii) is a nice concept to throw in, even if the authors 
      make no attempt to explain how it arises from the optimization problem 
      that training is solving. The part on phrase representation (ii) could be 
      given more emphasis given it could be more controversial. In particular, 
      testing the limits of the proposed technique might make the paper more 
      insightful. ** NCE & Subsampling ** I feel that you 
      should spend more time introducing NCE and highlighting the difference 
      with ICE. From the text, it seems that the notation in (4) is wrong: you 
      sample k sample w according to Pn(w) then you use each individual sample 
      log(sigma<,>) as an unbiased estimator of the expectation but you 
      never manipulate the expectation itself. For subsampling frequent 
      words, it might be worth mentioning that this strategy might be 
      inefficient for less semantic NLP task (e.g. POS tagging or parsing) where 
      the representation of common words might matter more than for the semantic 
      test from [7]. ** Additive compositionality ** This 
      property is similar to the one highlighted in [Tomas Mikolov, Wen-tau Yih, 
      and Geoffrey Zweig, Linguistic Regularities in Continuous SpaceWord 
      Representations] which observed [Koruna-Czech] \simeq [Yen-Japan] 
      You make the remark that these two kind of vectors are also close to 
      currency. This is an interesting observation. It would be nice if one 
      would attempt to explain how the optimization problem solved by maximizing 
      (4) actually yields to such property. This remains puzzling to me and 
      maybe to some other member of the ML community. ** Phrase 
      representation ** You propose to learn a representation for each 
      frequent (according to (6)) phrase. This is a valid proposal and your "Air 
      Canada" example against Socher like models make sense. I do not have a 
      strong argument against your strategy when data are plentiful. However, a 
      strong argument in favor of distributed LM as opposed to n-gram LMs that 
      they allow parameter sharing across similar contexts. This help modeling 
      language because there will always be infrequent phrases that can benefit 
      from similar frequent phrases, e.g. learning that "the (small town name) 
      warriors" is a sports team need to share parameters with other popular 
      teams. It might worse testing the limit of your strategy in terms of 
      number of occurrences as it might go a long way. ** Typos ** 
      - Table 3: samsampling -> subsampling - It might be worth 
      citing Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig, Linguistic 
      Regularities in Continuous SpaceWord Representations which first 
      introduced the unigram fact dataset.