Learning word embeddings efficiently with noise-contrastive estimation
Continuous-valued word embeddings learned by neural language models have recently been shown to capture semantic and syntactic information about words very well, setting performance records on several word similarity tasks. The best results are obtained by learning high-dimensional embeddings from very large quantities of data, which makes scalability of the training method a critical factor. We propose a simple and scalable new approach to learning word embeddings based on training log-bilinear models with noise-contrastive estimation. Our approach is simpler, faster, and produces better results than the current state-of-theart method. We achieve results comparable to the best ones reported, which were obtained on a cluster, using four times less data and more than an order of magnitude less computing time. We also investigate several model types and find that the embeddings learned by the simpler models perform at least as well as those learned by the more complex ones.
Summary: The paper proposes an efficient 
      approach to learning word embeddings which, when evaluated using three 
      publicly available analogy question challenges, outperform other recently 
      proposed systems (controlled for training time and train set size). The 
      major contributions appear to be: i) the use of Adagrad (Adaptive 
      Gradient Descent) to accelerate learning; ii) simplification of the 
      pooling layer (from matrix to vector products) in the LBL algorithm of 
      Mnih and Hinton (2007); iii) elimination of hierarchical softmax layer in 
      favor of Noise-contrastive Estimation (NCE) to speed up training. When 
      combined, these result in a reduction in absolute error of 3-9% 
      relative to the next best reported results for the analogy question 
      challenge sets under consideration when training time and training set 
      size are held constant. Comments: The paper is well 
      written but is incremental, has little novelty and is more of an 
      engineering paper. It might be better fit for an NLP venue like EMNLP, 
      NAACL, ACL etc. Of the three contributions of the paper, I found 
      the last one, most novel, though I wonder how much of the speedup comes 
      from the reduction in time complexity from the use of NCE and how much 
      from Adagrad (which we know to significantly reduce training times needed 
      to reach a given error rate when applied to acoustic modeling or image 
      classification (see the NIPS and ICML papers that Andrew Ng, Jeff Dean, et 
      al. had in 2012)). Also, while the performance improvements on the 
      analogy tasks are certainly substantial, I do wonder how useful this is as 
      an evaluation of the quality of the resultant representations; however, as 
      I don't have a better direct evaluation, I can hardly fault the use of 
      this for a testbed and their model certainly improves on the state of the 
      art here. Additional comments: - The authors don't compare 
      their embeddings against word embeddings learned using Spectral Learning 
      techniques e.g. CCA, PCA (Dhillon et. al. NIPS 11, ICML 12) as being 
      spectral they are known to be fast. I can see that the authors only want 
      to cater to the Deep Learning audience, but then it significantly 
      decreases the appeal of the paper to the broad NIPS audience. - 
      The paper looks like a rebuttal to (Mikolov et. al 2013), as that paper 
      gets mentioned ~20 times, including the abstract which narrows the scope 
      and appeal of the paper. -The authors repeatedly introduce 
      acronyms without explicit definition. NPLM (Neural probabilistic language 
      model) is not defined before first use in the second paragraph of Section 
      2 (though its meaning can be deduced from the section heading).Similarly, 
      CBOW (Continuous Bag of Words) is not defined before first use in the last 
      paragraph of Section 1. Nor is MSR (Microsoft Research) defined before 
      first use in Table 1. Should somewhere in caption or text indicate that 
      this refers to the Microsoft Research analogy question test set from: 
      Mikolov et al. (2013). "Linguistic regularities in continuous space word 
      representations" -Precisely how the embeddings are learned is a 
      bit obscure. In Section 3, where they introduce LBL models, it seems to 
      assume that we have an embedding prior to training the LBL model for 
      prediction. Going back to the Mnih and Hinton papers cited in the 
      references helps a little, but it still is not entirely clear. I assume 
      they jointly learn the embedding matrix with the LBL model? -It 
      would be nice to compare LBL models trained with Adagrad but not NCE to 
      models trained with both to see to what extent the performance 
      improvements come from each.