Pre-trained word vectors are ubiquitous in Natural Language Processing applications. In this paper, we show how training word embeddings jointly with bigram and even trigram embeddings, results in improved unigram embeddings. We claim that training word embeddings along with higher n-gram embeddings helps in the removal of the contextual information from the unigrams, resulting in better stand-alone word embeddings. We empirically show the validity of our hypothesis by outperforming other competing word representation models by a significant margin on a wide variety of tasks. We make our models publicly available. Distributed word representations are essential building blocks of modern NLP systems. Used as features in downstream applications, they often enhance generalization of models trained on a limited amount of data. They do so by capturing relevant distributional information about words from large volumes of unlabeled text. Efficient methods to learn word vectors have been introduced in the past, most of them based on the distributional hypothesis of Harris (1954); Firth (1957): a word is characterized by the company it keeps . While a standard approach relies on global corpus statistics (Pennington et al., 2014) formulated as a matrix factorization using mean square reconstruction loss, other widely used methods are the bilinear word2vec architectures introduced by Mikolov et al. (2013a): While skip-gram aims to predict nearby words from a given word, CBOW predicts a target word from its set of context words. augmenting word-context pairs with sub-word information in the form of character n-grams (Bo-janowski et al., 2017), especially for morphologically rich languages. Nevertheless, to the best of our knowledge, no method has been introduced leveraging collocations of words with higher order word n-grams such as bigrams or trigrams as well as character n-grams together. In this paper, we show how using higher order word n-grams along with unigrams during training can significantly improve the quality of obtained word embeddings. The addition furthermore helps to disentangle contextual information present in the training data from the unigrams and results in overall better distributed word representations. To validate our claim, we train two modifica-tions of CBOW augmented with word-n-gram information during training. One is a recent sentence embedding method, Sent2Vec (Pagliardini et al., 2018), which we repurpose to obtain word vectors. The second method we propose is a modification of CBOW enriched with character ngram information (Bojanowski et al., 2017) that we again augment with word n-gram information. In both cases, we compare the resulting vectors with the most widely used word embedding methods on word similarity and analogy tasks and show significant quality improvements. The code used to train the models presented in this paper as well as the models themselves are made available to the public 1 . Before introducing our model, we recapitulate fundamental existing word embeddings methods. CBOW and skip-gram models . Continuous bag-of-words (CBOW) and skip-gram models are 1 publicly available on http://github.com/ epfml/sent2vec standard log-bilinear models for obtaining word embeddings based on word-context pair information (Mikolov et al., 2013a). Context here refers to a symmetric window centered on the target word w t , containing the surrounding tokens at a distance less than some window size ws : C t = { w k | k  [ t  ws , t + ws ] } . The CBOW model tries to predict the target word given its context, maximizing the likelihood (cid:81) Tt =1 p ( w t | C t ) , whereas skip-gram learns by predicting the context for a given target word maximizing (cid:81) Tt =1 p ( C t | w t ) . To model those probabilities, a softmax activation is used on top of the inner product between a target vector u w t and its context vector 1 | C t | (cid:80) w  C t v w . To overcome the computational bottleneck of the softmax for large vocabulary, negative sampling or noise contrastive estimation are well-established (Mikolov et al., 2013b), with the idea of employing simpler pairwise binary classifier loss functions to differentiate between the valid context C t and fake contexts NC t sampled at random. While generating target-context pairs, both CBOW and skip-gram also use input word subsampling, discarding higher-frequency words with higher probability during training, in order to prevent the model from overfitting the most frequent tokens. Standard CBOW also uses a dynamic context window size: for each subsampled target word w , the size of its associated context window is sampled uniformly between 1 and ws (Mikolov et al., 2013b). Adding character n-grams . Bojanowski et al. (2017) have augmented CBOW and skip-gram by adding character n-grams to the context representations. Word vectors are expressed as the sum of its unigram and average of its character n-gram embeddings W w : v := v w + 1 | W w | (cid:88) c  W w v c Character n-grams are hashed to an index in the embedding matrix . The training remains the same as for CBOW and skip-gram. This approach greatly improves the performances of CBOW and skip-gram on morpho-syntactic tasks. For the rest of the paper, we will refer to the CBOW and skip-gram methods enriched with subword-information as CBOW-char and skip-gram-char respectively. GloVe . Instead of training online on local window contexts, GloVe vectors (Penning-ton et al., 2014) are trained using global co-occurrence statistics by factorizing the word-context co-occurrence matrix. Ngram2vec . In order to leverage the performance of word vectors, training of word vectors using the skip-gram objective function with negative sampling is augmented with n-gram co-occurrence information (Zhao et al., 2017). CBOW-char with word n-grams . We propose to augment CBOW-char to additionally use word ngram context vectors (in addition to char n-grams and the context word itself). More precisely, during training, the context vector for a given word w t is given by the average of all word-n-grams N t , all char-n-grams, and all unigrams in the span of the current context window C t : v := (cid:80) w  C t v w + (cid:80) n  N t v n + (cid:80) w  C t (cid:80) c  W w v c | C t | + | N t | + (cid:80) w  C t | W w | (1) For a given sentence, we apply input subsampling and a sliding context window as for standard CBOW. In addition, we keep the mapping from the subsampled sentence to the original sentence for the purpose of extracting word n-grams from the original sequence of words, within the span of the context window. Word n-grams are added to the context using the hashing trick in the same way char-n-grams are handled. We use two different hashing index ranges to ensure there is no collision between char n-gram and word n-gram representations. Sent2Vec for word embeddings . Initially implemented for sentence embeddings, Sent2Vec (Pagliardini et al., 2018) can be seen as a derivative of word2vec's CBOW. The key differences between CBOW and Sent2Vec are the removal of the input subsampling, considering the entire sentence as context, as well as the addition of word-n-grams. Here, word and n-grams embeddings from an entire sentence are averaged to form the corresponding sentence (context) embedding. For both proposed CBOW-char and Sent2Vec models, we employ dropout on word n-grams during training. For both models, word embeddings are obtained by simply discarding the higher order n-gram embeddings after training. We train all competing models on a wikipedia dump of 69 million sentences containing 1.7 billion words, following (Pagliardini et al., 2018). Sentences are tokenized using the Stanford NLP library (Manning et al., 2014). All algorithms are implemented using a modified version of the fast-text (Bojanowski et al., 2017; Joulin et al., 2017) and sent2vec (Pagliardini et al., 2018) libraries respectively. Detailed training hyperparameters for all models included in the comparison are provided in Table 3 in the supplementary material. During training, we save models checkpoints at 20 equidistant intervals and found out that the best performance for CBOW models occurs around 60  80% of the total training. As a result, we also indicate the checkpoint at which we stop training the CBOW models. We use 300-dimension vectors for all our word embedding models. For the Ngram2vec model, learning source and target embeddings for all the n-grams upto bigrams was the best performing model and is included in the comparison. For each method, we extensively tuned hyperparameters starting from the recommended values. For each model, we select the parameters which give the best averaged results on our word-similarity and analogy tasks. After selecting the best hyperparameters, we train 5 models for each method, using a different random seed. The reported results are given as mean and standard deviation for those five models. Word-similarity tasks . Word-similarity tasks consist of word pairs along with their human annotated similarity scores. To evaluate the performance of our models on pair-wise word-similarity tasks, we use WordSim353 (353 word-pairs) (Finkelstein et al., 2002) divided into two datasets, WordSim Similarity (203 word-pairs) and Model WS 353 WS 353 Relatedness WS 353 Similarity CBOW-char + bi. WordSim Relatedness (252 word-pairs) (Agirre et al., 2009); MEN (3000 word-pairs) (Bruni et al., 2012); Mechanical Turk dataset (Radinsky et al., 2011) (287 word-pairs); Rare words dataset (2034 word-pairs) (Luong et al., 2013); and SimLex-999 (999 word-pairs) (Hill et al., 2015) dataset. To calculate the similarity between two words, we use the cosine similarity between their word representations. The similarity scores then, are compared to the human ratings using Spear-man's  (Spearman, 1904) correlation scores. Word-analogy tasks . Word analogy tasks pose analogy relations of the form  x is to y as x (cid:63) is to y (cid:63) , where y is hidden and must be guessed from the dataset vocabulary. We use the MSR (Mikolov et al., 2013c) and the Google (Mikolov et al., 2013a) analogy datasets. The MSR dataset contains 8000 syntactic analogy quadruplets while the Google set has 8869 semantic and 10675 syntactic relations. To calculate the missing word in the relation, we use the 3CosMul method (Levy and Goldberg, 2014): y (cid:63) := arg max z V\{ x,y,x (cid:63) } cos ( v z , v y )  cos ( v z , v x (cid:63) ) cos ( v z , v x ) +  (2) where  = 0 . We remove all the out of vocabulary words and are left with 6946 syntactic relations for the MSR dataset and 1959 word-pairs for the Rare Words dataset. All other datasets do not have any out of vocabulary words. Impact of word n-grams . In Table 1, we evaluate the impact of adding contextual word ngrams to two CBOW variations: CBOW-char and Sent2Vec. By adding n-gram information, we consistently observe a boost in the Spearman correlation on the word similarity tasks. On the few datasets where we do not observe an improvement, the results for word-n-gram augmented methods are within standard deviation reach. The Rare Words dataset for Sent2Vec is the only exception, despite getting some improvement for CBOW-char based methods. This observation can be attributed to the fact that character ngrams are shared between unigrams, enhancing generalization to infrequent words. Without char n-grams, the model might underfit those rare words, even more so with word n-grams. We also see that the boost obtained by adding ngrams on word-similarity tasks is much larger for Sent2Vec models as compared to the CBOW-char ones possibly due to the fact that during training, Sent2Vec models use a much larger context and hence can use much more n-gram information for obtaining a better context representation. For analogy tasks, we see an improvement in the augmented CBOW-char methods for morpho-syntactic analogy datasets with little or no gain for semantic analogy datasets. Yet, for Sent2Vec models, the gain is the other way around. This observation indicates the strong role played by character n-grams in boosting the performance on the syntactic tasks as well as restricting the word ngrams from improving the performance on semantic analogies. Comparison with competing methods . In Table 2, we compare word n-gram augmented methods with the most prominent word embedding models. We obtain state-of-the-art results for standalone unigram embeddings on most of the datasets confirming our hypothesis. The Mechanical Turk dataset is the only exception. We notice that Sent2Vec trigrams model dominates the word-similarity tasks as well as the semantic analogy tasks. However, character n-grams are quite helpful when it comes to syntactic analogy tasks underlining the importance of subword information. We also note that the Ngram2vec model outperforms our augmented CBOW-char model in some of the tasks but is always inferior to Sent2Vec in those cases. We empirically show how augmenting the context representations using higher-order word ngrams improves the quality of word representations. The empirical success also calls for a new theoretical model on the composite effect of training higher order n-grams simultaneously with unigrams. Also, the success of Sent2Vec on word-level tasks, a method originally geared towards obtaining general purposed sentence embeddings, hints towards the additional benefits of using compositional methods for obtaining sentence/phrase representations.
abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain result abstain objective result abstain result method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method method abstain method method method abstain abstain abstain method other method abstain abstain abstain method abstain method method abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain abstain abstain result abstain abstain abstain abstain objective abstain abstain