How much were the gains they obtained?
Only 0.2% accuracy gain in morpho-sintactic questions in original test set, and 12.7% accuracy gain on their test set
Skip-gram (word2vec) is a recent method for creating vector representations of words ("distributed word representations") using a neural network. The representation gained popularity in various areas of natural language processing, because it seems to capture syntactic and semantic information about words without any explicit supervision in this respect. We propose SubGram, a refinement of the Skip-gram model to consider also the word structure during the training process, achieving large gains on the Skip-gram original test set.
Vector representations of words learned using neural networks (NN) have proven helpful in many algorithms for image annotation BIBREF0 or BIBREF1 , language modeling BIBREF2 , BIBREF3 and BIBREF4 or other natural language processing (NLP) tasks BIBREF5 or BIBREF6 . Traditionally, every input word of an NN is stored in the “one-hot” representation, where the vector has only one element set to one and the rest of the vector are zeros. The size of the vector equals to the size of the vocabulary. The NN is trained to perform some prediction, e.g. to predict surrounding words given a word of interest. Instead of using this prediction capacity in some task, the practice is to extract the output of NN's hidden layer of each word (called distributed representation) and directly use this deterministic mapping INLINEFORM0 of word forms to the vectors of real numbers as the word representation. The input one-hot representation of words has two weaknesses: the bloat of the size of the vector with more words in vocabulary and the inability to provide any explicit semantic or syntactic information to the NN. The learned distributed representation of words relies on much shorter vectors (e.g. vocabularies containing millions words are represented in vectors of a few hundred elements) and semantic or syntactic information is often found to be implicitly present (“embedded”) in the vector space. For example, the Euclidean distance between two words in the vector space may be related to semantic or syntactic similarity between them.