Is there a difference between the model's performance for morphologically impoverished and morphologically complex languages?
They did not report results for English but expect that morphologically complex languages will perform better.
This paper presents a joint model for performing unsupervised morphological analysis on words, and learning a character-level composition function from morphemes to word embeddings. Our model splits individual words into segments, and weights each segment according to its ability to predict context words. Our morphological analysis is comparable to dedicated morphological analyzers at the task of morpheme boundary recovery, and also performs better than word-based embedding models at the task of syntactic analogy answering. Finally, we show that incorporating morphology explicitly into character-level models help them produce embeddings for unseen words which correlate better with human judgments.
Word embedding models associate each word in a corpus with a vector in a semantic space. These vectors can either be learnt to optimize performance in a downstream task BIBREF0 , BIBREF1 or learnt via the distributional hypothesis: words with similar contexts have similar meanings BIBREF2 , BIBREF3 . Current word embedding models treat words as atomic. However, words follow a power law distribution BIBREF4 , and word embedding models suffer from the problem of sparsity: a word like `unbelievableness' does not appear at all in the first 17 million words of Wikipedia, even though it is derived from common morphemes. This leads to three problems: One approach to smooth word distributions is to operate on the smallest meaningful semantic unit, the morpheme BIBREF6 , BIBREF7 . However, previous work on the morpheme level has all used external morphological analyzers. These require a separate pre-processing step, and cannot be adapted to suit the problem at hand. Another is to operate on the smallest orthographic unit, the character BIBREF8 , BIBREF9 . However, the link between shape and meaning is often complicated BIBREF10 , as alphabetic characters carry no inherent semantic meaning. To account for this, the model has to learn complicated dependencies between strings of characters to accurately capture word meaning. We hypothesize that explicitly introducing morphology into character-level models can help them learn morphological features, and hence word meaning. In this paper, we introduce a word embedding model that jointly learns word morphology and word embeddings. To the best of our knowledge, this is the first word embedding model that learns morphology as part of the model. Our guiding intuition is that the words with the same stem have similar contexts. Thus, when considering word segments in terms of context-predictive power, the segment corresponding to the stem will have the most weight. Our model `reads' the word and outputs a sequence of word segments. We weight each segment, and then combine the segments to obtain the final word representation. These representations are trained to predict context words, as this has been shown to give word representations which capture word semantics well BIBREF11 . As the root morpheme has the most context-predictive power, we expect our model to assign high weight to this segment, thereby learning to separate root+affix structures. One exciting feature of character-level models is their ability to represent open-vocabulary words. After training, they can predict a vector for any word, not just words that they have seen before. Our model has an advantage in that it can split unknown words into known and unknown components. Hence, it can potentially generalise better over seen morphemes and words and apply existing knowledge to new cases. To evaluate our model, we evaluate its use as a morphological analyzer (§ "Morphological awareness" ), test how well it learns word semantics, including for unseen words (§ "Capturing semantic similarity" ), and examine the structure of the embedding space (§ "Capturing syntactic and semantic regularity" ).