How is unstability defined?
An embedding is unstable if it has a low number of nearest neighbor embeddings of the words within the same frequency bucket.
Despite the recent popularity of word embedding methods, there is only a small body of work exploring the limitations of these representations. In this paper, we consider one aspect of embedding spaces, namely their stability. We show that even relatively high frequency words (100-200 occurrences) are often unstable. We provide empirical evidence for how various factors contribute to the stability of word embeddings, and we analyze the effects of stability on downstream tasks.
Word embeddings are low-dimensional, dense vector representations that capture semantic properties of words. Recently, they have gained tremendous popularity in Natural Language Processing (NLP) and have been used in tasks as diverse as text similarity BIBREF0 , part-of-speech tagging BIBREF1 , sentiment analysis BIBREF2 , and machine translation BIBREF3 . Although word embeddings are widely used across NLP, their stability has not yet been fully evaluated and understood. In this paper, we explore the factors that play a role in the stability of word embeddings, including properties of the data, properties of the algorithm, and properties of the words. We find that word embeddings exhibit substantial instabilities, which can have implications for downstream tasks. Using the overlap between nearest neighbors in an embedding space as a measure of stability (see sec:definingStability below for more information), we observe that many common embedding spaces have large amounts of instability. For example, Figure FIGREF1 shows the instability of the embeddings obtained by training word2vec on the Penn Treebank (PTB) BIBREF4 . As expected, lower frequency words have lower stability and higher frequency words have higher stability. What is surprising however about this graph is the medium-frequency words, which show huge variance in stability. This cannot be explained by frequency, so there must be other factors contributing to their instability. In the following experiments, we explore which factors affect stability, as well as how this stability affects downstream tasks that word embeddings are commonly used for. To our knowledge, this is the first study comprehensively examining the factors behind instability.