  Static word embeddings encode word associations, extensively utilized in
downstream NLP tasks. Although prior studies have discussed the nature of such
word associations in terms of biases and lexical regularities captured, the
variation in word associations based on the embedding training procedure
remains in obscurity. This work aims to address this gap by assessing
attributive word associations across five different static word embedding
architectures, analyzing the impact of the choice of the model architecture,
context learning flavor and training corpora. Our approach utilizes a
semi-supervised clustering method to cluster annotated proper nouns and
adjectives, based on their word embedding features, revealing underlying
attributive word associations formed in the embedding space, without
introducing any confirmation bias. Our results reveal that the choice of the
context learning flavor during embedding training (CBOW vs skip-gram) impacts
the word association distinguishability and word embeddings' sensitivity to
deviations in the training corpora. Moreover, it is empirically shown that even
when trained over the same corpora, there is significant inter-model disparity
and intra-model similarity in the encoded word associations across different
word embedding models, portraying specific patterns in the way the embedding
space is created for each embedding architecture.

Computation and Language