What exactly is meant by isotropic and anisotropic with word vectors From this paper https://aclanthology.org/D19-1006.pdf "How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings"
When they say word vectors are anisotropic, do they mean all embeddings for words in the vocabulary are distributed in this cone shape in semantic space, and so there is a cosine similarity of 0.99 often between two different words from two different context? So a cosine similarity between the same word of two different contexts of 0.99 is not special?
If so, isn't it a problem that if there is a large semantic space, the vectors for the whole vocabulary are clustering to a region of the space?

Anisotropy is a problem if you treat the embeddings as vectors in Euclidian space. If your vocabulary ends up effectively in a tiny volume inside Euclidian space, then you shouldn't treat the vectors as Euclidian vectors. This is consequential for distance metrics used in your problem. This also means that perhaps the embedding could be compressed (shortened), that it has too many dimensions.
The authors claim that the entire vocabulary's embeddings' distribution is not isotropic.
