Hello vainaijr,
I agree with you. I would add that PyTorch’s Tutorial specifically on Word Embeddings does a good job with communicating an intuition (https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html 58).
As you have kind of focused on the transformer architecture and also written about a use in CV, I just want to throw in this relatively new paper which could be interesting (https://openreview.net/forum?id=YicbFdNTTy 2).
Furthermore, I don’t think these embeddings (speaking about word embeddings) claim to consider order, that’s why we have positional encoding in transformers for instance.
I guess one of the main advantages in using (word) embeddings is that we have dense vectors and also the ability to ‘compare the meanings’ of words just using the embeddings.
Greetings,
Unity05