Interestingly, by reading the papers closely I think the following is true: there is an embedding for each word in the model that is independent of the given sentence.
What the transformer architecture does, however, is - for each word - to create a context specific. embedding that is essentially a weighted average of the embeddings of all the words in the sentence.
The key question is: @BramVanroy can we recover these two embeddings from huggingface? The  context independent and the more useful context dependent one? Does that make sense?
thanks!