olaffson:

What the transformer architecture does, however, is - for each word - to create a context specific. embedding that is essentially a weighted average of the embeddings of all the words in the sentence.


yes this is correct, although to be precise it’s for each token (which can be a word or subword) 



 olaffson:

Interestingly, by reading the papers closely I think the following is true: there is an embedding for each word in the model that is independent of the given sentence.


i don’t understand this - which papers and where in them does this come from?