@lewtun my understanding is the following: consider the sentence a happy dog.
The happy token has an initial context-independent embedding that is transformed into another, context dependent, embedding via the self-attention mechanism that is essentially a linear combination such as: a*embedding('a') + b*embedding('happy') + c*embedding('dog') where a, b, c are the attention weights.
My point was to ask whether the initial embeddings (likely randomly initialized) are available and optimized at all during training?