Hi @rezhv ,
I think your confusion comes from thinking that the words are fed one by one. You’re exactly right that there could be no meaningful attention score when computing based on a single word embedding.
Please refer to Chapter 9.4 in the newly updated: https://web.stanford.edu/~jurafsky/slp3/9.pdf 6
Best of luck,
Chris