Towards Data Science, Neural Networks, Deep Learning, Machine Learning, Naturallanguageprocessing.
step. 2. The Attention Mechanism 2.1 Self-Attention Figure 2. Example to explain self-attention (Source: Image created by author) Consider the sentence — ” Bark is very cute and he is a dog”. This sentence has 9 words or tokens. If we just consider the word ‘he’ in the sentence, we see that ‘and’