Answers.Body: <p>Let's start with a bit of notation and a couple of important clarifications.</p>
  <p><span class="math-container">$\mathbf{Q}$</span> refers to the <strong>query</strong> vectors matrix, <span class="math-container">$q_i$</span> being a single query vector associated with a single input word.</p>
  <p><span class="math-container">$\mathbf{V}$</span> refers to the <strong>values</strong> vectors matrix, <span class="math-container">$v_i$</span> being a single value vector associated with a single input word.</p>
  <p><span class="math-container">$\mathbf{K}$</span> refers to the <strong>keys</strong> vectors matrix, <span class="math-container">$k_i$</span> being a single key vector associated with a single input word.</p>
  <p>Where do these matrices come from? Something that is not stressed out enough in a lot of tutorials is that these matrices are the result of a <strong>matrix product between the input <em>embeddings</em> and 3 matrices of trained weights:</strong> <span class="math-container">$\mathbf{W_q}$</span>, <span class="math-container">$\mathbf{W_v}$</span>, <span class="math-container">$\mathbf{W_k}$</span>.</p>
  <p>The fact that these three matrices are learned during training explains why the query, value and key vectors end up being different despite the identical input sequence of embeddings. It also explains why it makes sense to talk about multi-head attention. Performing multiple attention steps on the same sentence produces different results, because, for each attention 'head', new <span class="math-container">$\mathbf{W_q}$</span>, <span class="math-container">$\mathbf{W_v}$</span>, <span class="math-container">$\mathbf{W_k}$</span> are randomly initialised.</p>
  <p>Another important aspect not stressed out enough is that for the encoder and decoder first attention layers, all the three matrices comes from the previous layer (either the input or the previous attention layer) but for the <strong>encoder/decoder</strong> attention layer, the <span class="math-container">$\mathbf{Q}$</span> matrix comes from the previous decoder layer, whereas the <span class="math-container">$\mathbf{V}$</span> and <span class="math-container">$\mathbf{K}$</span> matrices come from the encoder. And this is a crucial step to explain how the representation of two languages in an encoder is mixed together.</p>
  <p>Once computed the three matrices, the transformer moves on to the calculation of the dot product between query and key vectors. The dot product is used to compute a sort of similarity score between the query and key vectors. Indeed, the authors used the names <em>query</em>, <em>key</em> and <em>value</em> to indicate that what they propose is similar to what is done in information retrieval. For example, in question answering, usually, given a query, you want to retrieve the closest sentence in meaning among all possible answers, and this is done by computing the similarity between sentences (question vs possible answers).</p>
  <p>Of course, here, the situation is not exactly the same, but the guy who did the video you linked did a great job in explaining what happened during the attention computation (the two equations you wrote are exactly the same in vector and matrix notation and represent these passages):</p>
  <ul>
  <li>closer query and key vectors will have higher dot products.</li>
  <li>applying the softmax will normalise the dot product scores between 0 and 1.</li>
  <li>multiplying the softmax results to the value vectors will push down close to zero all value vectors for words that had a low dot product score between query and key vector.</li>
  </ul>
  <p>In the paper, the authors explain the attention mechanisms saying that the purpose is to determine which words of a sentence the transformer should focus on. I personally prefer to think of attention as a sort of <em>coreference resolution step</em>. The reason why I think so is the following image (taken from this <a href="https://www.slideshare.net/ilblackdragon/attention-is-all-you-need" rel="noreferrer">presentation</a> by the original authors).</p>
  <p><a href="https://i.sstatic.net/Nnm8a.jpg" rel="noreferrer"><img src="https://i.sstatic.net/Nnm8a.jpg" alt="enter image description here" /></a></p>
  <p>This image shows basically the result of the attention computation (at a specific layer that they don't mention). Bigger lines connecting words mean bigger values in the dot product between the words query and key vectors, which means basically that only those words value vectors will pass for further processing to the next attention layer. But, please, note that some words are actually related even if not similar at all, for example, 'Law' and 'The' are not similar, they are simply related to each other in these specific sentences (that's why I like to think of attention as a coreference resolution). Computing similarities between embeddings would never provide information about this relationship in a sentence, the only reason why transformer learn these relationships is the presences of the trained matrices <span class="math-container">$\mathbf{W_q}$</span>, <span class="math-container">$\mathbf{W_v}$</span>, <span class="math-container">$\mathbf{W_k}$</span> (plus the presence of positional embeddings).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T16:16:03.210Z
Answers.Text: Ive been searching for how the attention is calculated, for the past 3 days. Your answer provided the closest explanation. Thank you. If you have more clarity on it, please write a blog post or create a Youtube video. It'd be a great help for everyone.
Answers.UserDisplayName: Nav
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-31T14:34:51.723Z
Answers.Text: @Nav Hi, sorry but I saw your comment only now. I'm not really planning to write a blog post on this topic, mainly because I think that there are already good tutorials and video around that describe transformers in detail. Also, I saw that new posts are share every month, this one for example is really well made, hope you'll find it useful: http://peterbloem.nl/blog/transformers
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-13T09:57:35.080Z
Answers.Text: @Avatrin The weight matrices Eduardo is talking about here are not the raw dot product softmax wij that Bloem is writing about at the beginning of the article. The weight matrices here are an arbitrary choice of a linear operation that you make BEFORE applying the raw dot product self attention mechanism.
  
  These can technically come from anywhere, sure, but if you look at ANY implementation of the transformer architecture you will find that these are indeed learned parameters. Bloem covers this in entirety actually, so I don't quite understand your implication that Eduardo needs to reread it
Answers.UserDisplayName: Tim Seguine
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-13T11:47:36.270Z
Answers.Text: @TimSeguine Those linear layers are before the "scaled dot-product attention" as defined in Vaswani (seen in both equation 1 and figure 2 on page 4). They are however in the "multi-head attention". OPs question explicitly asks about equation 1. There are no weights in it.
Answers.UserDisplayName: Avatrin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-13T14:46:08.787Z
Answers.Text: @Avatrin Yes that's true, the attention function itself is matrix valued and parameter free(And I never disputed that fact), but your original comment is still false: "the three matrices W_q, W_k and W_v are not trained". Neither how they are defined here nor in the referenced blog post is that true.
Answers.UserDisplayName: Tim Seguine
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-13T15:12:48.177Z
Answers.Text: @TimSeguine Okay, fair enough. I think you are right. I misread the notation Bloem used. While I still believe this answer is misleading, and that attention is better explained through the properties of dot products, my initial comment was wrong as well. I'll delete it.
Answers.UserDisplayName: Avatrin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-13T17:29:14.077Z
Answers.Text: @Avatrin Yeah, I can understand that. Even still, the additional weight matrices don't really change it that much. You can interpret them as a learnable choice of basis for the query, key and value spaces. But that logic also applies to the weight matrices of many different structures in the right light, so it is fair to say that it distracts more than it helps.
Answers.UserDisplayName: Tim Seguine
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-12T21:36:41.570Z
Answers.Text: @EdoardoGuerriero. I wish I saw this before. Thank you
Answers.UserDisplayName: Avv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-26T00:25:26.593Z
Answers.Text: @EdoardoGuerriero Do the 3 projection matrices render the embedding (linear) layer redundant? (Performing project (linear) after embedding (linear) is just a single linear operation)
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-04T14:29:36.430Z
Answers.Text: Clearly explanation. But where input embeddings come from?
Answers.UserDisplayName: Wuaner
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-04T15:40:53.487Z
Answers.Text: One more question:  Since 3 W matrices are trained weights, then how can they come from random initialize?
Answers.UserDisplayName: Wuaner
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-11T14:49:29.190Z
Answers.LastActivityDate: 2021-12-05T08:34:15.117Z
Answers.LastEditDate: 2021-12-05T08:34:15.117Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I am watching the video <a href="https://www.youtube.com/watch?v=iDulhoQ2pro&amp;t=6s" rel="noreferrer">Attention Is All You Need</a> by Yannic Kilcher.</p>
  <p>My question is: what is the intuition behind the dot product attention?</p>
  <p><span class="math-container">$$A(q,K, V) = \sum_i\frac{e^{q.k_i}}{\sum_j e^{q.k_j}} v_i$$</span></p>
  <p>becomes:</p>
  <p><span class="math-container">$$A(Q,K, V) = \text{softmax}(QK^T)V$$</span></p>
  
Tags: <natural-language-processing><papers><transformer><attention><bert>
Title: What is the intuition behind the dot product attention?
UserDisplayName: DRV
ThreadText: # What is the intuition behind the dot product attention?
  
  (Asked by: username_0 on 2020-04-11)
  
  I am watching the video [Attention Is All You Need](https://www.youtube.com/watch?v=iDulhoQ2pro&t=6s) by Yannic Kilcher.
  
  My question is: what is the intuition behind the dot product attention?
  
  $$A(q,K, V) = \sum\_i\frac{e^{q.k\_i}}{\sum\_j e^{q.k\_j}} v\_i$$
  
  becomes:
  
  $$A(Q,K, V) = \text{softmax}(QK^T)V$$
  
  **Tags:** natural-language-processing, papers, transformer, attention, bert
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-11. Score: 34
  
  Let's start with a bit of notation and a couple of important clarifications.
  
  $\mathbf{Q}$ refers to the **query** vectors matrix, $q\_i$ being a single query vector associated with a single input word.
  
  $\mathbf{V}$ refers to the **values** vectors matrix, $v\_i$ being a single value vector associated with a single input word.
  
  $\mathbf{K}$ refers to the **keys** vectors matrix, $k\_i$ being a single key vector associated with a single input word.
  
  Where do these matrices come from? Something that is not stressed out enough in a lot of tutorials is that these matrices are the result of a **matrix product between the input *embeddings* and 3 matrices of trained weights:** $\mathbf{W\_q}$, $\mathbf{W\_v}$, $\mathbf{W\_k}$.
  
  The fact that these three matrices are learned during training explains why the query, value and key vectors end up being different despite the identical input sequence of embeddings. It also explains why it makes sense to talk about multi-head attention. Performing multiple attention steps on the same sentence produces different results, because, for each attention 'head', new $\mathbf{W\_q}$, $\mathbf{W\_v}$, $\mathbf{W\_k}$ are randomly initialised.
  
  Another important aspect not stressed out enough is that for the encoder and decoder first attention layers, all the three matrices comes from the previous layer (either the input or the previous attention layer) but for the **encoder/decoder** attention layer, the $\mathbf{Q}$ matrix comes from the previous decoder layer, whereas the $\mathbf{V}$ and $\mathbf{K}$ matrices come from the encoder. And this is a crucial step to explain how the representation of two languages in an encoder is mixed together.
  
  Once computed the three matrices, the transformer moves on to the calculation of the dot product between query and key vectors. The dot product is used to compute a sort of similarity score between the query and key vectors. Indeed, the authors used the names *query*, *key* and *value* to indicate that what they propose is similar to what is done in information retrieval. For example, in question answering, usually, given a query, you want to retrieve the closest sentence in meaning among all possible answers, and this is done by computing the similarity between sentences (question vs possible answers).
  
  Of course, here, the situation is not exactly the same, but the guy who did the video you linked did a great job in explaining what happened during the attention computation (the two equations you wrote are exactly the same in vector and matrix notation and represent these passages):
  
  * closer query and key vectors will have higher dot products.
  * applying the softmax will normalise the dot product scores between 0 and 1.
  * multiplying the softmax results to the value vectors will push down close to zero all value vectors for words that had a low dot product score between query and key vector.
  
  In the paper, the authors explain the attention mechanisms saying that the purpose is to determine which words of a sentence the transformer should focus on. I personally prefer to think of attention as a sort of *coreference resolution step*. The reason why I think so is the following image (taken from this [presentation](https://www.slideshare.net/ilblackdragon/attention-is-all-you-need) by the original authors).
  
  [![enter image description here](https://i.sstatic.net/Nnm8a.jpg)](https://i.sstatic.net/Nnm8a.jpg)
  
  This image shows basically the result of the attention computation (at a specific layer that they don't mention). Bigger lines connecting words mean bigger values in the dot product between the words query and key vectors, which means basically that only those words value vectors will pass for further processing to the next attention layer. But, please, note that some words are actually related even if not similar at all, for example, 'Law' and 'The' are not similar, they are simply related to each other in these specific sentences (that's why I like to think of attention as a coreference resolution). Computing similarities between embeddings would never provide information about this relationship in a sentence, the only reason why transformer learn these relationships is the presences of the trained matrices $\mathbf{W\_q}$, $\mathbf{W\_v}$, $\mathbf{W\_k}$ (plus the presence of positional embeddings).
  
  > **Comments:**
  > **username_2**: Ive been searching for how the attention is calculated, for the past 3 days. Your answer provided the closest explanation. Thank you. If you have more clarity on it, please write a blog post or create a Youtube video. It'd be a great help for everyone.  
  > **username_1**: @Nav Hi, sorry but I saw your comment only now. I'm not really planning to write a blog post on this topic, mainly because I think that there are already good tutorials and video around that describe transformers in detail. Also, I saw that new posts are share every month, this one for example is really well made, hope you'll find it useful: http://peterbloem.nl/blog/transformers  
  > **username_3**: @Avatrin The weight matrices Eduardo is talking about here are not the raw dot product softmax wij that Bloem is writing about at the beginning of the article. The weight matrices here are an arbitrary choice of a linear operation that you make BEFORE applying the raw dot product self attention mechanism.  These can technically come from anywhere, sure, but if you look at ANY implementation of the transformer architecture you will find that these are indeed learned parameters. Bloem covers this in entirety actually, so I don't quite understand your implication that Eduardo needs to reread it  
  > **username_4**: @TimSeguine Those linear layers are before the "scaled dot-product attention" as defined in Vaswani (seen in both equation 1 and figure 2 on page 4). They are however in the "multi-head attention". OPs question explicitly asks about equation 1. There are no weights in it.  
  > **username_3**: @Avatrin Yes that's true, the attention function itself is matrix valued and parameter free(And I never disputed that fact), but your original comment is still false: "the three matrices W_q, W_k and W_v are not trained". Neither how they are defined here nor in the referenced blog post is that true.  
  > **username_4**: @TimSeguine Okay, fair enough. I think you are right. I misread the notation Bloem used. While I still believe this answer is misleading, and that attention is better explained through the properties of dot products, my initial comment was wrong as well. I'll delete it.  
  > **username_3**: @Avatrin Yeah, I can understand that. Even still, the additional weight matrices don't really change it that much. You can interpret them as a learnable choice of basis for the query, key and value spaces. But that logic also applies to the weight matrices of many different structures in the right light, so it is fair to say that it distracts more than it helps.  
  > **username_5**: @EdoardoGuerriero. I wish I saw this before. Thank you  
  > **username_6**: @EdoardoGuerriero Do the 3 projection matrices render the embedding (linear) layer redundant? (Performing project (linear) after embedding (linear) is just a single linear operation)  
  > **username_7**: Clearly explanation. But where input embeddings come from?  
  > **username_7**: One more question:  Since 3 W matrices are trained weights, then how can they come from random initialize?  
  
  ---
  
  