Answers.Body: <p>As I understand it (and I'm not an AI researcher, so any helpful comments from folks who know the topic better will be illuminating) the output of layer <span class="math-container">$l \in 1 ...\bf{L}$</span>, <span class="math-container">$\bf{X}^l$</span>,  is</p>
  <p><a href="https://i.sstatic.net/SKQyt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/SKQyt.png" alt="enter image description here" /></a></p>
  <p>where <span class="math-container">$a\in 1...A$</span> is the head number, and <span class="math-container">$f$</span> is some function like RELU or whatever and the <span class="math-container">$\bf{b}$</span>s are biases (<span class="math-container">$M$</span> is the attention mask and <span class="math-container">$d_E$</span> is the size of the embedding). The first bit corresponds to <a href="https://ai.stackexchange.com/a/39195/10649">@Soltius's correction</a> (and the second bit is the FFN). (And <span class="math-container">$\underset{\mathsf{vocab}}{\mathsf{softmax}}\left(\bf{X}^L\bf{W}_E^{-1}\right)$</span>
  is what's used in calculating cost).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-20T20:09:05.690Z
Answers.Text: This assumes that there are no biases in the QKV stuff (none are mentioned in anything I've read).
Answers.UserDisplayName: orome
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-20T20:08:02.107Z
Answers.LastActivityDate: 2023-04-20T21:53:35.390Z
Answers.LastEditDate: 2023-04-20T21:53:35.390Z
Answers.UserDisplayName: orome
Answers.Body: <p><em>(OP auto-answer) After having dug further in and read more papers on attention, and with help from Chillston in the comments, I think I've got it narrowed down to an issue of confusing notation. If anyone thinks this is not the right answer, please don't hesitate to submit another one, which I'll mark as correct if I think it's better.</em></p>
  <hr />
  <p><span class="math-container">$Q$</span>, <span class="math-container">$K$</span> and <span class="math-container">$V$</span> values <em>are</em> defined in the paper, and they <em>do</em> come from multiplication with learnt projection matrices. Those matrices are <span class="math-container">$W^Q_i$</span>, <span class="math-container">$W^K_i$</span> and <span class="math-container">$W^V_i$</span>, defined in section 3.2.2 of the paper.</p>
  <p>The confusion stems from the fact that the notation used in the multihead attention equation and in Figure 2 (right) of the paper is wrong/confusing.  The equation would be be clearer if it read :
  <a href="https://i.sstatic.net/cR99V.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/cR99V.png" alt="![enter image description here" /></a>
  And Figure 2 right could be modified accordingly :<br />
  <a href="https://i.sstatic.net/9ahfh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9ahfh.png" alt="enter image description here" /></a></p>
  <p>In this new notation, <span class="math-container">$X$</span> and <span class="math-container">$Y$</span> are the inputs to the current attention unit.</p>
  <ul>
  <li>For self attention, we'd have <span class="math-container">$X = Y$</span> which would both be the previous en/decoder block output (or word embedding for the first encoder block).</li>
  <li>For cross-attention, <span class="math-container">$X$</span> would be the output of the last encoder block and <span class="math-container">$Y$</span> the output of the previous decoder block.</li>
  </ul>
  <p><em>Technically</em>, the way it's written in the paper could be correct but you need to consider that <span class="math-container">$Q, K, V$</span> refer to different tensors when they're written :</p>
  <ul>
  <li>in the multihead(<span class="math-container">$Q,K,V$</span>) equation where they represent inputs, <span class="math-container">$i.e.$</span> what they call <span class="math-container">$V$</span> is <span class="math-container">$X$</span> in my suggested re-writing ;</li>
  <li>in the attention(<span class="math-container">$Q,K,V$</span>) equation where they represent &quot;true&quot; query/key/values, meaning inputs multiplied by projections matrices, <span class="math-container">$i.e.$</span> what they call <span class="math-container">$V$</span> is <span class="math-container">$XW^V_i$</span> in my suggested re-writing.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-20T19:52:46.050Z
Answers.Text: It would be great to have a weigh (no pun) in on this from someone who has worked in the field and reviewed the paper, because I agree with @Soltius that the notation in the paper is at best confusing and might even be wrong. The alternate version here is certainly both clearer and correct.
Answers.UserDisplayName: orome
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-08T18:14:37.710Z
Answers.Text: This makes sense to me and I think you kinda nailed it. It's wild that this is the first I'm seeing this but I do believe you're correct.
Answers.UserDisplayName: SuaveSouris
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-17T14:52:59.457Z
Answers.LastActivityDate: 2024-01-09T08:52:40.820Z
Answers.LastEditDate: 2024-01-09T08:52:40.820Z
Answers.UserDisplayName: Soltius
Body: <p>The seminal <a href="https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" rel="nofollow noreferrer">Attention is all you need</a> paper (Google Brain team, 2017) introduces Transformers and implements the attention mecanism with &quot;queries, keys, values&quot;, in an analogy to a retrieval system.</p>
  <p>I understand the whole process of multi-head attention and such (<em>i.e.</em>, what is done with the <span class="math-container">$Q$</span>, <span class="math-container">$K$</span>, <span class="math-container">$V$</span> values and why), but I'm confused on <strong>how these values are computed in the first place</strong>. AFAICT, the paper seems to completely leave that out.</p>
  <p>Both Figure 2 of the paper and equations explaining Attention and Multihead attention start with <span class="math-container">$Q$</span>,<span class="math-container">$K$</span>,<span class="math-container">$V$</span> already there :</p>
  <p><a href="https://i.sstatic.net/t6qJz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/t6qJz.png" alt="enter image description here" /></a></p>
  <p>The answers regaridng the origin of <span class="math-container">$Q$</span>,<span class="math-container">$K$</span>,<span class="math-container">$V$</span> I've found so far haven't satisfied me :</p>
  <ul>
  <li><p>In this <a href="https://stats.stackexchange.com/questions/421935/what-exactly-are-keys-queries-and-values-in-attention-mechanisms">similar question</a>, the <a href="https://stats.stackexchange.com/a/424127/201218">accepted answer</a> says &quot;<em>The proposed multihead attention alone doesn't say much about how the queries, keys, and values are obtained, they can come from different sources depending on the application scenario.</em>&quot;. If this is the case, then why isn't the computing of <span class="math-container">$Q$</span>,<span class="math-container">$K$</span>,<span class="math-container">$V$</span> made more clear in the paper, at the very least for the task of language translation for which they show some numerical results and so obviously did compute <span class="math-container">$Q$</span>,<span class="math-container">$K$</span>,<span class="math-container">$V$</span> in some way ?</p>
  </li>
  <li><p>I also see some answers (eg <a href="https://stats.stackexchange.com/a/463320/201218">this one on the same question</a>) which say that <span class="math-container">$Q$</span>, <span class="math-container">$K$</span>, <span class="math-container">$V$</span> are the result of multiplication of the input embedding with some matrices <span class="math-container">$W$</span>. This is also what is shown in the popular blog post <a href="http://jalammar.github.io/illustrated-transformer/" rel="nofollow noreferrer">The Illustrated Transformer</a> :</p>
  </li>
  </ul>
  <p><a href="https://i.sstatic.net/8QJ6pl.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8QJ6pl.png" alt="enter image description here" /></a></p>
  <p>These &quot;projection&quot; matrices (<span class="math-container">$W^Q$</span>, <span class="math-container">$W^K$</span>, <span class="math-container">$W^V$</span>) do seem to appear in the the definition of attention in the definition of <span class="math-container">$head_i$</span> (see top figure), but according to that equation, these matrices are multiplied by <span class="math-container">$Q$</span>, <span class="math-container">$K$</span>, <span class="math-container">$V$</span> (still appearing out of thin air, so the problem of their definition remains) and so the resulting product can't also be <span class="math-container">$Q$</span>, <span class="math-container">$K$</span>, <span class="math-container">$V$</span>.</p>
  <p>How are the <span class="math-container">$Q$</span>, <span class="math-container">$K$</span>, <span class="math-container">$V$</span> values computed ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-15T10:30:15.457Z
Comments.Text: I agree with you that this is missing in the paper and I cannot find any justification for that either. However, currently in your proposed correction, the difference in attention between encoder and decoder is not covered, it only supports self-attention and not the cross-attention between encoder and decoder blocks (Also, why did you leave the $V W_i^V$ in there?). This can be easily adapted and clarified though and then I would definitely favor your notation, which makes it much clearer where the $K$, $Q$, $V$ vectors come from.
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-15T11:46:48.757Z
Comments.Text: @Chillston Thanks for your comment ! I corrected the $V$ typo and proposed a notation which enables to cover both self and cross-attention (and more generally reworded the whole last paragraph).
Comments.UserDisplayName: Soltius
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-15T12:34:26.640Z
Comments.Text: Nice! AFAIK the cross-attention takes the key and value vectors from the encoder and only the query vector comes from the decoder ([here](https://www.tensorflow.org/text/tutorials/transformer#the_cross_attention_layer) it is described by the guys from Tensorflow). If I'm not mistaken this should translate to $\text{Attention}(YW_i^Q, XW_i^K, YW_i^V)$ using your notation.
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-15T12:56:12.447Z
Comments.Text: That's the way I understand it too, but isn't my current notation saying exactly that ? I don't get why you changed $X$ to $Y$ for $V$ since you say just before that "key and value vectors from the encoder"
Comments.UserDisplayName: Soltius
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-15T13:36:26.410Z
Comments.Text: Oh, my bad I got confused :D 
  you are right the way you have it fits perfectly
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-15T13:53:29.137Z
Comments.Text: OK great ! I'll leave this question the way it is for a couple days in case anyone has some alternative answer and if not, auto-answer it with the edit. Thanks for having taken the time to read and check my supposition !
Comments.UserDisplayName: Soltius
Tags: <transformer><attention><word-embedding>
Title: "Attention is all you need" paper : How are the Q, K, V values calculated?
UserDisplayName: Soltius
ThreadText: # "Attention is all you need" paper : How are the Q, K, V values calculated?
  
  (Asked by: username_0 on 2023-02-14)
  
  The seminal [Attention is all you need](https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) paper (Google Brain team, 2017) introduces Transformers and implements the attention mecanism with "queries, keys, values", in an analogy to a retrieval system.
  
  I understand the whole process of multi-head attention and such (*i.e.*, what is done with the $Q$, $K$, $V$ values and why), but I'm confused on **how these values are computed in the first place**. AFAICT, the paper seems to completely leave that out.
  
  Both Figure 2 of the paper and equations explaining Attention and Multihead attention start with $Q$,$K$,$V$ already there :
  
  [![enter image description here](https://i.sstatic.net/t6qJz.png)](https://i.sstatic.net/t6qJz.png)
  
  The answers regaridng the origin of $Q$,$K$,$V$ I've found so far haven't satisfied me :
  
  * In this [similar question](https://stats.stackexchange.com/questions/421935/what-exactly-are-keys-queries-and-values-in-attention-mechanisms), the [accepted answer](https://stats.stackexchange.com/a/424127/201218) says "*The proposed multihead attention alone doesn't say much about how the queries, keys, and values are obtained, they can come from different sources depending on the application scenario.*". If this is the case, then why isn't the computing of $Q$,$K$,$V$ made more clear in the paper, at the very least for the task of language translation for which they show some numerical results and so obviously did compute $Q$,$K$,$V$ in some way ?
  * I also see some answers (eg [this one on the same question](https://stats.stackexchange.com/a/463320/201218)) which say that $Q$, $K$, $V$ are the result of multiplication of the input embedding with some matrices $W$. This is also what is shown in the popular blog post [The Illustrated Transformer](http://jalammar.github.io/illustrated-transformer/) :
  
  [![enter image description here](https://i.sstatic.net/8QJ6pl.png)](https://i.sstatic.net/8QJ6pl.png)
  
  These "projection" matrices ($W^Q$, $W^K$, $W^V$) do seem to appear in the the definition of attention in the definition of $head\_i$ (see top figure), but according to that equation, these matrices are multiplied by $Q$, $K$, $V$ (still appearing out of thin air, so the problem of their definition remains) and so the resulting product can't also be $Q$, $K$, $V$.
  
  How are the $Q$, $K$, $V$ values computed ?
  
  **Tags:** transformer, attention, word-embedding
  
  > **Comments:**
  > **username_1**: I agree with you that this is missing in the paper and I cannot find any justification for that either. However, currently in your proposed correction, the difference in attention between encoder and decoder is not covered, it only supports self-attention and not the cross-attention between encoder and decoder blocks (Also, why did you leave the $V W_i^V$ in there?). This can be easily adapted and clarified though and then I would definitely favor your notation, which makes it much clearer where the $K$, $Q$, $V$ vectors come from.  
  > **username_0**: @username_1 Thanks for your comment ! I corrected the $V$ typo and proposed a notation which enables to cover both self and cross-attention (and more generally reworded the whole last paragraph).  
  > **username_1**: Nice! AFAIK the cross-attention takes the key and value vectors from the encoder and only the query vector comes from the decoder ([here](https://www.tensorflow.org/text/tutorials/transformer#the_cross_attention_layer) it is described by the guys from Tensorflow). If I'm not mistaken this should translate to $\text{Attention}(YW_i^Q, XW_i^K, YW_i^V)$ using your notation.  
  > **username_0**: That's the way I understand it too, but isn't my current notation saying exactly that ? I don't get why you changed $X$ to $Y$ for $V$ since you say just before that "key and value vectors from the encoder"  
  > **username_1**: Oh, my bad I got confused :D  you are right the way you have it fits perfectly  
  > **username_0**: OK great ! I'll leave this question the way it is for a couple days in case anyone has some alternative answer and if not, auto-answer it with the edit. Thanks for having taken the time to read and check my supposition !  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2023-02-17. Score: 5 | Accepted answer
  
  *(OP auto-answer) After having dug further in and read more papers on attention, and with help from username_1 in the comments, I think I've got it narrowed down to an issue of confusing notation. If anyone thinks this is not the right answer, please don't hesitate to submit another one, which I'll mark as correct if I think it's better.*
  
  ---
  
  $Q$, $K$ and $V$ values *are* defined in the paper, and they *do* come from multiplication with learnt projection matrices. Those matrices are $W^Q\_i$, $W^K\_i$ and $W^V\_i$, defined in section 3.2.2 of the paper.
  
  The confusion stems from the fact that the notation used in the multihead attention equation and in Figure 2 (right) of the paper is wrong/confusing. The equation would be be clearer if it read :
  [![![enter image description here](https://i.sstatic.net/cR99V.png)](https://i.sstatic.net/cR99V.png)
  And Figure 2 right could be modified accordingly :  
  [![enter image description here](https://i.sstatic.net/9ahfh.png)](https://i.sstatic.net/9ahfh.png)
  
  In this new notation, $X$ and $Y$ are the inputs to the current attention unit.
  
  * For self attention, we'd have $X = Y$ which would both be the previous en/decoder block output (or word embedding for the first encoder block).
  * For cross-attention, $X$ would be the output of the last encoder block and $Y$ the output of the previous decoder block.
  
  *Technically*, the way it's written in the paper could be correct but you need to consider that $Q, K, V$ refer to different tensors when they're written :
  
  * in the multihead($Q,K,V$) equation where they represent inputs, $i.e.$ what they call $V$ is $X$ in my suggested re-writing ;
  * in the attention($Q,K,V$) equation where they represent "true" query/key/values, meaning inputs multiplied by projections matrices, $i.e.$ what they call $V$ is $XW^V\_i$ in my suggested re-writing.
  
  > **Comments:**
  > **username_2**: It would be great to have a weigh (no pun) in on this from someone who has worked in the field and reviewed the paper, because I agree with @username_0 that the notation in the paper is at best confusing and might even be wrong. The alternate version here is certainly both clearer and correct.  
  > **username_3**: This makes sense to me and I think you kinda nailed it. It's wild that this is the first I'm seeing this but I do believe you're correct.  
  
  ---
  
  ### Answer by username_2 on 2023-04-20. Score: 0
  
  As I understand it (and I'm not an AI researcher, so any helpful comments from folks who know the topic better will be illuminating) the output of layer $l \in 1 ...\bf{L}$, $\bf{X}^l$, is
  
  [![enter image description here](https://i.sstatic.net/SKQyt.png)](https://i.sstatic.net/SKQyt.png)
  
  where $a\in 1...A$ is the head number, and $f$ is some function like RELU or whatever and the $\bf{b}$s are biases ($M$ is the attention mask and $d\_E$ is the size of the embedding). The first bit corresponds to [@username_0's correction](https://ai.stackexchange.com/a/39195/10649) (and the second bit is the FFN). (And $\underset{\mathsf{vocab}}{\mathsf{softmax}}\left(\bf{X}^L\bf{W}\_E^{-1}\right)$
  is what's used in calculating cost).
  
  > **Comments:**
  > **username_2**: This assumes that there are no biases in the QKV stuff (none are mentioned in anything I've read).  
  
  ---
  
  