Answers.Body: <p>As far as I understand your question, you have problems with multiple heads. Let's take the input <span class="math-container">$\mathbf{Q}$</span> (with dimension: <span class="math-container">$\textit{seq_length}$</span> x <span class="math-container">$d_{model}$</span>), which is the same for <span class="math-container">$\mathbf{Q}, \mathbf{K}$</span> and <span class="math-container">$\mathbf{V}$</span> (thus <span class="math-container">$\mathbf{Q} = \mathbf{K} = \mathbf{V}$</span>). then you need to multiply them with 3 different matrices (with dimension: <span class="math-container">$d_{model}$</span> x <span class="math-container">$d_{model}$</span>), thus: <span class="math-container">$\mathbf{Q}$</span> x <span class="math-container">$\mathbf{W}_{q}$</span>, <span class="math-container">$\mathbf{K}$</span> x <span class="math-container">$\mathbf{W}_{k}$</span>, <span class="math-container">$\mathbf{V}$</span> x <span class="math-container">$\mathbf{W}_{v}$</span>. Why this multiplication? Because those are the matrices with the learnable parameters. Now... in case of multi head attention, you need to split them by the number heads, that you want. Since the model dimension (<span class="math-container">$d_{model}$</span>) must be divisible by the number of heads (<span class="math-container">$h$</span>). At this point you do not need to create or use any complicate function, you can just reshape the three matrices in this way:</p>
  <pre><code># (batch, seq_len, d_model) --&gt; (batch, seq_len, h, d_k) --&gt; (batch, h, seq_len, d_k)
  </code></pre>
  <p>or in PyTorch code if you prefer:</p>
  <pre><code>query = query.view(query.shape[0], query.shape[1], self.h, self.d_k).transpose(1, 2) (where d_k = dim_model // h)
  </code></pre>
  <p>In case you don't &quot;see&quot; the division in chunks, open a terminal, import torch and try the line of code above (removing the batch in all the calculation)
  allright?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-07T12:54:22.703Z
Answers.Text: so you say that we directly multiple the input x by Wq , Wk and Wv and then split them into a number of head that we want, we don't need for each head i  to multiply x by Wiq
  because in the paper "attention is all you need" they say that each head i has as input QWiq and KWik and VWiv that's make me in confusion and don't understand which method is correct
Answers.UserDisplayName: LAILA EL OUEDEGHYRY
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-07T16:08:44.477Z
Answers.Text: Forget the concept of "head" for a moment. Imagine only one block: an attention block.
  In this block you have one input. This input is copied three times (Q, K, and V).
  Each of them is then multiplied by a weight matrix. Stop.
  Once you have it, you can think about how split those three matrices in $h$ heads.
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-07T16:11:12.627Z
Answers.Text: About the paper: once you multiply Q by its weight matrix, you get a new matrix $Q'$. which is exactly $Q' = Q W_{q}$. The same for K and V. Just Multiply Q, K and V by their respectively matrices and go on with the code
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-07T18:30:07.473Z
Answers.Text: Look at this video. It has been published 3 hours ago: 
  https://youtu.be/eMlx5fFNoYc?si=SlEtj84pUHh4gxBh
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-05T19:30:57.530Z
Answers.LastActivityDate: 2024-04-07T07:23:59.173Z
Answers.LastEditDate: 2024-04-07T07:23:59.173Z
Answers.UserDisplayName: Dave
Answers.Body: <p>Given the input <span class="math-container">$x$</span> into the head, Q, K, and V are simply <span class="math-container">$x$</span>, but then adapted by distinct fully connected neural network layer.</p>
  <p>To clarify, each head has 3 linear layers, one for constructing Q, one for constructing K, and one for constructing V. The input for each of these linear layers is just a copy of <span class="math-container">$x$</span>. Each head (of multi head attention) has their own set of parameters.</p>
  <p>Notably, these learnable linear layers are <em>the only learnable parameters</em> of the head.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-07T12:59:33.480Z
Answers.Text: so you say that  for each head i to multiply x by Wiq , etc and in the paper "attention is all you need"  when they say that each head i has as input QWiq and KWik and VWiv they mean by Q,K,V the copy of the input?
Answers.UserDisplayName: LAILA EL OUEDEGHYRY
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-07T10:33:26.940Z
Answers.LastActivityDate: 2024-04-07T10:33:26.940Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>I want to understand the transformer architecture, so I start with self attention and I understand their mechanism, but when I pass to the multi-head attention I find some difficulties like how calculate Q , K and V for each head.
  I find many way to calculate Q , K and V but I don't know which  way is correct.<br>
  <strong>method 1:</strong>
  <img src="https://jalammar.github.io/images/t/transformer_multi-headed_self-attention-recap.png" alt="this image illustrate how calcule these matrices " />
  <strong>method 2:</strong>
  I find this method in YouTube.
  <a href="https://i.sstatic.net/HEZHv.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HEZHv.png" alt="enter image description here" /></a>
  <strong>method 3:</strong>
  in this method it just for headi we have:
  Qi=xWqi
  Ki=xWki
  Vi=xWvi
  so I don't know which method is correct
  there is the links of my references
  <a href="https://jalammar.github.io/illustrated-transformer/" rel="nofollow noreferrer">The Illustrated Transformer</a>
  <a href="https://youtu.be/bCz4OMemCcA?si=4cJa0YojjkPgzcs0" rel="nofollow noreferrer">YouTube video</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-02T10:05:27.783Z
Comments.Text: Q,K,V are the input matrices. In self attention they would just be 3 copies of the input sequence(s)
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-02T10:47:37.540Z
Comments.Text: so you man that for each headi we have:
  For the query matrix: Qi​=X×Wqi​
  For the key matrix: Ki​=X×Wki
  For the value matrix: Vi​=X×Wvi
  which X in the input of encoder or the output of the previous encoder if we have maby layer of decoder
Comments.UserDisplayName: LAILA EL OUEDEGHYRY
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-02T10:57:10.227Z
Comments.Text: Yes. Important to note that the project matrices W_{q,v,k} are different for each head.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-02T11:54:21.810Z
Comments.Text: Yes, I get it ,for each head we initialize randomly   W_{q,v,k}
Comments.UserDisplayName: LAILA EL OUEDEGHYRY
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-07T10:26:46.503Z
Comments.Text: Ill add this, which has an intuitive explanation on its calculation as well:https://ai.stackexchange.com/questions/40179/how-does-the-decoder-only-transformer-architecture-work/40180#40180
Comments.UserDisplayName: Robin van Hoorn
Tags: <machine-learning><deep-learning><natural-language-processing><transformer><attention>
Title: How the Q,K,V be calculated in multi-head attention
UserDisplayName: LAILA EL OUEDEGHYRY
ThreadText: # How the Q,K,V be calculated in multi-head attention
  
  (Asked by: username_0 on 2024-04-02)
  
  I want to understand the transformer architecture, so I start with self attention and I understand their mechanism, but when I pass to the multi-head attention I find some difficulties like how calculate Q , K and V for each head.
  I find many way to calculate Q , K and V but I don't know which way is correct.  
  **method 1:**
  ![this image illustrate how calcule these matrices ](https://jalammar.github.io/images/t/transformer_multi-headed_self-attention-recap.png)
  **method 2:**
  I find this method in YouTube.
  [![enter image description here](https://i.sstatic.net/HEZHv.png)](https://i.sstatic.net/HEZHv.png)
  **method 3:**
  in this method it just for headi we have:
  Qi=xWqi
  Ki=xWki
  Vi=xWvi
  so I don't know which method is correct
  there is the links of my references
  [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/)
  [YouTube video](https://youtu.be/bCz4OMemCcA?si=4cJa0YojjkPgzcs0)
  
  **Tags:** machine-learning, deep-learning, natural-language-processing, transformer, attention
  > **Comments:**
  > **username_1**: Q,K,V are the input matrices. In self attention they would just be 3 copies of the input sequence(s)  
  > **username_0**: so you man that for each headi we have: For the query matrix: Qi​=X×Wqi​ For the key matrix: Ki​=X×Wki For the value matrix: Vi​=X×Wvi which X in the input of encoder or the output of the previous encoder if we have maby layer of decoder  
  > **username_1**: Yes. Important to note that the project matrices W_{q,v,k} are different for each head.  
  > **username_0**: Yes, I get it ,for each head we initialize randomly   W_{q,v,k}  
  > **username_2**: Ill add this, which has an intuitive explanation on its calculation as well:https://ai.stackexchange.com/questions/40179/how-does-the-decoder-only-transformer-architecture-work/40180#40180  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2024-04-05. Score: 1
  
  As far as I understand your question, you have problems with multiple heads. Let's take the input $\mathbf{Q}$ (with dimension: $\textit{seq_length}$ x $d_{model}$), which is the same for $\mathbf{Q}, \mathbf{K}$ and $\mathbf{V}$ (thus $\mathbf{Q} = \mathbf{K} = \mathbf{V}$). then you need to multiply them with 3 different matrices (with dimension: $d_{model}$ x $d_{model}$), thus: $\mathbf{Q}$ x $\mathbf{W}_{q}$, $\mathbf{K}$ x $\mathbf{W}_{k}$, $\mathbf{V}$ x $\mathbf{W}_{v}$. Why this multiplication? Because those are the matrices with the learnable parameters. Now... in case of multi head attention, you need to split them by the number heads, that you want. Since the model dimension ($d_{model}$) must be divisible by the number of heads ($h$). At this point you do not need to create or use any complicate function, you can just reshape the three matrices in this way:
  # (batch, seq_len, d_model) --> (batch, seq_len, h, d_k) --> (batch, h, seq_len, d_k)
  
  or in PyTorch code if you prefer:
  query = query.view(query.shape[0], query.shape[1], self.h, self.d_k).transpose(1, 2) (where d_k = dim_model // h)
  
  In case you don't "see" the division in chunks, open a terminal, import torch and try the line of code above (removing the batch in all the calculation)
  allright?
  
  > **Comments:**
  > **username_0**: so you say that we directly multiple the input x by Wq , Wk and Wv and then split them into a number of head that we want, we don't need for each head i  to multiply x by Wiq because in the paper "attention is all you need" they say that each head i has as input QWiq and KWik and VWiv that's make me in confusion and don't understand which method is correct  
  > **username_3**: Forget the concept of "head" for a moment. Imagine only one block: an attention block. In this block you have one input. This input is copied three times (Q, K, and V). Each of them is then multiplied by a weight matrix. Stop. Once you have it, you can think about how split those three matrices in $h$ heads.  
  > **username_3**: About the paper: once you multiply Q by its weight matrix, you get a new matrix $Q'$. which is exactly $Q' = Q W_{q}$. The same for K and V. Just Multiply Q, K and V by their respectively matrices and go on with the code  
  > **username_3**: Look at this video. It has been published 3 hours ago:  https://youtu.be/eMlx5fFNoYc?si=SlEtj84pUHh4gxBh  
  
  ---
  
  ### Answer by username_2 on 2024-04-07. Score: 0
  
  Given the input $x$ into the head, Q, K, and V are simply $x$, but then adapted by distinct fully connected neural network layer.
  
  To clarify, each head has 3 linear layers, one for constructing Q, one for constructing K, and one for constructing V. The input for each of these linear layers is just a copy of $x$. Each head (of multi head attention) has their own set of parameters.
  
  Notably, these learnable linear layers are *the only learnable parameters* of the head.
  
  > **Comments:**
  > **username_0**: so you say that  for each head i to multiply x by Wiq , etc and in the paper "attention is all you need"  when they say that each head i has as input QWiq and KWik and VWiv they mean by Q,K,V the copy of the input?  
  
  ---
  
  