Answers.Body: <p>The original paper &quot;Attention is all you need&quot; mentions the following.</p>
  <p>&quot;
  Instead of performing a single attention function with <span class="math-container">$ d_{model} $</span>-dimensional keys, values and queries,
  we found it beneficial to linearly project the queries, keys and values <span class="math-container">$h$</span> times with different, learned
  linear projections to <span class="math-container">$d_k$</span>, <span class="math-container">$d_k$</span> and <span class="math-container">$d_v$</span> dimensions, respectively. &quot;</p>
  <p>What is single-headed attention?</p>
  <p>The input matrix <span class="math-container">$ X $</span> of dimension <span class="math-container">$ T \times d_{model} $</span> is multipled with <span class="math-container">$ W^Q, W^K $</span> and <span class="math-container">$W^V $</span> matrices of dimension <span class="math-container">$ d_{model} \times d_{model} $</span> to give output of dimension <span class="math-container">$ T \times d_{model} $</span> Now, while calculating attention  <span class="math-container">$ A=softmax(\frac{QK^{T}}{\sqrt{d_k}}) $</span>yields one attention matrix of dimension  <span class="math-container">$ T \times T $</span>.</p>
  <p>What is multi-headed attention?</p>
  <p>The input matrix <span class="math-container">$ X $</span> of dimension <span class="math-container">$ T \times d_{model} $</span> is multipled with <span class="math-container">$ W^Q, W^K $</span> and <span class="math-container">$W^V $</span> matrices of dimension <span class="math-container">$ d_{model} \times d_{k} $</span> to give output of dimension <span class="math-container">$ T \times d_{k} $</span>. Note that <span class="math-container">$d_{model} = h \times d_{k} $</span> where <span class="math-container">$ h $</span> is the number of heads. Now, while calculating attention  <span class="math-container">$ A=softmax(\frac{QK^{T}}{\sqrt{d_k}}) $</span>yields <span class="math-container">$h$</span> attention matrices of dimension  <span class="math-container">$ T \times T $</span>.</p>
  <p>It is called multi-headed attention because each head produces one different attention matrix.</p>
  <p>(I feel your confusion may have come from the knowledge that matrix multiplication happens in blocks. So what single-headed attention does in  <span class="math-container">$ d_{model} \times d_{model} $</span> matrix, multi-headed attention does in <span class="math-container">$ h $</span> numbers of <span class="math-container">$ d_{model} \times d_{k} $</span> matrices. Why not make one single <span class="math-container">$ d_{model} \times d_{model} $</span>  matrix from these <span class="math-container">$ h $</span> numbers of <span class="math-container">$ d_{model} \times d_{k} $</span> matrices? and proceed as in single-headed attention. The only problem with that argument is we end up with one attention matrix instead  <span class="math-container">$h$</span> attention matrices. )</p>
  <p>These <span class="math-container">$ h $</span> attention matrices help in modeling different relationships between words at different positions. The original paper mentions</p>
  <p>&quot;Multi-head attention allows the model to jointly attend to information from different representation
  subspaces at different positions. With a single attention head, averaging inhibits this.&quot;</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-15T15:40:33.107Z
Answers.Text: I meant that single headed attention is in concept only and in actual implementation the Q, K, V blocks for a single layer is just a single matrix. For example if single Q matrix is 10x8 and there are 8 heads then the actual Q matrix is just 10x64. Same for K and V and the forward operations are applied. Then in the end we have seq_lenx64 which is multiplied by 64x8 to get the final dimensions correct. So why isn't it just considered a single weight instead of multi-headed?
Answers.UserDisplayName: Tarique
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-16T02:03:42.083Z
Answers.Text: Work out the dimensions of your attention matrix. Imagine you have 10x8 Q,K, V matrices. Then dimension of your attention matrix is arrived as ($ KQ^T $ ) (10 x 8) x (8x10) = 10 x 10. That is the size of your attention matrix for a single head. Do it for 8 heads and you get 8 numbers of 10 x 10 attention matrices. Now , because these 10 x8 matrices can be treated as blocks of a single matrix, you have one 10 x 64 matrix for K. Same applies for Q and V. Now work out the dimensions of your attention matrix (10 x64) x (10 x 64) = 10 x 10. You get only one attention matrix of dimension 10 x10.
Answers.UserDisplayName: Sathishkumar Thirumalai
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-16T04:24:11.227Z
Answers.Text: Each attention matrix is used to generate a part of the output in multi-headed attention. All such outputs are concatenated to produce the final output. This is not the same output as one full sized output generated using one attention matrix in single headed case. So both in actual implementation and concept, multi-headed attention is different from single-headed attention.
Answers.UserDisplayName: Sathishkumar Thirumalai
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-15T06:19:47.747Z
Answers.LastActivityDate: 2023-11-15T06:22:36.160Z
Answers.LastEditDate: 2023-11-15T06:22:36.160Z
Answers.UserDisplayName: Sathishkumar Thirumalai
Body: <p>Why do we call the attention layer in transformers <strong>multi-headed</strong> attention when in practice all the attention matrices from different heads <strong>(W,K,V)</strong> for a single layer are concatenated to perform the calculation in one go and then the result is multiplied by another matrix Wo to get the dimensions required?</p>
  <p>My understanding of self-attention calculation is based on the following blog: <a href="https://jalammar.github.io/illustrated-transformer/" rel="nofollow noreferrer">https://jalammar.github.io/illustrated-transformer/</a></p>
  
Tags: <neural-networks><terminology><transformer><attention>
Title: Why is it called multi-headed attention?
UserDisplayName: Tarique
ThreadText: # Why is it called multi-headed attention?
  
  (Asked by: username_0 on 2023-11-12)
  
  Why do we call the attention layer in transformers **multi-headed** attention when in practice all the attention matrices from different heads **(W,K,V)** for a single layer are concatenated to perform the calculation in one go and then the result is multiplied by another matrix Wo to get the dimensions required?
  
  My understanding of self-attention calculation is based on the following blog: <https://jalammar.github.io/illustrated-transformer/>
  
  **Tags:** neural-networks, terminology, transformer, attention
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-15. Score: 2 | Accepted answer
  
  The original paper "Attention is all you need" mentions the following.
  
  "
  Instead of performing a single attention function with $ d\_{model} $-dimensional keys, values and queries,
  we found it beneficial to linearly project the queries, keys and values $h$ times with different, learned
  linear projections to $d\_k$, $d\_k$ and $d\_v$ dimensions, respectively. "
  
  What is single-headed attention?
  
  The input matrix $ X $ of dimension $ T \times d\_{model} $ is multipled with $ W^Q, W^K $ and $W^V $ matrices of dimension $ d\_{model} \times d\_{model} $ to give output of dimension $ T \times d\_{model} $ Now, while calculating attention $ A=softmax(\frac{QK^{T}}{\sqrt{d\_k}}) $yields one attention matrix of dimension $ T \times T $.
  
  What is multi-headed attention?
  
  The input matrix $ X $ of dimension $ T \times d\_{model} $ is multipled with $ W^Q, W^K $ and $W^V $ matrices of dimension $ d\_{model} \times d\_{k} $ to give output of dimension $ T \times d\_{k} $. Note that $d\_{model} = h \times d\_{k} $ where $ h $ is the number of heads. Now, while calculating attention $ A=softmax(\frac{QK^{T}}{\sqrt{d\_k}}) $yields $h$ attention matrices of dimension $ T \times T $.
  
  It is called multi-headed attention because each head produces one different attention matrix.
  
  (I feel your confusion may have come from the knowledge that matrix multiplication happens in blocks. So what single-headed attention does in $ d\_{model} \times d\_{model} $ matrix, multi-headed attention does in $ h $ numbers of $ d\_{model} \times d\_{k} $ matrices. Why not make one single $ d\_{model} \times d\_{model} $ matrix from these $ h $ numbers of $ d\_{model} \times d\_{k} $ matrices? and proceed as in single-headed attention. The only problem with that argument is we end up with one attention matrix instead $h$ attention matrices. )
  
  These $ h $ attention matrices help in modeling different relationships between words at different positions. The original paper mentions
  
  "Multi-head attention allows the model to jointly attend to information from different representation
  subspaces at different positions. With a single attention head, averaging inhibits this."
  
  > **Comments:**
  > **username_0**: I meant that single headed attention is in concept only and in actual implementation the Q, K, V blocks for a single layer is just a single matrix. For example if single Q matrix is 10x8 and there are 8 heads then the actual Q matrix is just 10x64. Same for K and V and the forward operations are applied. Then in the end we have seq_lenx64 which is multiplied by 64x8 to get the final dimensions correct. So why isn't it just considered a single weight instead of multi-headed?  
  > **username_1**: Work out the dimensions of your attention matrix. Imagine you have 10x8 Q,K, V matrices. Then dimension of your attention matrix is arrived as ($ KQ^T $ ) (10 x 8) x (8x10) = 10 x 10. That is the size of your attention matrix for a single head. Do it for 8 heads and you get 8 numbers of 10 x 10 attention matrices. Now , because these 10 x8 matrices can be treated as blocks of a single matrix, you have one 10 x 64 matrix for K. Same applies for Q and V. Now work out the dimensions of your attention matrix (10 x64) x (10 x 64) = 10 x 10. You get only one attention matrix of dimension 10 x10.  
  > **username_1**: Each attention matrix is used to generate a part of the output in multi-headed attention. All such outputs are concatenated to produce the final output. This is not the same output as one full sized output generated using one attention matrix in single headed case. So both in actual implementation and concept, multi-headed attention is different from single-headed attention.  
  
  ---
  
  