Answers.Body: <p>First of all, I believe that in self-attention mechanism for Query, Key and Value vectors the different linear transformations are used,
  <span class="math-container">$$
  Q = XW_Q,\,K = XW_K,\,V = XW_V; W_Q \neq W_K, W_K \neq W_V, W_Q \neq W_V
  $$</span>
  The self-attention itself is a way of using more general attention mechanism. You can check <a href="https://lilianweng.github.io/posts/2018-06-24-attention/" rel="nofollow noreferrer">this post</a> for examples of other mechanisms that can be used.</p>
  <p>Multi-head attention is a way of grouping together a bunch of attention mechanism ( Usually they are all the same type ), which consists in just running multiple mechanism in parallel and aggregating the resulting set in some way. In <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">here</a> for example, the aggregation is done by concatenation and weighted sum of the outputs, e.g.
  <span class="math-container">\begin{equation*}
  \begin{aligned}
  &amp;\text{MultiHead}(Q, K, V)= [\text{head}_1, .., \text{head}_n]W^O &amp; [..] \text{ meaning concatenation}\\
  &amp;\text{where head}_i = \text{Attention}(QW_i^Q,KW_i^K,VW_i^V) &amp; \text{each head}_i\text{ is unique attention mechanism} \\
  &amp;\text{Attention}(\textbf{Q},\textbf{K},\textbf{V}) = \text{softmax}(\frac{\textbf{Q}\textbf{K}}{\sqrt{d_k}})\textbf{V} &amp; \text{this can be other attention mechanism}\\
  \end{aligned}
  \end{equation*}</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-07T11:42:21.677Z
Answers.Text: are you sure that they are different? In the following document people.tamu.edu/~sji/classes/attn-slides.pdf (slide 5) it is clearly stated that *in self-attention, we let Q=K=V=X *
Answers.UserDisplayName: James Arten
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-08T12:03:19.613Z
Answers.Text: Thanks for the comment, i checked it out.
  I think what happens is a misunderstanding in what Q, K, V are. In the slides you have provided they are seen as already transformed X that is passed to attention mechanism and transformed once again, kind of in the same way as I have established i-th head in the multi-head attention.
Answers.UserDisplayName: vl_knd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-08T12:21:16.860Z
Answers.Text: But since they are transformed again after being passed to the self attention, it is actually equivalent to what I have described as self attention. The only difference is that its applied to pre-transformed X.
  
  Imagine, that we are pre-transforming X to X*W. Now by applying the self attention I have described you are going to obtain the exact same result as in the slide 3 you have mentioned.
  I adopted this point of view, since this is the way its described in Attention is All you need paper.
  Check this for an example https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a
Answers.UserDisplayName: vl_knd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-26T12:48:36.947Z
Answers.LastActivityDate: 2022-03-08T12:08:00.520Z
Answers.LastEditDate: 2022-03-08T12:08:00.520Z
Answers.UserDisplayName: vl_knd
Body: <p>I'm trying to understand the difference between the concept of <em>self-attention</em> and <em>multi-head attention</em>. The latter is not actually too clear to me.</p>
  <p>I understand that, in the case of self-attention, we start with a feature matrix <span class="math-container">$X \in \mathbb{R}^{n \times d}$</span>, and then we use the same linear transformation <span class="math-container">$W$</span> to produce</p>
  <p><span class="math-container">\begin{align}
  Q &amp;= XW \\ 
  K &amp;= XW \\
  V &amp;= XW
  \end{align}</span></p>
  <p>and then we compute the following</p>
  <p><span class="math-container">$$X' = \text{softmax} \left(\frac{Q\cdot K^T}{\sqrt{d}} \right)V$$</span></p>
  <p>where <span class="math-container">$X' \in \mathbb{R}^{n \times d}$</span> is a new version of the input matrix, where the pairwise interactions between the points will be encoded.</p>
  <p>What is multi-head attention doing, from a mathematical point of view, and what's the difference? I know we are going to use three <em>different</em> linear transformations in this case (so no weight-sharing), but what exactly will be encoded using three different <span class="math-container">$W$</span>? Maybe it's more the conceptual view that it's not too clear in this case.</p>
  
Tags: <deep-learning><comparison><transformer><attention>
Title: What is multi-head attention doing mathematically, and how is it different from self-attention?
UserDisplayName: James Arten
ThreadText: # What is multi-head attention doing mathematically, and how is it different from self-attention?
  
  (Asked by: username_0 on 2022-02-26)
  
  I'm trying to understand the difference between the concept of *self-attention* and *multi-head attention*. The latter is not actually too clear to me.
  
  I understand that, in the case of self-attention, we start with a feature matrix $X \in \mathbb{R}^{n \times d}$, and then we use the same linear transformation $W$ to produce
  
  \begin{align}
  Q &= XW \\
  K &= XW \\
  V &= XW
  \end{align}
  
  and then we compute the following
  
  $$X' = \text{softmax} \left(\frac{Q\cdot K^T}{\sqrt{d}} \right)V$$
  
  where $X' \in \mathbb{R}^{n \times d}$ is a new version of the input matrix, where the pairwise interactions between the points will be encoded.
  
  What is multi-head attention doing, from a mathematical point of view, and what's the difference? I know we are going to use three *different* linear transformations in this case (so no weight-sharing), but what exactly will be encoded using three different $W$? Maybe it's more the conceptual view that it's not too clear in this case.
  
  **Tags:** deep-learning, comparison, transformer, attention
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-02-26. Score: 2 | Accepted answer
  
  First of all, I believe that in self-attention mechanism for Query, Key and Value vectors the different linear transformations are used,
  $$
  Q = XW\_Q,\,K = XW\_K,\,V = XW\_V; W\_Q \neq W\_K, W\_K \neq W\_V, W\_Q \neq W\_V
  $$
  The self-attention itself is a way of using more general attention mechanism. You can check [this post](https://lilianweng.github.io/posts/2018-06-24-attention/) for examples of other mechanisms that can be used.
  
  Multi-head attention is a way of grouping together a bunch of attention mechanism ( Usually they are all the same type ), which consists in just running multiple mechanism in parallel and aggregating the resulting set in some way. In [here](https://arxiv.org/pdf/1706.03762.pdf) for example, the aggregation is done by concatenation and weighted sum of the outputs, e.g.
  \begin{equation\*}
  \begin{aligned}
  &\text{MultiHead}(Q, K, V)= [\text{head}\_1, .., \text{head}\_n]W^O & [..] \text{ meaning concatenation}\\
  &\text{where head}\_i = \text{Attention}(QW\_i^Q,KW\_i^K,VW\_i^V) & \text{each head}\_i\text{ is unique attention mechanism} \\
  &\text{Attention}(\textbf{Q},\textbf{K},\textbf{V}) = \text{softmax}(\frac{\textbf{Q}\textbf{K}}{\sqrt{d\_k}})\textbf{V} & \text{this can be other attention mechanism}\\
  \end{aligned}
  \end{equation\*}
  
  > **Comments:**
  > **username_0**: are you sure that they are different? In the following document people.tamu.edu/~sji/classes/attn-slides.pdf (slide 5) it is clearly stated that *in self-attention, we let Q=K=V=X *  
  > **username_1**: Thanks for the comment, i checked it out. I think what happens is a misunderstanding in what Q, K, V are. In the slides you have provided they are seen as already transformed X that is passed to attention mechanism and transformed once again, kind of in the same way as I have established i-th head in the multi-head attention.  
  > **username_1**: But since they are transformed again after being passed to the self attention, it is actually equivalent to what I have described as self attention. The only difference is that its applied to pre-transformed X.  Imagine, that we are pre-transforming X to X*W. Now by applying the self attention I have described you are going to obtain the exact same result as in the slide 3 you have mentioned. I adopted this point of view, since this is the way its described in Attention is All you need paper. Check this for an example https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a  
  
  ---
  
  