Answers.Body: <p>I'll use notation from the paper you cited, and any other readers should refer to the paper (widely available) for definitions of notation.  The utility of using <span class="math-container">$W^Q$</span> and <span class="math-container">$W^K$</span>, rather than <span class="math-container">$W$</span>, lies in the fact that they allow us to add fewer parameters to our architecture.  <span class="math-container">$W$</span> has dimension <span class="math-container">$d_{model} \times d_{model}$</span>, which means that we are adding <span class="math-container">$d_{model}^2$</span> parameters to our architecture.  <span class="math-container">$W^Q$</span> and <span class="math-container">$W^K$</span> each have dimension <span class="math-container">$d_{model} \times d_k$</span>, and <span class="math-container">$d_k=\frac{d_{model}}{h}$</span>. If we use these two matrices, we only add <span class="math-container">$2\frac{d_{model}^2}{h}$</span> parameters to our architecture, even though their multiplication (with the transpose) allows us to have the correct dimensions for matrix multiplication with <span class="math-container">$Q$</span> and <span class="math-container">$K$</span>.</p>
  <p>We do use <span class="math-container">$h$</span> attention heads, which then brings our number of parameters back up, but the multiple heads let the model attend to different pieces of information in our data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-17T05:42:06.173Z
Answers.Text: I understand now, actually the resulting $W$ matrix is rank deficient when defined by inner product of $W^Q$ and $W^K$. This makes parameters fewer :) Thanks.
Answers.UserDisplayName: Uğurcan Özalp
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-17T17:59:09.797Z
Answers.Text: I just read a paper (https://arxiv.org/pdf/1906.07510.pdf) in which the authors use multi-head attention from Vaswani et al, but have $W^Q$ and $W^K$ be of dimension $d_{model} \times d_{model}$.  I don't know if there is a reason for this, or if it's a typo.  I plan to email them, and if there is a reason for it, I'll post it here.
Answers.UserDisplayName: BioBroo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-25T20:20:58.793Z
Answers.Text: @BioBroo Might be a dumb question, but do these projections render the embedding (linear) layer redundant?
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-16T20:53:41.610Z
Answers.LastActivityDate: 2020-12-16T21:01:58.257Z
Answers.LastEditDate: 2020-12-16T21:01:58.257Z
Answers.UserDisplayName: BioBroo
Answers.Body: <p>In practice, matrices <span class="math-container">$W^Q, W^K, W^V$</span> (each of size <span class="math-container">$d_{model}$</span> x <span class="math-container">$d_{model}$</span>) are completely removed instead, and Transformer implementations just learn a single set of matrices <span class="math-container">$\{ W_i^{Q*}, W_i^{K*}, W_i^{V*} \}$</span> (each of size <span class="math-container">$d_{model}$</span> x <span class="math-container">$\frac{d_{model}}{h}$</span>) for each head, where</p>
  <p><span class="math-container">$W_i^{Q*} = W^Q W_i^Q \\
  W_i^{K*} = W^K W_i^K \\
  W_i^{V*} = W^V W_i^V $</span></p>
  <p>so that:</p>
  <p><span class="math-container">$Q_i(x) = x W_i^{Q*} = x W^Q W_i^{Q*} = Q W_i^Q\\
  K_i(x) = x W_i^{K*} = x W^K W_i^{K*} = K W_i^K\\
  V_i(x) = x W_i^{V*} = x W^V W_i^{V*} = V W_i^V\\
  head_i(x) = softmax \left(\frac{Q_i(x) K_i(x)^T}{\sqrt{d_k}} \right) V_i(x)$</span>.</p>
  <p>I can confirm this with the original Transformer implementation in Tensor2Tensor, and also the BERT code that uses the encoding part of the Transformer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-17T05:42:56.713Z
Answers.Text: I missed the part where $d_k=d_{model} / h$ . Thank you.
Answers.UserDisplayName: Uğurcan Özalp
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-17T11:19:18.743Z
Answers.Text: I just stumbled upon this Question after not understanding the original paper. $$
  \text { where head }_{\mathrm{i}}=\text { Attention }\left(Q W_{i}^{Q}, K W_{i}^{K}, V W_{i}^{V}\right)
  $$I would really like to know if this redundant now and it can be$$ \text { head }_{\mathrm{i}}=\text { Attention }\left(Q_{i}, K_{i}, V_{i}\right)$$ where $$Q_{i}, K_{i}, V_{i} $$are simply generated from the input to the layer and the matrices $$
  \left\{W_{i}^{Q }, W_{i}^{K }, W_{i}^{V }\right\}\in \mathbb{R} ^{D\times \dfrac{D}{h}}
  $$ respectively
Answers.UserDisplayName: JimSi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-17T00:57:52.470Z
Answers.LastActivityDate: 2020-12-17T00:57:52.470Z
Answers.UserDisplayName: user3667125
Body: <p>In the <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">Attention is all you need</a> paper, on the 4th page, we have equation 1, which describes the self-attention mechanism of the transformer architecture</p>
  <p><span class="math-container">$$
  \text { Attention }(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^{T}}{\sqrt{d_{k}}}\right) V
  $$</span></p>
  <p>Everything is fine up to here.</p>
  <p>Then they introduce the multi-head attention, which is described by the following equation.</p>
  <p><span class="math-container">$$
  \begin{aligned}
  \text { MultiHead }(Q, K, V) &amp;=\text { Concat}\left(\text {head}_{1}, \ldots, \text {head}_{\mathrm{h}}\right) W^{O} \\
  \text { where head}_{\mathrm{i}} &amp;=\text {Attention}\left(Q W_{i}^{Q}, K W_{i}^{K}, V W_{i}^{V}\right)
  \end{aligned}
  $$</span></p>
  <p>Once the multi-head attention is motivated at the end of page 4, they state that for a single head (the <span class="math-container">$i$</span>th head), the query <span class="math-container">$Q$</span> and key <span class="math-container">$K$</span> inputs are first linearly projected by <span class="math-container">$W_i^Q$</span> and <span class="math-container">$W_i^K$</span>, then dot product is calculated, let's say <span class="math-container">$Q_i^p =  Q W_i^Q$</span> and <span class="math-container">$K_i^p = K W_i^K$</span>.</p>
  <p>Therefore, the dot product of the projected query and key becomes the following from simple linear algebra.</p>
  <p><span class="math-container">$$Q_i^p {K_i^p}^\intercal = Q W_i^Q {W_i^K}^T K^T =  Q W_i K^T,$$</span></p>
  <p>where</p>
  <p><span class="math-container">$$W_i = W_i^Q {W_i^K}^T$$</span></p>
  <p>Here, <span class="math-container">$W$</span> is the outer product of query projection by the key projection matrix. However, it is a matrix with shape <span class="math-container">$d_{model} \times d_{model}$</span>. Why did the authors not define only a <span class="math-container">$W_i$</span> instead of <span class="math-container">$W_i^Q$</span> and <span class="math-container">$W_i^K$</span> pair which have <span class="math-container">$2 \times d_{model} \times d_{k}$</span> elements? In deep learning applications, I think it would be very inefficient.</p>
  <p>Is there something that I am missing, like these 2 matrices <span class="math-container">$W_i^Q$</span> and <span class="math-container">$W_i^K$</span> should be separate because of this and that?</p>
  
Tags: <deep-learning><transformer><attention><weights><efficiency>
Title: In the multi-head attention mechanism of the transformer, why do we need both $W_i^Q$ and ${W_i^K}^T$?
UserDisplayName: Uğurcan Özalp
ThreadText: # In the multi-head attention mechanism of the transformer, why do we need both $W_i^Q$ and ${W_i^K}^T$?
  
  (Asked by: username_0 on 2020-12-16)
  
  In the [Attention is all you need](https://arxiv.org/pdf/1706.03762.pdf) paper, on the 4th page, we have equation 1, which describes the self-attention mechanism of the transformer architecture
  
  $$
  \text { Attention }(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^{T}}{\sqrt{d\_{k}}}\right) V
  $$
  
  Everything is fine up to here.
  
  Then they introduce the multi-head attention, which is described by the following equation.
  
  $$
  \begin{aligned}
  \text { MultiHead }(Q, K, V) &=\text { Concat}\left(\text {head}\_{1}, \ldots, \text {head}\_{\mathrm{h}}\right) W^{O} \\
  \text { where head}\_{\mathrm{i}} &=\text {Attention}\left(Q W\_{i}^{Q}, K W\_{i}^{K}, V W\_{i}^{V}\right)
  \end{aligned}
  $$
  
  Once the multi-head attention is motivated at the end of page 4, they state that for a single head (the $i$th head), the query $Q$ and key $K$ inputs are first linearly projected by $W\_i^Q$ and $W\_i^K$, then dot product is calculated, let's say $Q\_i^p = Q W\_i^Q$ and $K\_i^p = K W\_i^K$.
  
  Therefore, the dot product of the projected query and key becomes the following from simple linear algebra.
  
  $$Q\_i^p {K\_i^p}^\intercal = Q W\_i^Q {W\_i^K}^T K^T = Q W\_i K^T,$$
  
  where
  
  $$W\_i = W\_i^Q {W\_i^K}^T$$
  
  Here, $W$ is the outer product of query projection by the key projection matrix. However, it is a matrix with shape $d\_{model} \times d\_{model}$. Why did the authors not define only a $W\_i$ instead of $W\_i^Q$ and $W\_i^K$ pair which have $2 \times d\_{model} \times d\_{k}$ elements? In deep learning applications, I think it would be very inefficient.
  
  Is there something that I am missing, like these 2 matrices $W\_i^Q$ and $W\_i^K$ should be separate because of this and that?
  
  **Tags:** deep-learning, transformer, attention, weights, efficiency
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-12-16. Score: 2 | Accepted answer
  
  I'll use notation from the paper you cited, and any other readers should refer to the paper (widely available) for definitions of notation. The utility of using $W^Q$ and $W^K$, rather than $W$, lies in the fact that they allow us to add fewer parameters to our architecture. $W$ has dimension $d\_{model} \times d\_{model}$, which means that we are adding $d\_{model}^2$ parameters to our architecture. $W^Q$ and $W^K$ each have dimension $d\_{model} \times d\_k$, and $d\_k=\frac{d\_{model}}{h}$. If we use these two matrices, we only add $2\frac{d\_{model}^2}{h}$ parameters to our architecture, even though their multiplication (with the transpose) allows us to have the correct dimensions for matrix multiplication with $Q$ and $K$.
  
  We do use $h$ attention heads, which then brings our number of parameters back up, but the multiple heads let the model attend to different pieces of information in our data.
  
  > **Comments:**
  > **username_0**: I understand now, actually the resulting $W$ matrix is rank deficient when defined by inner product of $W^Q$ and $W^K$. This makes parameters fewer :) Thanks.  
  > **username_1**: I just read a paper (https://arxiv.org/pdf/1906.07510.pdf) in which the authors use multi-head attention from Vaswani et al, but have $W^Q$ and $W^K$ be of dimension $d_{model} \times d_{model}$.  I don't know if there is a reason for this, or if it's a typo.  I plan to email them, and if there is a reason for it, I'll post it here.  
  > **username_2**: @BioBroo Might be a dumb question, but do these projections render the embedding (linear) layer redundant?  
  
  ---
  
  ### Answer by username_3 on 2020-12-17. Score: 0
  
  In practice, matrices $W^Q, W^K, W^V$ (each of size $d\_{model}$ x $d\_{model}$) are completely removed instead, and Transformer implementations just learn a single set of matrices $\{ W\_i^{Q\*}, W\_i^{K\*}, W\_i^{V\*} \}$ (each of size $d\_{model}$ x $\frac{d\_{model}}{h}$) for each head, where
  
  $W\_i^{Q\*} = W^Q W\_i^Q \\
  W\_i^{K\*} = W^K W\_i^K \\
  W\_i^{V\*} = W^V W\_i^V $
  
  so that:
  
  $Q\_i(x) = x W\_i^{Q\*} = x W^Q W\_i^{Q\*} = Q W\_i^Q\\
  K\_i(x) = x W\_i^{K\*} = x W^K W\_i^{K\*} = K W\_i^K\\
  V\_i(x) = x W\_i^{V\*} = x W^V W\_i^{V\*} = V W\_i^V\\
  head\_i(x) = softmax \left(\frac{Q\_i(x) K\_i(x)^T}{\sqrt{d\_k}} \right) V\_i(x)$.
  
  I can confirm this with the original Transformer implementation in Tensor2Tensor, and also the BERT code that uses the encoding part of the Transformer.
  
  > **Comments:**
  > **username_0**: I missed the part where $d_k=d_{model} / h$ . Thank you.  
  > **username_4**: I just stumbled upon this Question after not understanding the original paper. $$ \text { where head }_{\mathrm{i}}=\text { Attention }\left(Q W_{i}^{Q}, K W_{i}^{K}, V W_{i}^{V}\right) $$I would really like to know if this redundant now and it can be$$ \text { head }_{\mathrm{i}}=\text { Attention }\left(Q_{i}, K_{i}, V_{i}\right)$$ where $$Q_{i}, K_{i}, V_{i} $$are simply generated from the input to the layer and the matrices $$ \left\{W_{i}^{Q }, W_{i}^{K }, W_{i}^{V }\right\}\in \mathbb{R} ^{D\times \dfrac{D}{h}} $$ respectively  
  
  ---
  
  