Answers.Body: <p>In multi-head attention the Keys, Values and Queries are chunked along the &quot;channel&quot; dimension, you need to apply a linear layer before you do this.</p>
  <p>What I wanted to do wouldn't work because you would be chunking <span class="math-container">$X$</span>. The chunks of <span class="math-container">$Q,K,V$</span> depend on all the channels of <span class="math-container">$X$</span>, where as the chunks of <span class="math-container">$X$</span> do not.</p>
  <p>There is code <a href="http://einops.rocks/pytorch-examples.html" rel="nofollow noreferrer">here</a> to see exactly what is happening with multi-head attention:</p>
  <pre><code>q = rearrange(self.w_qs(q), 'b l (head k) -&gt; head b l k', head=self.n_head)
  k = rearrange(self.w_ks(k), 'b t (head k) -&gt; head b t k', head=self.n_head)
  v = rearrange(self.w_vs(v), 'b t (head v) -&gt; head b t v', head=self.n_head)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-12T00:30:36.923Z
Answers.LastActivityDate: 2022-04-12T00:30:36.923Z
Answers.UserDisplayName: Tom Huntington
Body: <p>Transformer attention is calculated <span class="math-container">$Attention(X) =X W^V\times \text{columnwise-softmax} (Att(X))$</span> where the attention attention matrix is <span class="math-container">$$Att(X) = Q \times K = {X W}^Q \times ({X W}^K)^T = {X W}^Q (W^K)^T X^T $$</span></p>
  <p>But then couldn't one of <span class="math-container">$W^Q$</span> and <span class="math-container">$W^K$</span> be absorbed into the other? So one of them is redundant say <span class="math-container">$W^K$</span> and so attention only needs the parameters from <span class="math-container">$W^Q$</span> and <span class="math-container">$W^V$</span>.</p>
  <p><span class="math-container">$$Attention(X) = {XW}^V \times \text{columnwise-softmax}({XW}^Q  X^T).$$</span></p>
  <p><a href="https://jalammar.github.io/illustrated-transformer/" rel="nofollow noreferrer">Diagrams</a> I use for thinking.</p>
  <p><img src="https://jalammar.github.io/images/t/self-attention-matrix-calculation.png" alt="" /></p>
  <p><img src="https://jalammar.github.io/images/t/self-attention-matrix-calculation-2.png" alt="" /></p>
  
Tags: <transformer><attention>
Title: Attention: Isn't it redundant to apply a linear layer to both the keys and values?
UserDisplayName: Tom Huntington
ThreadText: # Attention: Isn't it redundant to apply a linear layer to both the keys and values?
  
  (Asked by: username_0 on 2022-04-11)
  
  Transformer attention is calculated $Attention(X) =X W^V\times \text{columnwise-softmax} (Att(X))$ where the attention attention matrix is $$Att(X) = Q \times K = {X W}^Q \times ({X W}^K)^T = {X W}^Q (W^K)^T X^T $$
  
  But then couldn't one of $W^Q$ and $W^K$ be absorbed into the other? So one of them is redundant say $W^K$ and so attention only needs the parameters from $W^Q$ and $W^V$.
  
  $$Attention(X) = {XW}^V \times \text{columnwise-softmax}({XW}^Q X^T).$$
  
  [Diagrams](https://jalammar.github.io/illustrated-transformer/) I use for thinking.
  
  ![](https://jalammar.github.io/images/t/self-attention-matrix-calculation.png)
  
  ![](https://jalammar.github.io/images/t/self-attention-matrix-calculation-2.png)
  
  **Tags:** transformer, attention
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-04-12. Score: 2 | Accepted answer
  
  In multi-head attention the Keys, Values and Queries are chunked along the "channel" dimension, you need to apply a linear layer before you do this.
  
  What I wanted to do wouldn't work because you would be chunking $X$. The chunks of $Q,K,V$ depend on all the channels of $X$, where as the chunks of $X$ do not.
  
  There is code [here](http://einops.rocks/pytorch-examples.html) to see exactly what is happening with multi-head attention:
  
  ```
  q = rearrange(self.w_qs(q), 'b l (head k) -> head b l k', head=self.n_head)
  k = rearrange(self.w_ks(k), 'b t (head k) -> head b t k', head=self.n_head)
  v = rearrange(self.w_vs(v), 'b t (head v) -> head b t v', head=self.n_head)
  
  ```
  
  
  ---
  
  