Answers.Body: <p>The technical reason is the residual connection around the self-attention block (first line in your code: <code>x += self_attention(x)</code>). The transformation <span class="math-container">$W_O$</span> is required, because it projects the vector of concatenated heads of dimension <span class="math-container">$n_{head} \times d_{head}$</span> back to dimension <span class="math-container">$d_{head}$</span>. Without it, the operation <code>x += self_attention(x)</code> is not possible, because <code>x</code> and <code>self_attention(x)</code> would have different dimensionalities. Regarding the necessity of the residual connection.</p>
  <p>Now the question becomes, is the residual connection necessary? It helps to carry the positional encoding deeper through the network and improves gradient propagation. While the first point is to be taken with a grain of salt (because I cannot find the references for that anymore) the second point is a well-known property of residual connections.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-21T22:43:32.693Z
Answers.Text: If I am not mistaken, the dimensionality of each head is $d_{input}/n$ where $n$ is the number of heads. Meaning that after concatenation, residual blocks apply without problem. Why is then necessary this projection block?
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-12T12:20:59.950Z
Answers.LastActivityDate: 2024-02-12T12:20:59.950Z
Answers.UserDisplayName: Chillston
Body: <p>As I understand it, the forward pass for a transformer model looks as follows:</p>
  <pre><code>x += self_attention(x)
  x = layernorm(x)
  x += ffn(x)
  </code></pre>
  <p>Breaking that down a bit (excuse the hand-waving, this is meant to be illustrative):</p>
  <pre><code>def self_attention(x):
    qkv_list = get_qkvs(x)
    heads = [softmax(q @ t(v) / sqrt(d))@v for (q,k,v) in qkv_list]
    concatenated_heads = concat(heads)
    projected_values = concatenated_heads @ W_O # projection matrix for concatenated heads
    return projected_values
  </code></pre>
  <p>and</p>
  <pre><code>def ffn(x):
    return (relu(x @ W_1 + b1))@W_2 + b2
  </code></pre>
  <p>Perhaps I am missing something obvious, but I note that if one were to drop the layernorm in between the self-attention and FFN module, you have two linear projections in a row (<span class="math-container">$W_O$</span>, <span class="math-container">$W_1$</span>). <span class="math-container">$W_O$</span> is a costly matrix: it should be dimensions (n_head * d_head x d_embed) = (d_embed x d_embed), so contributes (n_layers * d_embed^2) parameters over the course of the network.</p>
  <p>My question is: if you drop the layernorm, it appears the <span class="math-container">$W_O$</span> matrix would be entirely redundant, and you could save a massive amount of compute by essentially fusing these two operations into a single learned matrix (that matrix would both be mixing the streams from the different heads like <span class="math-container">$W_O$</span> does, and up-projecting into the FFN embedding layer like <span class="math-container">$W_1$</span> does). This means either:</p>
  <ul>
  <li>The layernorm is incredibly important, and I assume this has been shown somewhere empirically?</li>
  <li>I've missed something completely obvious about the forward pass / even if you dropped the layernorm the <span class="math-container">$W_O$</span> matrix is not redundant for some reason</li>
  </ul>
  <p>Would love to be corrected!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-22T08:30:30.337Z
Comments.Text: The technical reason is the residual connection around the self-attention block. The transformation $W_O$ is required, because it projects the vector of concatenated heads $n_{head} * d_head$ back to $1 * d_head$. Without it, the operation $x = x + self_attention(x)$ is not possible, because $x$ and $self_attention(x)$ would have different dimensionalities. Also, the residual connection is not redundant, as it helps carry the positional encoding through the network and improves gradient propagation.
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-22T14:35:34.057Z
Comments.Text: @Chillston that's the correct answer, consider writing one
Comments.UserDisplayName: Alberto
Tags: <transformer><attention><performance><linear-algebra>
Title: Is the multi-headed projection matrix in self-attention redundant?
UserDisplayName: Sue Doh Nimh
ThreadText: # Is the multi-headed projection matrix in self-attention redundant?
  
  (Asked by: username_0 on 2024-01-21)
  
  As I understand it, the forward pass for a transformer model looks as follows:
  
  ```
  x += self_attention(x)
  x = layernorm(x)
  x += ffn(x)
  
  ```
  
  Breaking that down a bit (excuse the hand-waving, this is meant to be illustrative):
  
  ```
  def self_attention(x):
    qkv_list = get_qkvs(x)
    heads = [softmax(q @ t(v) / sqrt(d))@v for (q,k,v) in qkv_list]
    concatenated_heads = concat(heads)
    projected_values = concatenated_heads @ W_O # projection matrix for concatenated heads
    return projected_values
  
  ```
  
  and
  
  ```
  def ffn(x):
    return (relu(x @ W_1 + b1))@W_2 + b2
  
  ```
  
  Perhaps I am missing something obvious, but I note that if one were to drop the layernorm in between the self-attention and FFN module, you have two linear projections in a row ($W\_O$, $W\_1$). $W\_O$ is a costly matrix: it should be dimensions (n\_head \* d\_head x d\_embed) = (d\_embed x d\_embed), so contributes (n\_layers \* d\_embed^2) parameters over the course of the network.
  
  My question is: if you drop the layernorm, it appears the $W\_O$ matrix would be entirely redundant, and you could save a massive amount of compute by essentially fusing these two operations into a single learned matrix (that matrix would both be mixing the streams from the different heads like $W\_O$ does, and up-projecting into the FFN embedding layer like $W\_1$ does). This means either:
  
  * The layernorm is incredibly important, and I assume this has been shown somewhere empirically?
  * I've missed something completely obvious about the forward pass / even if you dropped the layernorm the $W\_O$ matrix is not redundant for some reason
  
  Would love to be corrected!
  
  **Tags:** transformer, attention, performance, linear-algebra
  
  > **Comments:**
  > **username_1**: The technical reason is the residual connection around the self-attention block. The transformation $W_O$ is required, because it projects the vector of concatenated heads $n_{head} * d_head$ back to $1 * d_head$. Without it, the operation $x = x + self_attention(x)$ is not possible, because $x$ and $self_attention(x)$ would have different dimensionalities. Also, the residual connection is not redundant, as it helps carry the positional encoding through the network and improves gradient propagation.  
  > **username_2**: @username_1 that's the correct answer, consider writing one  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-12. Score: 1
  
  The technical reason is the residual connection around the self-attention block (first line in your code: `x += self_attention(x)`). The transformation $W\_O$ is required, because it projects the vector of concatenated heads of dimension $n\_{head} \times d\_{head}$ back to dimension $d\_{head}$. Without it, the operation `x += self_attention(x)` is not possible, because `x` and `self_attention(x)` would have different dimensionalities. Regarding the necessity of the residual connection.
  
  Now the question becomes, is the residual connection necessary? It helps to carry the positional encoding deeper through the network and improves gradient propagation. While the first point is to be taken with a grain of salt (because I cannot find the references for that anymore) the second point is a well-known property of residual connections.
  
  > **Comments:**
  > **username_3**: If I am not mistaken, the dimensionality of each head is $d_{input}/n$ where $n$ is the number of heads. Meaning that after concatenation, residual blocks apply without problem. Why is then necessary this projection block?  
  
  ---
  
  