Answers.Body: <p>For certain types of layers, such as transformers and convolutional layers, including a bias term is unnecessary and adds unnecessary overhead to the model.</p>
  <p>The reason for this is that these layers are typically followed by a normalization layer, such as Batch Normalization or Layer Normalization. These normalization layers center the data at mean=0 (and std=1), effectively removing any bias.</p>
  <p>Therefore, it is common practice to omit the bias term in transformers and convolutional layers that are preceded by a normalization layer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-30T20:27:55.203Z
Answers.Text: "preceded by a normalization layer", preceded or followed by? So, are you saying that batch normalization comes before or after the attention layer? Moreover, can you also clarify why centering the data in that way removes "biases"? You need to define bias precisely in order for that to make sense.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-30T23:18:43.703Z
Answers.Text: I'm saying that the batch layer comes after the attention layer. Corrected it. 
  As for why centering the data in that way removes "biases": x - mean(x) == (x+bias) - mean(x+bias). Same for std. The normalisation layer cancels out the bias term.
Answers.UserDisplayName: Marc Dumon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-01T11:38:30.310Z
Answers.Text: How is your definition of bias related to the bias term in a layer? It seems that you're implying that the bias term in a layer would be 1. different for each $x$ (i.e. changes from input to input) and 2. the difference between each x and the mean of the distribution of the observations. What's the motivation for these implications?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-30T02:18:00.510Z
Answers.Text: The reasonings in this answer and comments are incorrect. First, the query, key, and value multiplications are not typically followed by normalization. They're typically followed by multiplication with a projection matrix that assembles the outputs from multiple attention heads. Second, the normalization is a layer normalization (LN) which does not cancel bias. The bias itself is a vector, just like the input x to the LN layer. This means x - mean(x) != (x+bias) - mean(x+bias) because bias - mean(bias) is non-zero.
Answers.UserDisplayName: Fijoy Vadakkumpadan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-30T19:43:52.820Z
Answers.LastActivityDate: 2023-04-30T23:03:05.583Z
Answers.LastEditDate: 2023-04-30T23:03:05.583Z
Answers.UserDisplayName: Marc Dumon
Body: <p>Watching <a href="https://youtu.be/kCc8FmEb1nY?t=4767" rel="nofollow noreferrer">this video</a> implementing attention in a transformer. He set query, key, and value biases to <code>False</code> and said &quot;Typically, people don't use biases for these&quot;.</p>
  <p>Even in <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/activation.html#MultiheadAttention" rel="nofollow noreferrer">official PyTorch code</a> the default bias is <code>False</code>:</p>
  <blockquote>
  <p>add_bias_kv: If specified, adds bias to the key and value sequences at dim=0. Default: <code>False</code>.</p>
  </blockquote>
  <p>What is the reason behind that?</p>
  
Tags: <neural-networks><deep-learning><natural-language-processing><transformer><attention>
Title: Why are biases (typically) not used in attention mechanism?
UserDisplayName: Peyman
ThreadText: # Why are biases (typically) not used in attention mechanism?
  
  (Asked by: username_0 on 2023-04-30)
  
  Watching [this video](https://youtu.be/kCc8FmEb1nY?t=4767) implementing attention in a transformer. He set query, key, and value biases to `False` and said "Typically, people don't use biases for these".
  
  Even in [official PyTorch code](https://pytorch.org/docs/stable/_modules/torch/nn/modules/activation.html#MultiheadAttention) the default bias is `False`:
  
  > add\_bias\_kv: If specified, adds bias to the key and value sequences at dim=0. Default: `False`.
  
  What is the reason behind that?
  
  **Tags:** neural-networks, deep-learning, natural-language-processing, transformer, attention
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-30. Score: 10 | Accepted answer
  
  For certain types of layers, such as transformers and convolutional layers, including a bias term is unnecessary and adds unnecessary overhead to the model.
  
  The reason for this is that these layers are typically followed by a normalization layer, such as Batch Normalization or Layer Normalization. These normalization layers center the data at mean=0 (and std=1), effectively removing any bias.
  
  Therefore, it is common practice to omit the bias term in transformers and convolutional layers that are preceded by a normalization layer.
  
  > **Comments:**
  > **username_2**: "preceded by a normalization layer", preceded or followed by? So, are you saying that batch normalization comes before or after the attention layer? Moreover, can you also clarify why centering the data in that way removes "biases"? You need to define bias precisely in order for that to make sense.  
  > **username_1**: I'm saying that the batch layer comes after the attention layer. Corrected it.  As for why centering the data in that way removes "biases": x - mean(x) == (x+bias) - mean(x+bias). Same for std. The normalisation layer cancels out the bias term.  
  > **username_2**: How is your definition of bias related to the bias term in a layer? It seems that you're implying that the bias term in a layer would be 1. different for each $x$ (i.e. changes from input to input) and 2. the difference between each x and the mean of the distribution of the observations. What's the motivation for these implications?  
  > **username_3**: The reasonings in this answer and comments are incorrect. First, the query, key, and value multiplications are not typically followed by normalization. They're typically followed by multiplication with a projection matrix that assembles the outputs from multiple attention heads. Second, the normalization is a layer normalization (LN) which does not cancel bias. The bias itself is a vector, just like the input x to the LN layer. This means x - mean(x) != (x+bias) - mean(x+bias) because bias - mean(bias) is non-zero.  
  
  ---
  
  