Answers.Body: <p>I guess that bias terms have the ability to increase the expression power of the model.According to the paper:&quot;BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models&quot;.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-29T13:51:25.050Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-18T09:58:01.070Z
Answers.LastActivityDate: 2023-06-18T09:58:01.070Z
Answers.UserDisplayName: ericcccw
Body: <p>In the <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">original transformer paper</a>, the attention mechanism uses parameter matrices, but no bias terms. However, in more recent implementations I see people often using a bias term when computing &quot;key&quot;, &quot;query&quot;, and &quot;value&quot;. For example, in Andrej Karpathy's recent <a href="https://github.com/karpathy/nanoGPT/blob/master/model.py" rel="nofollow noreferrer">implementation of GPT</a>, whether a bias term is used can be determined in the config:</p>
  <pre><code>bias: bool = True # True: bias in Linears and LayerNorms, like GPT-2. False: a bit better and faster
  </code></pre>
  <p>This makes me wonder whether there is any evidence that the bias terms help. In particular, if, according to Karpathy, not using bias is &quot;a bit better and faster&quot;, why is he using them by default?</p>
  
Tags: <transformer><bias>
Title: Is there any evidence that the bias terms help in the attention mechanism of the transformers?
UserDisplayName: Tarvoc
ThreadText: # Is there any evidence that the bias terms help in the attention mechanism of the transformers?
  
  (Asked by: username_0 on 2023-01-31)
  
  In the [original transformer paper](https://arxiv.org/pdf/1706.03762.pdf), the attention mechanism uses parameter matrices, but no bias terms. However, in more recent implementations I see people often using a bias term when computing "key", "query", and "value". For example, in Andrej Karpathy's recent [implementation of GPT](https://github.com/karpathy/nanoGPT/blob/master/model.py), whether a bias term is used can be determined in the config:
  
  ```
  bias: bool = True # True: bias in Linears and LayerNorms, like GPT-2. False: a bit better and faster
  
  ```
  
  This makes me wonder whether there is any evidence that the bias terms help. In particular, if, according to Karpathy, not using bias is "a bit better and faster", why is he using them by default?
  
  **Tags:** transformer, bias
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-06-18. Score: 0
  
  I guess that bias terms have the ability to increase the expression power of the model.According to the paper:"BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models".
  
  > **Comments:**
  > **username_2**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  