Answers.Body: <p>The most usual case of <code>bias=False</code> is in layers before/after Batch Normalization with no activators in between. The BatchNorm layer will re-center the data anyway, removing the bias and making it a useless trainable parameter. Quoting the <a href="https://arxiv.org/abs/1502.03167" rel="noreferrer">original BatchNorm paper</a>:</p>
  <blockquote>
  <p>Note that, since we normalize <span class="math-container">$Wu+b$</span>, the bias <span class="math-container">$b$</span> can be ignored since its effect will be canceled by the subsequent mean subtraction</p>
  </blockquote>
  <p>Similar thing happens in transformers' LinearNormalization and (as far as I understand how conditioning works) in the GANs' conditioning layer - the data gets re-centered, effectively cancelling the bias.</p>
  <p>In my experience, that's the most frequent reason to see <code>bias=False</code>, but one can imagine other reasons to remove the bias. As a rule of thumb, I'd say that you don't include bias if you want to &quot;transform zeros to zeros&quot; - things like <a href="https://ai.stackexchange.com/questions/27428/unable-to-learn-a-rotational-angle-by-parametrising-the-angle-as-a-neural-netw">learned rotations</a> can be an example of such (rather exotic) application.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-10T23:06:26.250Z
Answers.LastActivityDate: 2021-05-10T23:06:26.250Z
Answers.UserDisplayName: Kostya
Body: <p>I'm not really that experienced with deep learning, and I've been looking at research code (mostly PyTorch) for deep neural networks, specifically GANs, and, in many cases, I see the authors setting <code>bias=False</code> in some layers without much justification. This isn't usually done in a long stack of layers that have a similar purpose, but mostly in unique layers like the initial linear layer after a conditioning vector, or certain layers in an attention architecture.</p>
  <p>I imagined there must be a strategy to this, but most articles online seem to confirm my initial perception that bias is a good thing to have available for training in pretty much every layer.</p>
  <p>Is there a specific optimization / theoretical reason to turn off biases in specific layers in a network? How can I choose when to do it when designing my own architecture?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-10T11:45:21.297Z
Comments.Text: Hello. To have more context, I suggest that you include the link to 1-2 examples of models that do not use the bias.
Comments.UserDisplayName: nbro
Tags: <optimization><deep-neural-networks><pytorch>
Title: When should you not use the bias in a layer?
UserDisplayName: Nikos Tsakas
ThreadText: # When should you not use the bias in a layer?
  
  (Asked by: username_0 on 2021-05-09)
  
  I'm not really that experienced with deep learning, and I've been looking at research code (mostly PyTorch) for deep neural networks, specifically GANs, and, in many cases, I see the authors setting `bias=False` in some layers without much justification. This isn't usually done in a long stack of layers that have a similar purpose, but mostly in unique layers like the initial linear layer after a conditioning vector, or certain layers in an attention architecture.
  
  I imagined there must be a strategy to this, but most articles online seem to confirm my initial perception that bias is a good thing to have available for training in pretty much every layer.
  
  Is there a specific optimization / theoretical reason to turn off biases in specific layers in a network? How can I choose when to do it when designing my own architecture?
  
  **Tags:** optimization, deep-neural-networks, pytorch
  
  > **Comments:**
  > **username_1**: Hello. To have more context, I suggest that you include the link to 1-2 examples of models that do not use the bias.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-05-10. Score: 6 | Accepted answer
  
  The most usual case of `bias=False` is in layers before/after Batch Normalization with no activators in between. The BatchNorm layer will re-center the data anyway, removing the bias and making it a useless trainable parameter. Quoting the [original BatchNorm paper](https://arxiv.org/abs/1502.03167):
  
  > Note that, since we normalize $Wu+b$, the bias $b$ can be ignored since its effect will be canceled by the subsequent mean subtraction
  
  Similar thing happens in transformers' LinearNormalization and (as far as I understand how conditioning works) in the GANs' conditioning layer - the data gets re-centered, effectively cancelling the bias.
  
  In my experience, that's the most frequent reason to see `bias=False`, but one can imagine other reasons to remove the bias. As a rule of thumb, I'd say that you don't include bias if you want to "transform zeros to zeros" - things like [learned rotations](https://ai.stackexchange.com/questions/27428/unable-to-learn-a-rotational-angle-by-parametrising-the-angle-as-a-neural-netw) can be an example of such (rather exotic) application.
  
  
  ---
  
  