Answers.Body: <p>Short answer: The bias in a neural network <strong>is not used to avoid an output of 0</strong>. The bias is used to <strong>shift the activation function</strong>.</p>
  <h2>What this means</h2>
  <p>An activation I'm sure almost anyone is familiar with, the sigmoid function:</p>
  <p><span class="math-container">$$f(x)=\frac{1}{1+e^{-x}}$$</span>
  Which looks like:<a href="https://i.sstatic.net/Cp7uj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Cp7uj.png" alt="enter image description here" /></a></p>
  <p>If we use the sigmoid function as the activation of all neurons (without bias'), if a value needs to be shifted, we can never do this directly. Instead, it can only ever be scaled in such a way that it shifts on the next layer when we multiply by more weights and add. That might be a little confusing, so imagine it like this:</p>
  <p>If you had a desired output of 2, how would you ever reach that with the sigmoid function? The answer is you can't, and that's where a bias comes in. With a bias of 1.5 and an output of 0.5, you could reach your desired output of 2.</p>
  <p>For an even more concrete example, let's take a look at the Rectified Linear Unit (ReLU):</p>
  <p><span class="math-container">$$ f(x) = max(0, x)$$</span></p>
  <p>Which looks like:</p>
  <p><a href="https://i.sstatic.net/H8Djy.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/H8Djy.png" alt="enter image description here" /></a></p>
  <p>Now the goal of a neural network is to approximate a function, so lets in this case try and approximate a 2d function using the ReLU activation function. Conveniently, there's a really nice example in <a href="https://www.desmos.com/calculator/cfvtjusqmq" rel="nofollow noreferrer">desmos of exactly this</a>. As you can see, each ReLU equation employs a bias to move the function around. I challenge you to try and fit <span class="math-container">$f\left(x\right)=x^{3}+x^{2}-x-1$</span> <strong>without</strong> using a bias, and you'll really quickly see the need for a bias:</p>
  <p>*Note I should've included the sign along with the numeric value of each bias inside the red bubble below
  <a href="https://i.sstatic.net/e8T2S.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/e8T2S.png" alt="enter image description here" /></a></p>
  <p>ETA: Variance!!!</p>
  <p>How could I forget.</p>
  <p>An important distinction to make is this - An output of 0 is perfectly normal in a neural network. As an example, look at ReLU! A proven, extremely effective activation function that for half of all inputs, outputs a 0!</p>
  <p>The issue, is when 0 decreases your variance. It's extremely important to maintain variance in a neural network. By this I mean: The spread of the inputs should roughly equal the spread of the outputs. The reason for this is obvious once you know:</p>
  <ul>
  <li>If spread gradually increases, the values in the network will explode to infinity, rendering most activation functions useless and slowing down calculations by orders of magnitude</li>
  <li>If spread gradually decreases, the values in the network will vanish to 0, meaning the network doesn't calculate anything at all!</li>
  </ul>
  <p>So how does this tie in? Well, sticking with the ReLU example, say you have 9 positive inputs and 1 negative input. When that negative input goes through the ReLU function, it will come out as 0. This will invariably decrease variance <em>in the long run</em> (there are some cases where it increases initially, but eventually it will decrease), eventually vanishing all values to 0. But with a bias, this can be counteracted. By keeping it even slightly away from 0, in combination with the parameters following, you can ensure that the variance stays roughly the same. Alas, the need for a bias.</p>
  <p><a href="https://ai.stackexchange.com/questions/13106/how-are-exploding-numbers-in-a-forward-pass-of-a-cnn-combated">For more information on keeping variance level, see my question from when I was just learning to count, and how I came to know exploding numbers all too well...</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-28T17:10:38.187Z
Answers.Text: I think it is worth to mention that I have noticed an answer to another question about bias: https://stats.stackexchange.com/a/186284/247373: "If a neural network does not have a bias node in a given layer, it will not be able to produce output in the next layer that differs from 0 (on the linear scale, or the value that corresponds to the transformation of 0 when passed through the activation function) when the feature values are 0."
  I assume that this answer from https://stats.stackexchange.com/ is not entirely correct if "bias in a neural network is not used to avoid an output of 0." ?
Answers.UserDisplayName: bridgemnc
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-29T00:42:43.627Z
Answers.Text: @bridgemnc I edited the answer to add more information in regards to your comment
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-18T04:04:31.877Z
Answers.LastActivityDate: 2021-11-29T00:57:45.880Z
Answers.LastEditDate: 2021-11-29T00:57:45.880Z
Answers.UserDisplayName: Recessive
Body: <p>I have read that bias in neural networks is used to avoid situation in which output of neuron is equal to 0. But what if the same output is equal to -1 and we add 1 to it? Isn't it the same issue as in case of zero output and no bias?</p>
  
Tags: <neural-networks><bias>
Title: Bias equal to 1 and neuron output equal to -1 in neural networks
UserDisplayName: bridgemnc
ThreadText: # Bias equal to 1 and neuron output equal to -1 in neural networks
  
  (Asked by: username_0 on 2021-11-17)
  
  I have read that bias in neural networks is used to avoid situation in which output of neuron is equal to 0. But what if the same output is equal to -1 and we add 1 to it? Isn't it the same issue as in case of zero output and no bias?
  
  **Tags:** neural-networks, bias
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-11-18. Score: 3 | Accepted answer
  
  Short answer: The bias in a neural network **is not used to avoid an output of 0**. The bias is used to **shift the activation function**.
  
  ## What this means
  
  An activation I'm sure almost anyone is familiar with, the sigmoid function:
  
  $$f(x)=\frac{1}{1+e^{-x}}$$
  Which looks like:[![enter image description here](https://i.sstatic.net/Cp7uj.png)](https://i.sstatic.net/Cp7uj.png)
  
  If we use the sigmoid function as the activation of all neurons (without bias'), if a value needs to be shifted, we can never do this directly. Instead, it can only ever be scaled in such a way that it shifts on the next layer when we multiply by more weights and add. That might be a little confusing, so imagine it like this:
  
  If you had a desired output of 2, how would you ever reach that with the sigmoid function? The answer is you can't, and that's where a bias comes in. With a bias of 1.5 and an output of 0.5, you could reach your desired output of 2.
  
  For an even more concrete example, let's take a look at the Rectified Linear Unit (ReLU):
  
  $$ f(x) = max(0, x)$$
  
  Which looks like:
  
  [![enter image description here](https://i.sstatic.net/H8Djy.png)](https://i.sstatic.net/H8Djy.png)
  
  Now the goal of a neural network is to approximate a function, so lets in this case try and approximate a 2d function using the ReLU activation function. Conveniently, there's a really nice example in [desmos of exactly this](https://www.desmos.com/calculator/cfvtjusqmq). As you can see, each ReLU equation employs a bias to move the function around. I challenge you to try and fit $f\left(x\right)=x^{3}+x^{2}-x-1$ **without** using a bias, and you'll really quickly see the need for a bias:
  
  \*Note I should've included the sign along with the numeric value of each bias inside the red bubble below
  [![enter image description here](https://i.sstatic.net/e8T2S.png)](https://i.sstatic.net/e8T2S.png)
  
  ETA: Variance!!!
  
  How could I forget.
  
  An important distinction to make is this - An output of 0 is perfectly normal in a neural network. As an example, look at ReLU! A proven, extremely effective activation function that for half of all inputs, outputs a 0!
  
  The issue, is when 0 decreases your variance. It's extremely important to maintain variance in a neural network. By this I mean: The spread of the inputs should roughly equal the spread of the outputs. The reason for this is obvious once you know:
  
  * If spread gradually increases, the values in the network will explode to infinity, rendering most activation functions useless and slowing down calculations by orders of magnitude
  * If spread gradually decreases, the values in the network will vanish to 0, meaning the network doesn't calculate anything at all!
  
  So how does this tie in? Well, sticking with the ReLU example, say you have 9 positive inputs and 1 negative input. When that negative input goes through the ReLU function, it will come out as 0. This will invariably decrease variance *in the long run* (there are some cases where it increases initially, but eventually it will decrease), eventually vanishing all values to 0. But with a bias, this can be counteracted. By keeping it even slightly away from 0, in combination with the parameters following, you can ensure that the variance stays roughly the same. Alas, the need for a bias.
  
  [For more information on keeping variance level, see my question from when I was just learning to count, and how I came to know exploding numbers all too well...](https://ai.stackexchange.com/questions/13106/how-are-exploding-numbers-in-a-forward-pass-of-a-cnn-combated)
  
  > **Comments:**
  > **username_0**: I think it is worth to mention that I have noticed an answer to another question about bias: https://stats.stackexchange.com/a/186284/247373: "If a neural network does not have a bias node in a given layer, it will not be able to produce output in the next layer that differs from 0 (on the linear scale, or the value that corresponds to the transformation of 0 when passed through the activation function) when the feature values are 0." I assume that this answer from https://stats.stackexchange.com/ is not entirely correct if "bias in a neural network is not used to avoid an output of 0." ?  
  > **username_1**: @bridgemnc I edited the answer to add more information in regards to your comment  
  
  ---
  
  