Answers.Body: <p>It's not strictly &quot;needed.&quot;  In fact, if you look at things like Keras, you will see that layers have a <code>use_bias</code> parameter, which defaults to True, but you can set to False, of course.</p>
  <p>For an intuition about why bias is <em>useful</em> rather than required, consider a simple <span class="math-container">$\mathbb{R}^2$</span> example.  Imagine that we have some data that we are attempting to fit a straight line to.</p>
  <p>When generating our line, we can iteratively update the slope using something like gradient descent and completely ignore the y-intercept, or bias term.  In the end we will find a line, centered at the origin, that has the identical slope to a best-fit line that passes through the data.</p>
  <p>If you have this mental picture, take it a step further.  Using the bias term (y-intercept), we can then adjust that line up or down (bias the line up or down) by whatever amount is needed to minimize the loss.</p>
  <p>If you think about it, for any line <span class="math-container">$y=mx+b$</span>, we could think of a specific line <span class="math-container">$y=4x$</span> as representing the fundamental line for all lines with that slope.  Really, they are all the <em>same</em> line that we can slide up or down the y-axis to place them where we need them to be.</p>
  <p>Coming back to training a neural network, the bias, therefore, is not <em>required</em>, but can be very useful in allowing us to adjust the output of a neuron up or down as required to better fit the data, possibly easing the difficulty of training subsequent layers/neurons.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-10T13:58:13.697Z
Answers.LastActivityDate: 2023-03-10T13:58:13.697Z
Answers.UserDisplayName: David Hoelzer
Answers.Body: <p>Let's write some code, shall we? First I'll generate two 2D Gaussian blobs with means at (0,0) and at (3,3) and sigma = 1.0. The points for the blob at (0,0) will be in class <code>y=0</code> and the second blob will have the class <code>y=1</code>.</p>
  <pre><code>import numpy as np
  x = np.concatenate([
    np.random.normal(loc=0, scale=1, size=2*1000).reshape(-1,2),
    np.random.normal(loc=3, scale=1, size=2*1000).reshape(-1,2)
  ])
  y = np.concatenate([np.zeros(1000), np.ones(1000)])
  </code></pre>
  <p>We can plot it with something like <code>scatter(x[:,0], x[:,1], c=y)</code>:</p>
  <p><a href="https://i.sstatic.net/AQ9hs.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AQ9hs.png" alt="Enter image description here" /></a></p>
  <p>I'll use <code>torch</code>, so I convert these to torch tensors and use its data wrangling classes to shuffle and split into batches.</p>
  <pre><code>import torch
  x, y = torch.tensor(x).float() , torch.tensor(y).long()
  dataset = torch.utils.data.TensorDataset(x,y)
  dataloader = torch.utils.data.DataLoader(dataset, batch_size=32, shuffle=True)
  </code></pre>
  <p>Now let's make two neural networks - just a linear layer with 2D inputs and two outputs for each class. The only difference is that the first one will have <code>bias=False</code> and the second one <code>bias=True</code>.</p>
  <pre><code>model1 = torch.nn.Sequential(torch.nn.Linear(2, 2, bias=False))
  model2 = torch.nn.Sequential(torch.nn.Linear(2, 2, bias=True ))
  </code></pre>
  <p>I assume that our networks return logits of the classes - so cross-entropy as a loss function. I've hacked together this pretty standard code that, given a <code>model</code> and an <code>optimizer</code> goes through one epoch and returns average loss:</p>
  <pre><code>loss_fn = torch.nn.CrossEntropyLoss()
  def optimize_epoch(model, optimizer):
      total_loss = 0
      for n, (inputs, labels) in enumerate(dataloader):
          optimizer.zero_grad()
          outputs = model.forward(inputs)
          loss = loss_fn(outputs, labels)
          loss.backward()
          optimizer.step()
          total_loss += loss.item()
      return total_loss / n
  </code></pre>
  <p>Running this optimization for 50 epochs and collecting loss history:</p>
  <pre><code>optimizer1 = torch.optim.SGD(model1.parameters(), lr=0.01)
  optimizer2 = torch.optim.SGD(model2.parameters(), lr=0.01)
  
  losses1, losses2 = [] , []
  for _ in range(50):
      losses1.append(optimize_epoch(model1, optimizer1))
      losses2.append(optimize_epoch(model2, optimizer2))
  
  plot(losses1, label=&quot;model1&quot;); plot(losses2, label=&quot;model2&quot;); legend()
  </code></pre>
  <p>You'll see a striking difference in performance between the two models:</p>
  <p><a href="https://i.sstatic.net/PWjlm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PWjlm.png" alt="Enter image description here" /></a></p>
  <hr />
  <p>As per @Stef request, here are the scatterplots for predicted classes for each model. Obtainable via <code>scatter(x[:,0], x[:,1], c=model1.forward(x).argmax(axis=1)</code>.</p>
  <p><a href="https://i.sstatic.net/abq22.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/abq22.png" alt="enter image description here" /></a></p>
  <p>Here you can clearly see that the separating line for <code>bias=False</code> goes through <code>(0,0)</code> in accordance with most of other answers here.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-13T14:22:12.683Z
Answers.Text: I think this experiment would be much much more visually striking if you showed the result as the predicted classes on your scatterplot, rather than just a loss curve.
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-10T18:01:14.257Z
Answers.LastActivityDate: 2023-03-13T15:46:33.937Z
Answers.LastEditDate: 2023-03-13T15:46:33.937Z
Answers.UserDisplayName: Kostya
Answers.Body: <p>No matter what you make <span class="math-container">$W_1$</span> and <span class="math-container">$W_2$</span>, if <span class="math-container">$X_1$</span> is 0 and <span class="math-container">$X_2$</span> is 0 then <span class="math-container">$W_1X_1+W_2X_2$</span> is 0 which (in a typical classification application) means the classifier is completely unsure which class it belongs to.</p>
  <p>Additionally, mirroring a point across the origin (that is, assigning <span class="math-container">$X_1 \leftarrow -X_1$</span> and <span class="math-container">$X_2 \leftarrow -X_2$</span>) will also negate the output. The classifier is only able to provide classifications where one class is mirrored across the origin from the other class.</p>
  <p>Adding a bias term - <span class="math-container">$W_1X_1+W_2X_2+W_3$</span> - solves these problems. The classifier can draw any line to separate the two classes, not only lines that pass through the origin (0,0).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-10T20:59:34.827Z
Answers.LastActivityDate: 2023-03-10T22:40:02.480Z
Answers.LastEditDate: 2023-03-10T22:40:02.480Z
Answers.UserDisplayName: Criticize SE actions means ban
Answers.Body: <p>Let's interpret each node of a layer as a transformation of sub-feature-inputs into a certainty value for the presence (or absence) of a feature.</p>
  <p>Then, for example a dense-layer first looks at each of the sub-features independently. It classifies for each sub-feature whether it counts as evidence of presence or absence of the feature (which the node wants to detect) which is expressed by the sign of the weight of edge of the sub-feature to the node. Then the  absolute weight of the edge tells us how strong of an evidence the corresponding sub-feature is.</p>
  <p>Now, for some features it might be much easier to detect evidence for their absence, than to find evidence of their presence. In such cases, the easiest way for the node to learn the feature might be to just find lots of measures which are evidence for the absence of the feature it wants to detect, and then use the bias as a counter-term which decides whether the evidence for absence found by the sub-features is sufficient to rule out the existence of the feature.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-11T23:13:44.260Z
Answers.LastActivityDate: 2023-03-11T23:13:44.260Z
Answers.UserDisplayName: Sudix
Body: <p>I have read several resources, including previously asked questions such as <a href="https://stackoverflow.com/a/2499936/8243797">this</a>. I have also read arguments related to intercepts needed to separate linearly separable data. If my neural network can perform feature transformation, what is the need of a bias term?</p>
  <p>Since the weights are learnt, my network can optimise to fit the data. For example, if my data is in 2D coordinate plane, my equation without bias for a perceptron for the layer will be <span class="math-container">$W_1X_1 + W_2X_2$</span> where <span class="math-container">$X_1$</span> is <code>x</code> coordinate and <span class="math-container">$X_2$</span> is <code>y</code> coordinate, making <span class="math-container">$W_1$</span> and <span class="math-container">$W_2$</span> coefficients of all vectors along <code>x</code> and <code>y</code> direction. Their linear combination will cover the whole plane which allows my data to be transformed across a line with 0 intercept.</p>
  <p>For example, if my weight is <code>1.0</code> for input <code>x</code>, and my bias is <code>0.1</code>, I might as well have weight <span class="math-container">$1+(0.1/\bar x)$</span> (or any other value descriptive of x) and <code>0</code> bias to get the same result.</p>
  <p>Similar things happen for the arguments related to activation mentioned in the marked solution to the referenced question.</p>
  <p>In such a scenario, why is the bias needed?</p>
  <p>Edit: A lot of the answers offer reasonable arguments for the perceptron/single layer case, but perceptron was just an example. Do they hold for deep neural networks as well, because that allows for previous layers better transformation of inputs? As mentioned by some, <code>0</code> input will truly cause a problem which I agree with.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-11T21:32:33.497Z
Comments.Text: You will find a nicely illustrated answer in this post: https://stackoverflow.com/a/2499936/1785141
  Simply speaking, the bias modifies $x$ before it is passed to the activation function and, therefore, allows the activation function to be shifted on the x-axis.
  In other words, the bias operates on $x$, rather than $f(x)$.
Comments.UserDisplayName: user654123
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-12T06:55:07.207Z
Comments.Text: @user654123 I have read the answer and cited the same in my question which i then build upon
Comments.UserDisplayName: SajanGohil
Tags: <neural-networks><deep-learning><feedforward-neural-networks>
Title: Why is a bias parameter needed in neural networks?
UserDisplayName: SajanGohil
ThreadText: # Why is a bias parameter needed in neural networks?
  
  (Asked by: username_0 on 2023-03-10)
  
  I have read several resources, including previously asked questions such as [this](https://stackoverflow.com/a/2499936/8243797). I have also read arguments related to intercepts needed to separate linearly separable data. If my neural network can perform feature transformation, what is the need of a bias term?
  
  Since the weights are learnt, my network can optimise to fit the data. For example, if my data is in 2D coordinate plane, my equation without bias for a perceptron for the layer will be $W\_1X\_1 + W\_2X\_2$ where $X\_1$ is `x` coordinate and $X\_2$ is `y` coordinate, making $W\_1$ and $W\_2$ coefficients of all vectors along `x` and `y` direction. Their linear combination will cover the whole plane which allows my data to be transformed across a line with 0 intercept.
  
  For example, if my weight is `1.0` for input `x`, and my bias is `0.1`, I might as well have weight $1+(0.1/\bar x)$ (or any other value descriptive of x) and `0` bias to get the same result.
  
  Similar things happen for the arguments related to activation mentioned in the marked solution to the referenced question.
  
  In such a scenario, why is the bias needed?
  
  Edit: A lot of the answers offer reasonable arguments for the perceptron/single layer case, but perceptron was just an example. Do they hold for deep neural networks as well, because that allows for previous layers better transformation of inputs? As mentioned by some, `0` input will truly cause a problem which I agree with.
  
  **Tags:** neural-networks, deep-learning, feedforward-neural-networks
  
  > **Comments:**
  > **username_1**: You will find a nicely illustrated answer in this post: https://stackoverflow.com/a/2499936/1785141 Simply speaking, the bias modifies $x$ before it is passed to the activation function and, therefore, allows the activation function to be shifted on the x-axis. In other words, the bias operates on $x$, rather than $f(x)$.  
  > **username_0**: @username_1 I have read the answer and cited the same in my question which i then build upon  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_2 on 2023-03-10. Score: 16
  
  It's not strictly "needed." In fact, if you look at things like Keras, you will see that layers have a `use_bias` parameter, which defaults to True, but you can set to False, of course.
  
  For an intuition about why bias is *useful* rather than required, consider a simple $\mathbb{R}^2$ example. Imagine that we have some data that we are attempting to fit a straight line to.
  
  When generating our line, we can iteratively update the slope using something like gradient descent and completely ignore the y-intercept, or bias term. In the end we will find a line, centered at the origin, that has the identical slope to a best-fit line that passes through the data.
  
  If you have this mental picture, take it a step further. Using the bias term (y-intercept), we can then adjust that line up or down (bias the line up or down) by whatever amount is needed to minimize the loss.
  
  If you think about it, for any line $y=mx+b$, we could think of a specific line $y=4x$ as representing the fundamental line for all lines with that slope. Really, they are all the *same* line that we can slide up or down the y-axis to place them where we need them to be.
  
  Coming back to training a neural network, the bias, therefore, is not *required*, but can be very useful in allowing us to adjust the output of a neuron up or down as required to better fit the data, possibly easing the difficulty of training subsequent layers/neurons.
  
  
  ---
  
  ### Answer by username_3 on 2023-03-10. Score: 11
  
  Let's write some code, shall we? First I'll generate two 2D Gaussian blobs with means at (0,0) and at (3,3) and sigma = 1.0. The points for the blob at (0,0) will be in class `y=0` and the second blob will have the class `y=1`.
  
  ```
  import numpy as np
  x = np.concatenate([
    np.random.normal(loc=0, scale=1, size=2*1000).reshape(-1,2),
    np.random.normal(loc=3, scale=1, size=2*1000).reshape(-1,2)
  ])
  y = np.concatenate([np.zeros(1000), np.ones(1000)])
  
  ```
  
  We can plot it with something like `scatter(x[:,0], x[:,1], c=y)`:
  
  [![Enter image description here](https://i.sstatic.net/AQ9hs.png)](https://i.sstatic.net/AQ9hs.png)
  
  I'll use `torch`, so I convert these to torch tensors and use its data wrangling classes to shuffle and split into batches.
  
  ```
  import torch
  x, y = torch.tensor(x).float() , torch.tensor(y).long()
  dataset = torch.utils.data.TensorDataset(x,y)
  dataloader = torch.utils.data.DataLoader(dataset, batch_size=32, shuffle=True)
  
  ```
  
  Now let's make two neural networks - just a linear layer with 2D inputs and two outputs for each class. The only difference is that the first one will have `bias=False` and the second one `bias=True`.
  
  ```
  model1 = torch.nn.Sequential(torch.nn.Linear(2, 2, bias=False))
  model2 = torch.nn.Sequential(torch.nn.Linear(2, 2, bias=True ))
  
  ```
  
  I assume that our networks return logits of the classes - so cross-entropy as a loss function. I've hacked together this pretty standard code that, given a `model` and an `optimizer` goes through one epoch and returns average loss:
  
  ```
  loss_fn = torch.nn.CrossEntropyLoss()
  def optimize_epoch(model, optimizer):
      total_loss = 0
      for n, (inputs, labels) in enumerate(dataloader):
          optimizer.zero_grad()
          outputs = model.forward(inputs)
          loss = loss_fn(outputs, labels)
          loss.backward()
          optimizer.step()
          total_loss += loss.item()
      return total_loss / n
  
  ```
  
  Running this optimization for 50 epochs and collecting loss history:
  
  ```
  optimizer1 = torch.optim.SGD(model1.parameters(), lr=0.01)
  optimizer2 = torch.optim.SGD(model2.parameters(), lr=0.01)
  
  losses1, losses2 = [] , []
  for _ in range(50):
      losses1.append(optimize_epoch(model1, optimizer1))
      losses2.append(optimize_epoch(model2, optimizer2))
  
  plot(losses1, label="model1"); plot(losses2, label="model2"); legend()
  
  ```
  
  You'll see a striking difference in performance between the two models:
  
  [![Enter image description here](https://i.sstatic.net/PWjlm.png)](https://i.sstatic.net/PWjlm.png)
  
  ---
  
  As per @username_4 request, here are the scatterplots for predicted classes for each model. Obtainable via `scatter(x[:,0], x[:,1], c=model1.forward(x).argmax(axis=1)`.
  
  [![enter image description here](https://i.sstatic.net/abq22.png)](https://i.sstatic.net/abq22.png)
  
  Here you can clearly see that the separating line for `bias=False` goes through `(0,0)` in accordance with most of other answers here.
  
  > **Comments:**
  > **username_4**: I think this experiment would be much much more visually striking if you showed the result as the predicted classes on your scatterplot, rather than just a loss curve.  
  
  ---
  
  ### Answer by username_5 on 2023-03-10. Score: 8
  
  No matter what you make $W\_1$ and $W\_2$, if $X\_1$ is 0 and $X\_2$ is 0 then $W\_1X\_1+W\_2X\_2$ is 0 which (in a typical classification application) means the classifier is completely unsure which class it belongs to.
  
  Additionally, mirroring a point across the origin (that is, assigning $X\_1 \leftarrow -X\_1$ and $X\_2 \leftarrow -X\_2$) will also negate the output. The classifier is only able to provide classifications where one class is mirrored across the origin from the other class.
  
  Adding a bias term - $W\_1X\_1+W\_2X\_2+W\_3$ - solves these problems. The classifier can draw any line to separate the two classes, not only lines that pass through the origin (0,0).
  
  
  ---
  
  ### Answer by username_6 on 2023-03-11. Score: 0
  
  Let's interpret each node of a layer as a transformation of sub-feature-inputs into a certainty value for the presence (or absence) of a feature.
  
  Then, for example a dense-layer first looks at each of the sub-features independently. It classifies for each sub-feature whether it counts as evidence of presence or absence of the feature (which the node wants to detect) which is expressed by the sign of the weight of edge of the sub-feature to the node. Then the absolute weight of the edge tells us how strong of an evidence the corresponding sub-feature is.
  
  Now, for some features it might be much easier to detect evidence for their absence, than to find evidence of their presence. In such cases, the easiest way for the node to learn the feature might be to just find lots of measures which are evidence for the absence of the feature it wants to detect, and then use the bias as a counter-term which decides whether the evidence for absence found by the sub-features is sufficient to rule out the existence of the feature.
  
  
  ---
  
  