Answers.Body: <p>Yes, it is not unusual to omit the bias by adding a neuron which always outputs a constant 1, which will then be multiplied by an appropriate weight to give the same formula as you would get using an explicit bias.</p>
  <p>One notable text using this convention is <a href="https://www.cs.huji.ac.il/%7Eshais/UnderstandingMachineLearning/copy.html" rel="nofollow noreferrer"><em>Understanding Machine Learning: From Theory to Algorithms</em></a> by  Shai Shalev-Shwartz and Shai Ben-David. In section 20.1 there is a diagram of a neural network where a neuron outputting a constant value is added to each layer which you might find helpful.</p>
  <p>To understand why this works, suppose the outputs of the previous layer are <span class="math-container">$u_1, \dots, u_n, u_{n + 1}$</span>, where <span class="math-container">$u_{n + 1}$</span> is always <span class="math-container">$1$</span>. Then a neuron in the next layer (without a bias) computes</p>
  <p><span class="math-container">$$ \sigma\left(\sum_{i = 1}^{n + 1} w_i u_i \right) = \sigma\left(\sum_{i = 1}^n w_i u_i + w_{n + 1}\right),$$</span>
  where <span class="math-container">$\sigma$</span> is the activation function. So, the weight <span class="math-container">$w_{n + 1}$</span> just serves as the bias because it is multiplied by <span class="math-container">$u_{n + 1} = 1$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-14T08:55:56.117Z
Answers.LastActivityDate: 2021-04-14T08:55:56.117Z
Answers.UserDisplayName: htl
Body: <p>I'm learning about how neural networks are trained. I understand how a neuron works, backpropagation, and all that. In neurons, there is a clear distinction between a &quot;weight&quot; and a &quot;bias&quot;.</p>
  <p><span class="math-container">$$
  Y= \sigma(\text{weight} * \text{input})+ \text{bias}
  $$</span></p>
  <p>However, all the sources I've found when you train the network you just adjust the weights. Not the bias.</p>
  <p><a href="https://i.sstatic.net/uapH3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/uapH3.png" alt="enter image description here" /></a></p>
  <p>However, they never mention what the bias should do, which leads me to think that you just merge all weights and biases in a <span class="math-container">$W$</span> vector and call it weights, even though there are also biases. Is that correctly understood?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-14T08:39:52.123Z
Comments.Text: bias is just a simple kind of weight to shift the separation line, it can be imagined as a weight which is connected to input (original or previous layer) of value 1 always
Comments.UserDisplayName: Dan D.
Tags: <neural-networks><deep-learning><terminology><backpropagation><weights>
Title: Is the bias also a "weight" in a neural network?
UserDisplayName: nammerkage
ThreadText: # Is the bias also a "weight" in a neural network?
  
  (Asked by: username_0 on 2021-04-14)
  
  I'm learning about how neural networks are trained. I understand how a neuron works, backpropagation, and all that. In neurons, there is a clear distinction between a "weight" and a "bias".
  
  $$
  Y= \sigma(\text{weight} \* \text{input})+ \text{bias}
  $$
  
  However, all the sources I've found when you train the network you just adjust the weights. Not the bias.
  
  [![enter image description here](https://i.sstatic.net/uapH3.png)](https://i.sstatic.net/uapH3.png)
  
  However, they never mention what the bias should do, which leads me to think that you just merge all weights and biases in a $W$ vector and call it weights, even though there are also biases. Is that correctly understood?
  
  **Tags:** neural-networks, deep-learning, terminology, backpropagation, weights
  
  > **Comments:**
  > **username_1**: bias is just a simple kind of weight to shift the separation line, it can be imagined as a weight which is connected to input (original or previous layer) of value 1 always  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-04-14. Score: 4 | Accepted answer
  
  Yes, it is not unusual to omit the bias by adding a neuron which always outputs a constant 1, which will then be multiplied by an appropriate weight to give the same formula as you would get using an explicit bias.
  
  One notable text using this convention is [*Understanding Machine Learning: From Theory to Algorithms*](https://www.cs.huji.ac.il/%7Eshais/UnderstandingMachineLearning/copy.html) by Shai Shalev-Shwartz and Shai Ben-David. In section 20.1 there is a diagram of a neural network where a neuron outputting a constant value is added to each layer which you might find helpful.
  
  To understand why this works, suppose the outputs of the previous layer are $u\_1, \dots, u\_n, u\_{n + 1}$, where $u\_{n + 1}$ is always $1$. Then a neuron in the next layer (without a bias) computes
  
  $$ \sigma\left(\sum\_{i = 1}^{n + 1} w\_i u\_i \right) = \sigma\left(\sum\_{i = 1}^n w\_i u\_i + w\_{n + 1}\right),$$
  where $\sigma$ is the activation function. So, the weight $w\_{n + 1}$ just serves as the bias because it is multiplied by $u\_{n + 1} = 1$.
  
  
  ---
  
  