Answers.Body: <p>Let <span class="math-container">$L(\mathbf{w}, b) = \sigma \left(\sum_{i=1}^n w_i x_i \right)+b$</span>, then the <a href="https://en.wikipedia.org/wiki/Chain_rule" rel="nofollow noreferrer">partial derivative of <span class="math-container">$L$</span> with respect to <span class="math-container">$b$</span></a>, in <a href="https://en.wikipedia.org/wiki/Leibniz%27s_notation" rel="nofollow noreferrer">Leibniz's notation</a>, is <span class="math-container">$\frac{\partial L}{\partial b} = 1$</span>. Let <span class="math-container">$L(\mathbf{w}, b) = \sigma \left(\sum_{i=1}^n w_i x_i + b \right)$</span>, then the partial derivative of <span class="math-container">$L$</span> with respect to <span class="math-container">$b$</span> is <span class="math-container">$\frac{\partial L}{\partial b} = \frac{\partial L}{\partial \sigma} \frac{\partial \sigma}{\partial b}$</span>.</p>
  
  <p>So, in general, the partial derivatives with respect to <span class="math-container">$b$</span> would be different in these two cases, thus, in the gradient descent step, <span class="math-container">$b$</span> would be updated differently in both cases. In practice, it may or not affect the performance of the model, depending on the importance of the bias with respect to the given problem. Read <a href="https://stackoverflow.com/a/2499936/3924118">this answer</a> to understand the role of the bias.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-10T17:17:29.117Z
Answers.LastActivityDate: 2019-12-10T17:54:22.850Z
Answers.LastEditDate: 2019-12-10T17:54:22.850Z
Answers.UserDisplayName: nbro
Body: <p>I was looking at the source code for a personal project neural network implementation, and the bias for each node was mistakenly applied after the activation function. The output of each node was therefore <span class="math-container">$\sigma\big(\sum_{i=1}^n w_i x_i\big)+b$</span> instead of <span class="math-container">$\sigma\big(\sum_{i=1}^n w_i x_i + b\big)$</span>. Assuming standard back-propagation algorithms are used for training, what (if any) impact would this have?</p>
  
Tags: <neural-networks><machine-learning><training><backpropagation><activation-functions>
Title: What would be the implications of mistakenly adding bias after the activation function?
UserDisplayName: TimD1
ThreadText: # What would be the implications of mistakenly adding bias after the activation function?
  
  (Asked by: username_0 on 2019-12-10)
  
  I was looking at the source code for a personal project neural network implementation, and the bias for each node was mistakenly applied after the activation function. The output of each node was therefore $\sigma\big(\sum\_{i=1}^n w\_i x\_i\big)+b$ instead of $\sigma\big(\sum\_{i=1}^n w\_i x\_i + b\big)$. Assuming standard back-propagation algorithms are used for training, what (if any) impact would this have?
  
  **Tags:** neural-networks, machine-learning, training, backpropagation, activation-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-10. Score: 2
  
  Let $L(\mathbf{w}, b) = \sigma \left(\sum\_{i=1}^n w\_i x\_i \right)+b$, then the [partial derivative of $L$ with respect to $b$](https://en.wikipedia.org/wiki/Chain_rule), in [Leibniz's notation](https://en.wikipedia.org/wiki/Leibniz%27s_notation), is $\frac{\partial L}{\partial b} = 1$. Let $L(\mathbf{w}, b) = \sigma \left(\sum\_{i=1}^n w\_i x\_i + b \right)$, then the partial derivative of $L$ with respect to $b$ is $\frac{\partial L}{\partial b} = \frac{\partial L}{\partial \sigma} \frac{\partial \sigma}{\partial b}$.
  
  So, in general, the partial derivatives with respect to $b$ would be different in these two cases, thus, in the gradient descent step, $b$ would be updated differently in both cases. In practice, it may or not affect the performance of the model, depending on the importance of the bias with respect to the given problem. Read [this answer](https://stackoverflow.com/a/2499936/3924118) to understand the role of the bias.
  
  
  ---
  
  