Answers.Body: <ol>
  <li><p>is always wrong. Most implementations then &quot;silently&quot; correct this error, even if the docstrings have this wrong variant, in the actual use it is done correctly.</p>
  </li>
  <li><p>and 3. are the same, considering that <span class="math-container">$out=\phi(net)$</span>. 3. is more economic if the activation is a solution of an autonomous ODE like with the logistic function. 2. is more generally true, especially if the activation function gets modified/perturbed by some linear function to avoid random behavior in the flat parts of the sigmoid at large values.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-20T16:15:34.377Z
Answers.LastActivityDate: 2023-01-20T16:15:34.377Z
Answers.UserDisplayName: Lutz Lehmann
Body: <p>Assuming a single perceptron (see figure), I have found two versions of how to use backpropagation to update the weights. The perceptron is split in two, so we see the weighted sum on the left (the output of this is <em>net</em>) and then the sigmoid function phi on the right (the output of this is <em>out</em>).</p>
  <p><a href="https://i.sstatic.net/HqH9x.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HqH9x.png" alt="enter image description here" /></a></p>
  <p>So for the backpropagation portion, we compute <span class="math-container">$\frac{\delta cost}{\delta w}=\frac{\delta cost}{\delta out}\times \frac{\delta out}{\delta net} \times  \frac{\delta net}{\delta w}​$</span>, to find how the weight affects the error.</p>
  <p>For the portion <span class="math-container">$\frac{\delta out}{\delta net}$</span> I have seen three different versions for how to compute the value:</p>
  <ol>
  <li><span class="math-container">$\frac{\delta out}{\delta net}$</span> = <span class="math-container">$\phi'(out) = \phi(out) \times (1 - \phi(out))$</span></li>
  <li><span class="math-container">$\frac{\delta out}{\delta net}$</span> = <span class="math-container">$\phi'(net)= \phi(net) \times (1 - \phi(net))$</span></li>
  <li><span class="math-container">$\frac{\delta out}{\delta net}$</span> = <span class="math-container">$out \times (1 - out)$</span></li>
  </ol>
  <p>Can somebody explain to me which one is correct and why? Or is there one which should be preferred?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-20T15:41:30.273Z
Comments.Text: Could you provide sources for the three different versions you have seen?
Comments.UserDisplayName: Robin van Hoorn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-22T12:34:20.337Z
Comments.Text: For 1. https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/, for 2. https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6 and for 3. https://ai.stackexchange.com/questions/5638/are-my-computations-of-the-forward-and-backward-pass-of-a-neural-network-with-on
Comments.UserDisplayName: HTH
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-25T19:48:00.153Z
Comments.Text: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comments.UserDisplayName: Community
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-25T19:49:33.200Z
Comments.Text: If I'm understanding you, you have some input "x" and the output "out", but it's not clear to me what equations you are using to go from x to out. Are those the \sigma and \phi functions?
Comments.UserDisplayName: Taw
Tags: <neural-networks><backpropagation><gradient-descent>
Title: Single Layer Perceptron Backpropagation: How to compute affect of the net value on the output?
UserDisplayName: HTH
ThreadText: # Single Layer Perceptron Backpropagation: How to compute affect of the net value on the output?
  
  (Asked by: username_0 on 2023-01-20)
  
  Assuming a single perceptron (see figure), I have found two versions of how to use backpropagation to update the weights. The perceptron is split in two, so we see the weighted sum on the left (the output of this is *net*) and then the sigmoid function phi on the right (the output of this is *out*).
  
  [![enter image description here](https://i.sstatic.net/HqH9x.png)](https://i.sstatic.net/HqH9x.png)
  
  So for the backpropagation portion, we compute $\frac{\delta cost}{\delta w}=\frac{\delta cost}{\delta out}\times \frac{\delta out}{\delta net} \times \frac{\delta net}{\delta w}​$, to find how the weight affects the error.
  
  For the portion $\frac{\delta out}{\delta net}$ I have seen three different versions for how to compute the value:
  
  1. $\frac{\delta out}{\delta net}$ = $\phi'(out) = \phi(out) \times (1 - \phi(out))$
  2. $\frac{\delta out}{\delta net}$ = $\phi'(net)= \phi(net) \times (1 - \phi(net))$
  3. $\frac{\delta out}{\delta net}$ = $out \times (1 - out)$
  
  Can somebody explain to me which one is correct and why? Or is there one which should be preferred?
  
  **Tags:** neural-networks, backpropagation, gradient-descent
  
  > **Comments:**
  > **username_1**: Could you provide sources for the three different versions you have seen?  
  > **username_0**: For 1. https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/, for 2. https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6 and for 3. https://ai.stackexchange.com/questions/5638/are-my-computations-of-the-forward-and-backward-pass-of-a-neural-network-with-on  
  > **username_2**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
  > **username_3**: If I'm understanding you, you have some input "x" and the output "out", but it's not clear to me what equations you are using to go from x to out. Are those the \sigma and \phi functions?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2023-01-20. Score: 1
  
  1. is always wrong. Most implementations then "silently" correct this error, even if the docstrings have this wrong variant, in the actual use it is done correctly.
  2. and 3. are the same, considering that $out=\phi(net)$. 3. is more economic if the activation is a solution of an autonomous ODE like with the logistic function. 2. is more generally true, especially if the activation function gets modified/perturbed by some linear function to avoid random behavior in the flat parts of the sigmoid at large values.
  
  
  ---
  
  