Answers.Body: <p>I assume you are considering a network where the activation function of the last layer is a sigmoid, so the output of your network is <span class="math-container">$$\tilde{y}=\sigma(W^{L}\cdot f(X, W^1, \dots, W^{L-1})),$$</span>
  where <span class="math-container">$X$</span> is the input vector, and <span class="math-container">$f$</span> is obtained by feeding the input to the network up to the layer <span class="math-container">$L-1$</span>. Let's also call <span class="math-container">$Z:= W^{L}\cdot f(X, W^1, \dots, W^{L-1})$</span>.</p>
  <p>The error term is computed as <span class="math-container">$$E(y, \tilde{y})=E(y, \sigma(Z)),$$</span> where <span class="math-container">$y$</span> is the actual output. Let's get the derivative of the error with respect to the output of the last node (the input of the sigmoid)
  <span class="math-container">$$\frac{\partial E}{z_i}=\frac{\partial E}{\partial \tilde{y}}\frac{\partial\tilde{y}}{\partial z_i}=\frac{\partial E}{\partial \tilde{y}}\frac{\partial\sigma}{\partial z_i}.$$</span>
  The update rule is <span class="math-container">$$z_i =z_i - C\frac{\partial E}{z_i}= z_i - C\frac{\partial E}{\partial \tilde{y}}\frac{\partial\sigma}{\partial z_i}.$$</span>
  Now we can analyse your questions.</p>
  <ol>
  <li>To be close to the middle of the sigmoid means that <span class="math-container">$z_i$</span> is close to <span class="math-container">$0$</span>; moreover the derivative <span class="math-container">$\frac{\partial\sigma}{\partial z_i}$</span> reaches its maximum value when it is evaluated in <span class="math-container">$0$</span>. This means that the term <span class="math-container">$\frac{\partial\sigma}{\partial z_i}$</span> gets &quot;large&quot; as <span class="math-container">$z_i$</span> approaches the center of the sigmoid, contributing more to the update of the weight. Of course it is difficult to say what happens in general, as the term <span class="math-container">$\frac{\partial E}{\partial \tilde{y}}$</span> is also in the expression and it is possible that this term gets really small (or big) for <span class="math-container">$z_i$</span> close to <span class="math-container">$0$</span>. You just know that the term <span class="math-container">$\frac{\partial\sigma}{\partial z_i}$</span> is trying to push <span class="math-container">$z_i$</span> further away from <span class="math-container">$0$</span>, so the idea should be that the closer to the center the larger the update.</li>
  <li>To be close to an extremum point of the loss means that the derivative of the loss with respect to <span class="math-container">$\tilde{y}$</span> is close to <span class="math-container">$0$</span>. Since <span class="math-container">$\frac{\partial E}{\partial \tilde{y}}$</span> is backpropagated in a multiplicative fashion, the rule of thumb is that the closer you are to an extremum the smaller the updates get. Though, as above, is kind of difficult to say what will happen when updating a general node, as some of the term in the multiplication can be very large, making the update large even if <span class="math-container">$\frac{\partial E}{\partial \tilde{y}}$</span> is small.</li>
  </ol>
  <p>For instance, what happens if you are close to the center of the sigmoid but also close to an extremum of the loss? You will have a multiplication of <span class="math-container">$2$</span> terms, one trying to make the update small and the other trying to make the update large, and what matter are the orders of magnitude involved.<br />
  In conclusion, the rules of thumb are as in the points 1. and 2., but they are no guarantee that you won't find any special cases.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-01T16:02:19.593Z
Answers.LastActivityDate: 2021-01-01T16:08:16.507Z
Answers.LastEditDate: 2021-01-01T16:08:16.507Z
Answers.UserDisplayName: Uskebasi
Body: <p>This is the back-propogation rule for the output layer of a multi-layer network:</p>
  <p><span class="math-container">$$W_{jk} := W_{jk} - C \dfrac{\delta E}{\delta W_{jk}}$$</span></p>
  <p>What does this rule do in the more ambiguous cases such as:</p>
  <p>(1) The output of a hidden node is near the middle of a sigmoid curve?</p>
  <p>(2) The graph of error with respect to weight is near a maximum or minimum?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-01T15:13:42.527Z
Comments.Text: Please format any mathematics using MathJax, rather than using images.
Comments.UserDisplayName: The Pointer
Tags: <neural-networks><backpropagation>
Title: Backpropogation rule for the output layer of a multi-layer network - What does the rule do in ambiguous cases?
UserDisplayName: Slowat_Kela
ThreadText: # Backpropogation rule for the output layer of a multi-layer network - What does the rule do in ambiguous cases?
  
  (Asked by: username_0 on 2021-01-01)
  
  This is the back-propogation rule for the output layer of a multi-layer network:
  
  $$W\_{jk} := W\_{jk} - C \dfrac{\delta E}{\delta W\_{jk}}$$
  
  What does this rule do in the more ambiguous cases such as:
  
  (1) The output of a hidden node is near the middle of a sigmoid curve?
  
  (2) The graph of error with respect to weight is near a maximum or minimum?
  
  **Tags:** neural-networks, backpropagation
  
  > **Comments:**
  > **username_1**: Please format any mathematics using MathJax, rather than using images.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-01-01. Score: 1 | Accepted answer
  
  I assume you are considering a network where the activation function of the last layer is a sigmoid, so the output of your network is $$\tilde{y}=\sigma(W^{L}\cdot f(X, W^1, \dots, W^{L-1})),$$
  where $X$ is the input vector, and $f$ is obtained by feeding the input to the network up to the layer $L-1$. Let's also call $Z:= W^{L}\cdot f(X, W^1, \dots, W^{L-1})$.
  
  The error term is computed as $$E(y, \tilde{y})=E(y, \sigma(Z)),$$ where $y$ is the actual output. Let's get the derivative of the error with respect to the output of the last node (the input of the sigmoid)
  $$\frac{\partial E}{z\_i}=\frac{\partial E}{\partial \tilde{y}}\frac{\partial\tilde{y}}{\partial z\_i}=\frac{\partial E}{\partial \tilde{y}}\frac{\partial\sigma}{\partial z\_i}.$$
  The update rule is $$z\_i =z\_i - C\frac{\partial E}{z\_i}= z\_i - C\frac{\partial E}{\partial \tilde{y}}\frac{\partial\sigma}{\partial z\_i}.$$
  Now we can analyse your questions.
  
  1. To be close to the middle of the sigmoid means that $z\_i$ is close to $0$; moreover the derivative $\frac{\partial\sigma}{\partial z\_i}$ reaches its maximum value when it is evaluated in $0$. This means that the term $\frac{\partial\sigma}{\partial z\_i}$ gets "large" as $z\_i$ approaches the center of the sigmoid, contributing more to the update of the weight. Of course it is difficult to say what happens in general, as the term $\frac{\partial E}{\partial \tilde{y}}$ is also in the expression and it is possible that this term gets really small (or big) for $z\_i$ close to $0$. You just know that the term $\frac{\partial\sigma}{\partial z\_i}$ is trying to push $z\_i$ further away from $0$, so the idea should be that the closer to the center the larger the update.
  2. To be close to an extremum point of the loss means that the derivative of the loss with respect to $\tilde{y}$ is close to $0$. Since $\frac{\partial E}{\partial \tilde{y}}$ is backpropagated in a multiplicative fashion, the rule of thumb is that the closer you are to an extremum the smaller the updates get. Though, as above, is kind of difficult to say what will happen when updating a general node, as some of the term in the multiplication can be very large, making the update large even if $\frac{\partial E}{\partial \tilde{y}}$ is small.
  
  For instance, what happens if you are close to the center of the sigmoid but also close to an extremum of the loss? You will have a multiplication of $2$ terms, one trying to make the update small and the other trying to make the update large, and what matter are the orders of magnitude involved.  
  In conclusion, the rules of thumb are as in the points 1. and 2., but they are no guarantee that you won't find any special cases.
  
  
  ---
  
  