Answers.Body: <p>The basic idea of gradient descent is:</p>
  <ul>
  <li><p>Calculate the gradient of some score with respect to parameters that you can control</p>
  </li>
  <li><p>Take a step in the direction of that gradient that improves the score (subract a multiple of gradient - for gradient descent - if you want to minimise some cost function)</p>
  </li>
  </ul>
  <p>The backpropagation using chain rule in neural network layers is part of the first part, calculating the gradient.</p>
  <p>If you interleaved these steps during a single calculation, by taking an update step before propagating the gradient over all layers, you would not propagate the true gradient, but some interim value. There is no guarantee this value would reflect the true gradient of the affected layer when compared to training data and loss functions.</p>
  <p>There are some manipulations of the gradient that are acceptable and improve convergence. For instance momentum, and various forms of weighting based on previous gradient calculations. However, I am not aware of any successful attempt to perform updates with partially-calculated gradient then attempt to continue the interrupted gradient calculations over the mixed updated and not-yet-updated parameters.</p>
  <p>It is also possible to calculate gradients and then only update <em>some</em> of the possible parameters. In some cases this is useful, for instance it is a good choice when performing transfer learning, by only updating a few layers close to the output. This constrains the network to keep a lot of its trained parameters as-is, which reduces the chances of over-fitting to the smaller dataset that the network is learning from when transferring.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-01T20:57:33.797Z
Answers.LastActivityDate: 2021-02-01T22:17:59.123Z
Answers.LastEditDate: 2021-02-01T22:17:59.123Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>This is an excellent question, and the literature is generally silent on whether it's an improvement. I'm personally rather curious about this result!</p>
  <p>There are some practical reasons it's not generally done:</p>
  <p><strong>Framework structure</strong></p>
  <p>Most ML frameworks separate the calculation of the gradient from the update, because users often want to mess with the gradient before updating the weights (e.g. Adam, momentum, etc). The way that most frameworks are implemented makes it difficult to do the optimizer calculation inline with the back-propagation.</p>
  <p><strong>Batch sharding</strong></p>
  <p>Most large networks have the batch calculation sharded over multiple chips. This means that when actually doing the gradient update, the partial sums need to be sent off-chip to be summed and added to the weights.</p>
  <p>This normally isn't too bad because the transmission of the partials sums for one weight run in the parallel with the calculation of the gradients for the next weight.</p>
  <p>But if we update the weights before back-propagating the gradient, then we need to wait for all the partial sums to be received and totaled, then the resulting update being transmitted back to all the chips before the next step in back-propagation can occur. This puts the network latency on the critical path, which makes everything very slow... So it's almost certain that it's faster to do the dumb thing with more steps, than the smarter thing with fewer steps. (The improvement in the gradient would have to save a huge fraction of steps to be worth it).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-22T20:05:07.323Z
Answers.LastActivityDate: 2022-11-22T20:05:07.323Z
Answers.UserDisplayName: Michael O
Body: <p>Why are the weights of a neural net updated only considering the old values of the later layer, not the already updated values?</p>
  <p>I use <a href="https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/" rel="nofollow noreferrer">this example</a> to explain my problem. When applying the backpropagation chain rule, the weights  of the previous layer (<span class="math-container">$w_1, w_2, w_3, w_4$</span>) are updated making use of the chain rule:</p>
  <p><span class="math-container">$$\frac{\partial E_{total}}{\partial w_1} = \frac{\partial E_{total}}{\partial out_{h1}} * \frac{\partial out_{h1}}{\partial net_{h1}}*\frac{\partial net_{h1}}{\partial w_1}$$</span></p>
  <p>He then says:</p>
  <p><span class="math-container">$$\frac{\partial net_{o1}}{\partial out_{h1}}=w_5$$</span></p>
  <p>Although he has already calculated the updated value for <span class="math-container">$w_5$</span>, he uses the old value of <span class="math-container">$w_5$</span> to update <span class="math-container">$w_1$</span>? Because the updated value of <span class="math-container">$w_1$</span> will have an impact on the outcome together with the updated value of <span class="math-container">$w_5$</span>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-26T06:19:17.873Z
Comments.Text: Your idea is basically to use some intuitive kind of temporal difference, ie, use the difference between latest obtained value (by whatever method) and current value to update the attended independent variable's value, which is not used in backprop but reinforcement learning.
Comments.UserDisplayName: cinch
Tags: <deep-learning><backpropagation>
Title: Why are the weights of the previous layers updated only considering the old values of the weights of the later layer, not the updated values?
UserDisplayName: Pluxyy
ThreadText: # Why are the weights of the previous layers updated only considering the old values of the weights of the later layer, not the updated values?
  
  (Asked by: username_0 on 2021-02-01)
  
  Why are the weights of a neural net updated only considering the old values of the later layer, not the already updated values?
  
  I use [this example](https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/) to explain my problem. When applying the backpropagation chain rule, the weights of the previous layer ($w\_1, w\_2, w\_3, w\_4$) are updated making use of the chain rule:
  
  $$\frac{\partial E\_{total}}{\partial w\_1} = \frac{\partial E\_{total}}{\partial out\_{h1}} \* \frac{\partial out\_{h1}}{\partial net\_{h1}}\*\frac{\partial net\_{h1}}{\partial w\_1}$$
  
  He then says:
  
  $$\frac{\partial net\_{o1}}{\partial out\_{h1}}=w\_5$$
  
  Although he has already calculated the updated value for $w\_5$, he uses the old value of $w\_5$ to update $w\_1$? Because the updated value of $w\_1$ will have an impact on the outcome together with the updated value of $w\_5$?
  
  **Tags:** deep-learning, backpropagation
  > **Comments:**
  > **username_1**: Your idea is basically to use some intuitive kind of temporal difference, ie, use the difference between latest obtained value (by whatever method) and current value to update the attended independent variable's value, which is not used in backprop but reinforcement learning.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-02-01. Score: 2
  
  The basic idea of gradient descent is:
  
  * Calculate the gradient of some score with respect to parameters that you can control
  * Take a step in the direction of that gradient that improves the score (subract a multiple of gradient - for gradient descent - if you want to minimise some cost function)
  
  The backpropagation using chain rule in neural network layers is part of the first part, calculating the gradient.
  
  If you interleaved these steps during a single calculation, by taking an update step before propagating the gradient over all layers, you would not propagate the true gradient, but some interim value. There is no guarantee this value would reflect the true gradient of the affected layer when compared to training data and loss functions.
  
  There are some manipulations of the gradient that are acceptable and improve convergence. For instance momentum, and various forms of weighting based on previous gradient calculations. However, I am not aware of any successful attempt to perform updates with partially-calculated gradient then attempt to continue the interrupted gradient calculations over the mixed updated and not-yet-updated parameters.
  
  It is also possible to calculate gradients and then only update *some* of the possible parameters. In some cases this is useful, for instance it is a good choice when performing transfer learning, by only updating a few layers close to the output. This constrains the network to keep a lot of its trained parameters as-is, which reduces the chances of over-fitting to the smaller dataset that the network is learning from when transferring.
  
  
  ---
  
  ### Answer by username_3 on 2022-11-22. Score: 0
  
  This is an excellent question, and the literature is generally silent on whether it's an improvement. I'm personally rather curious about this result!
  
  There are some practical reasons it's not generally done:
  
  **Framework structure**
  
  Most ML frameworks separate the calculation of the gradient from the update, because users often want to mess with the gradient before updating the weights (e.g. Adam, momentum, etc). The way that most frameworks are implemented makes it difficult to do the optimizer calculation inline with the back-propagation.
  
  **Batch sharding**
  
  Most large networks have the batch calculation sharded over multiple chips. This means that when actually doing the gradient update, the partial sums need to be sent off-chip to be summed and added to the weights.
  
  This normally isn't too bad because the transmission of the partials sums for one weight run in the parallel with the calculation of the gradients for the next weight.
  
  But if we update the weights before back-propagating the gradient, then we need to wait for all the partial sums to be received and totaled, then the resulting update being transmitted back to all the chips before the next step in back-propagation can occur. This puts the network latency on the critical path, which makes everything very slow... So it's almost certain that it's faster to do the dumb thing with more steps, than the smarter thing with fewer steps. (The improvement in the gradient would have to save a huge fraction of steps to be worth it).
  
  
  ---
  
  