Answers.Body: <p>@serali is correct, there are many resources to describe this process.  I'm going to talk about it in a very general way.</p>
  <p><strong>Disclaimer</strong>: this is a work in progress, it is not yet complete.</p>
  <p><strong>Question</strong>:<br />
  How do we know what weights to change in the network?</p>
  <p><strong>General Answer</strong>:<br />
  This is a very good question.  This is a key component of ML, and even small improvements there would make a huge difference for the overall community.  There are gigawatt-hours of electricity spent using the current answers to this question.</p>
  <p>Training means you have an input that you want the network to transform into an output, and it means you have an output to compare it against.</p>
  <p>The difference between what it makes, and what it should make, the error, can be run in reverse through the network.  As you go through that reverse run, you can get an error associated with each weight. if you move that error to zero, by changing the weight, then the network can be less bad.</p>
  <p><strong>Some variations</strong>:<br />
  Here are some approaches for relating the per-weight errors to the new weights.</p>
  <p>Approaches:</p>
  <ul>
  <li>(sgd <a href="https://optimization.cbe.cornell.edu/index.php?title=Stochastic_gradient_descent" rel="nofollow noreferrer">link</a>) take just a few or one sample at random and compute the error, and then move weights in that direction, usually only a small percentage of it. The intention is that each new update occurs closer to the optimum, and works well in lower-noise cases.</li>
  <li>(conjugate gradient <a href="https://optimization.cbe.cornell.edu/index.php?title=Conjugate_gradient_methods" rel="nofollow noreferrer">link</a>, ) find a line along which to minimize, find the minimum, explore around that minimum to find a new direction for minimization.  For quadratic problems (shaped like parabola) it gives good solutions quickly.</li>
  <li>(momentum <a href="https://optimization.cbe.cornell.edu/index.php?title=Momentum" rel="nofollow noreferrer">link</a>) take a running average of the last several weight changes, and do most of the movement in that direction</li>
  <li>(regularization <a href="https://medium.com/intelligentmachines/convolutional-neural-network-and-regularization-techniques-with-tensorflow-and-keras-5a09e6e65dc7" rel="nofollow noreferrer">link</a>) penalize weights that are too high, push back against learning them, tries to keep weights in smaller subset of the space unless pushed there by the data.</li>
  <li>(dropout <a href="https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/" rel="nofollow noreferrer">link</a>) randomly set internal responses of neurons to zero, to force a robust and distributed representation of the processing in the network.  Keeps on network from &quot;dominating the show&quot; because in practice it tends to do more harm than good for generalization.</li>
  <li>(rotated locking <a href="https://paperswithcode.com/method/stochastic-depth" rel="nofollow noreferrer">link</a>) aka stochastic depth, randomly freeze layers in the architecture to avoid most learning occurring only in the last few layers, aka diminishing gradients.</li>
  <li>(pca in vectorized weights) given a batch of errors for weights, perform pca and only retain highest x percent of eigenvalues, move in that direction.</li>
  <li>(entropy-scaled momentum in pca elements) given batch of errors for weights, perform svd, and then map weight changes per sample to the largest eigenvectors associated with it</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-01T16:20:18.287Z
Answers.LastActivityDate: 2022-08-08T19:24:08.030Z
Answers.LastEditDate: 2022-08-08T19:24:08.030Z
Answers.UserDisplayName: EngrStudent
Body: <p>I'm currently working on constructing a neural network from scratch (in JavaScript). I'm in the middle of working on the backpropagation, but there's something I don't understand: how does the backprop algorithm know which weights to change or which paths to take? The way I did it, it always took all of the paths/weights and changed them all. So how does the algorithm know which paths to take, which weights to change, and whether to add or subtract X amount from said weight?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-06T12:40:05.170Z
Comments.Text: Basically using chain rule. Check [this video](https://www.youtube.com/watch?v=Ilg3gGewQ5U&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi&index=3) and if it is your taste, the following one on the playlist with the mathematical details to get a better understanding of the process. [This one](https://www.coursera.org/learn/neural-networks-deep-learning/lecture/6dDj7/backpropagation-intuition-optional) might also help.
Comments.UserDisplayName: serali
Tags: <neural-networks><backpropagation>
Title: How does backpropagation know which weights to change?
UserDisplayName: Blue Herring
ThreadText: # How does backpropagation know which weights to change?
  
  (Asked by: username_0 on 2021-10-06)
  
  I'm currently working on constructing a neural network from scratch (in JavaScript). I'm in the middle of working on the backpropagation, but there's something I don't understand: how does the backprop algorithm know which weights to change or which paths to take? The way I did it, it always took all of the paths/weights and changed them all. So how does the algorithm know which paths to take, which weights to change, and whether to add or subtract X amount from said weight?
  
  **Tags:** neural-networks, backpropagation
  
  > **Comments:**
  > **username_1**: Basically using chain rule. Check [this video](https://www.youtube.com/watch?v=Ilg3gGewQ5U&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi&index=3) and if it is your taste, the following one on the playlist with the mathematical details to get a better understanding of the process. [This one](https://www.coursera.org/learn/neural-networks-deep-learning/lecture/6dDj7/backpropagation-intuition-optional) might also help.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-06-01. Score: 3 | Accepted answer
  
  @username_1 is correct, there are many resources to describe this process. I'm going to talk about it in a very general way.
  
  **Disclaimer**: this is a work in progress, it is not yet complete.
  
  **Question**:  
  How do we know what weights to change in the network?
  
  **General Answer**:  
  This is a very good question. This is a key component of ML, and even small improvements there would make a huge difference for the overall community. There are gigawatt-hours of electricity spent using the current answers to this question.
  
  Training means you have an input that you want the network to transform into an output, and it means you have an output to compare it against.
  
  The difference between what it makes, and what it should make, the error, can be run in reverse through the network. As you go through that reverse run, you can get an error associated with each weight. if you move that error to zero, by changing the weight, then the network can be less bad.
  
  **Some variations**:  
  Here are some approaches for relating the per-weight errors to the new weights.
  
  Approaches:
  
  * (sgd [link](https://optimization.cbe.cornell.edu/index.php?title=Stochastic_gradient_descent)) take just a few or one sample at random and compute the error, and then move weights in that direction, usually only a small percentage of it. The intention is that each new update occurs closer to the optimum, and works well in lower-noise cases.
  * (conjugate gradient [link](https://optimization.cbe.cornell.edu/index.php?title=Conjugate_gradient_methods), ) find a line along which to minimize, find the minimum, explore around that minimum to find a new direction for minimization. For quadratic problems (shaped like parabola) it gives good solutions quickly.
  * (momentum [link](https://optimization.cbe.cornell.edu/index.php?title=Momentum)) take a running average of the last several weight changes, and do most of the movement in that direction
  * (regularization [link](https://medium.com/intelligentmachines/convolutional-neural-network-and-regularization-techniques-with-tensorflow-and-keras-5a09e6e65dc7)) penalize weights that are too high, push back against learning them, tries to keep weights in smaller subset of the space unless pushed there by the data.
  * (dropout [link](https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/)) randomly set internal responses of neurons to zero, to force a robust and distributed representation of the processing in the network. Keeps on network from "dominating the show" because in practice it tends to do more harm than good for generalization.
  * (rotated locking [link](https://paperswithcode.com/method/stochastic-depth)) aka stochastic depth, randomly freeze layers in the architecture to avoid most learning occurring only in the last few layers, aka diminishing gradients.
  * (pca in vectorized weights) given a batch of errors for weights, perform pca and only retain highest x percent of eigenvalues, move in that direction.
  * (entropy-scaled momentum in pca elements) given batch of errors for weights, perform svd, and then map weight changes per sample to the largest eigenvectors associated with it
  
  
  ---
  
  