Answers.Body: <h3>tl;dr</h3>
  <p>The whole point of gradient descent is to assess the <strong>contribution</strong> of each parameter towards the loss. This information is uncovered through the gradient of the loss w.r.t each parameter.</p>
  <h3>A deeper look...</h3>
  <p>Suppose we have a NN with parameters <span class="math-container">$w_{i}, \; i={1, 2, ...}$</span>. This NN makes some predictions, which we compare to the actual targets and compute a loss <span class="math-container">$J$</span>. The loss (or cost) function tells us <strong>how far off</strong> we are from the target. This is what we want to <strong>reduce</strong>, so that the predictions fall closer to the target.</p>
  <p>By computing the partial derivative of the loss function <span class="math-container">$J$</span> w.r.t a parameter <span class="math-container">$w_i$</span> (so this is just one partial derivative and not the full gradient vector)</p>
  <p><span class="math-container">$$
  \frac{\partial J}{\partial w_i}
  $$</span></p>
  <p>the NN uncovers two pieces of information:</p>
  <ul>
  <li>The <strong>slope</strong> of <span class="math-container">$J$</span> w.r.t <span class="math-container">$w_i$</span>, which tells the NN <em>how much <span class="math-container">$w_i$</span> affects <span class="math-container">$J$</span></em>.</li>
  <li>Its <strong>sign</strong>, which tells the NN <em>which way to tweak <span class="math-container">$w_i$</span> to decrease (or increase) the value of <span class="math-container">$J$</span></em>.</li>
  </ul>
  <p>By making the parameter updates depend on the derivative</p>
  <p><span class="math-container">$$
  w_i^{new} \leftarrow w_i^{old} - \lambda \frac{\partial J}{\partial w_i}
  $$</span></p>
  <p>the NN causes parameters that affect the loss the most, to be updated the most.</p>
  <p>You can think of this as: parameters that are more to blame for the network's mistakes (i.e. contribute more towards the loss) are forced to change the most, in the direction that will decrease the loss.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-22T23:58:17.790Z
Answers.LastActivityDate: 2021-01-20T21:46:20.003Z
Answers.LastEditDate: 2021-01-20T21:46:20.003Z
Answers.UserDisplayName: Djib2011
Body: <p>I know backpropagation uses cost and gradient descent to tweak the weights to minimize the cost. But how does it know which weights to give more weight to in the first place? Is there something inside each neuron in the hidden layers that defines how this is an important neuron for the correct result in some way? How does the network know how to tweak those weights for that specific neuron?</p>
  
Tags: <neural-networks><deep-learning><backpropagation><weights>
Title: How does the neural-network know how to tweak weights for a specific neuron?
UserDisplayName: namo
ThreadText: # How does the neural-network know how to tweak weights for a specific neuron?
  
  (Asked by: username_0 on 2019-11-22)
  
  I know backpropagation uses cost and gradient descent to tweak the weights to minimize the cost. But how does it know which weights to give more weight to in the first place? Is there something inside each neuron in the hidden layers that defines how this is an important neuron for the correct result in some way? How does the network know how to tweak those weights for that specific neuron?
  
  **Tags:** neural-networks, deep-learning, backpropagation, weights
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-22. Score: 3
  
  ### tl;dr
  
  The whole point of gradient descent is to assess the **contribution** of each parameter towards the loss. This information is uncovered through the gradient of the loss w.r.t each parameter.
  
  ### A deeper look...
  
  Suppose we have a NN with parameters $w\_{i}, \; i={1, 2, ...}$. This NN makes some predictions, which we compare to the actual targets and compute a loss $J$. The loss (or cost) function tells us **how far off** we are from the target. This is what we want to **reduce**, so that the predictions fall closer to the target.
  
  By computing the partial derivative of the loss function $J$ w.r.t a parameter $w\_i$ (so this is just one partial derivative and not the full gradient vector)
  
  $$
  \frac{\partial J}{\partial w\_i}
  $$
  
  the NN uncovers two pieces of information:
  
  * The **slope** of $J$ w.r.t $w\_i$, which tells the NN *how much $w\_i$ affects $J$*.
  * Its **sign**, which tells the NN *which way to tweak $w\_i$ to decrease (or increase) the value of $J$*.
  
  By making the parameter updates depend on the derivative
  
  $$
  w\_i^{new} \leftarrow w\_i^{old} - \lambda \frac{\partial J}{\partial w\_i}
  $$
  
  the NN causes parameters that affect the loss the most, to be updated the most.
  
  You can think of this as: parameters that are more to blame for the network's mistakes (i.e. contribute more towards the loss) are forced to change the most, in the direction that will decrease the loss.
  
  
  ---
  
  