Answers.Body: <blockquote>
    <p>However, the chain rule tells us how the first variable influences the second variable, and so on. Following that logic, we should only update the weights of the first hidden layer. </p>
  </blockquote>
  
  <p>I don't see how the second statement follows from the first.</p>
  
  <p>Each weight <span class="math-container">$w_i$</span> (not just the ones in the first layer) affects the loss <span class="math-container">$\mathcal{L}$</span> according to the partial derivative of <span class="math-container">$\mathcal{L}$</span> with respect to <span class="math-container">$w_i$</span>, i.e. <span class="math-container">$\frac{\partial \mathcal{L}}{\partial w_i}$</span>. Intuitively, the partial derivative with respect to parameter tells you how the function is changing with respect to that parameter. </p>
  
  <blockquote>
    <p>My thought is, that if we backtrack the influence of the first variable but also change the values of the subsequent weights (of the subsequent hidden layer), there is no need to calculate the influence of the first weights in the first place.</p>
  </blockquote>
  
  <p>I am not sure I understand your reasoning, but, typically, you update the parameters only after having computed <em>all</em> the partial derivatives. In other words, first, you compute all partial derivatives, i.e. the gradient with back-propagation (a fancy name to denote the application of the chain rule), then you update the parameters.</p>
  
  <p>Why do you do this? In this case, the loss function is multi-variable function, so it depends on multiple variables. The gradient <span class="math-container">$\nabla \mathcal{L} = \left[ \frac{\partial \mathcal{L}}{\partial w_1}, \dots, \frac{\partial \mathcal{L}}{\partial w_N} \right]$</span> represents the direction (note that the gradient is a vector and vectors have direction) towards which your function is increasing or decreasing (depending on the sign of the gradient). </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-26T16:35:07.267Z
Answers.LastActivityDate: 2020-04-26T16:35:07.267Z
Answers.UserDisplayName: nbro
Body: <p>Why are all weights of a neural net updated and not only the weights of the first hidden layer?</p>
  
  <p>The error-influence of the prediction by the weights of a neural net is calculated using the chain rule. However, the chain rule tells us how the first variable influences the second variable, and so on. Following that logic, we should only update the weights of the first hidden layer. My thought is, that if we backtrack the influence of the first variable but also change the values of the subsequent weights (of the subsequent hidden layer), there is no need to calculate the influence of the first weights in the first place. Where am I wrong? </p>
  
Tags: <neural-networks><machine-learning><backpropagation><math>
Title: Why are all weights of a neural net updated and not just the weights of the first layer
UserDisplayName: MScott
ThreadText: # Why are all weights of a neural net updated and not just the weights of the first layer
  
  (Asked by: username_0 on 2020-04-24)
  
  Why are all weights of a neural net updated and not only the weights of the first hidden layer?
  
  The error-influence of the prediction by the weights of a neural net is calculated using the chain rule. However, the chain rule tells us how the first variable influences the second variable, and so on. Following that logic, we should only update the weights of the first hidden layer. My thought is, that if we backtrack the influence of the first variable but also change the values of the subsequent weights (of the subsequent hidden layer), there is no need to calculate the influence of the first weights in the first place. Where am I wrong?
  
  **Tags:** neural-networks, machine-learning, backpropagation, math
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-26. Score: 1
  
  > However, the chain rule tells us how the first variable influences the second variable, and so on. Following that logic, we should only update the weights of the first hidden layer.
  
  I don't see how the second statement follows from the first.
  
  Each weight $w\_i$ (not just the ones in the first layer) affects the loss $\mathcal{L}$ according to the partial derivative of $\mathcal{L}$ with respect to $w\_i$, i.e. $\frac{\partial \mathcal{L}}{\partial w\_i}$. Intuitively, the partial derivative with respect to parameter tells you how the function is changing with respect to that parameter.
  
  > My thought is, that if we backtrack the influence of the first variable but also change the values of the subsequent weights (of the subsequent hidden layer), there is no need to calculate the influence of the first weights in the first place.
  
  I am not sure I understand your reasoning, but, typically, you update the parameters only after having computed *all* the partial derivatives. In other words, first, you compute all partial derivatives, i.e. the gradient with back-propagation (a fancy name to denote the application of the chain rule), then you update the parameters.
  
  Why do you do this? In this case, the loss function is multi-variable function, so it depends on multiple variables. The gradient $\nabla \mathcal{L} = \left[ \frac{\partial \mathcal{L}}{\partial w\_1}, \dots, \frac{\partial \mathcal{L}}{\partial w\_N} \right]$ represents the direction (note that the gradient is a vector and vectors have direction) towards which your function is increasing or decreasing (depending on the sign of the gradient).
  
  
  ---
  
  