Body: <p>I'm reading the book Grokking Deep Learning. Regarding weight updates during training, it has the following code and explanation:</p>
  <pre><code>direction_and_amount = (pred - goal_pred) * input
  weight = weight - direction_and_amount
  </code></pre>
  <p>It explains the motivation behind multiplying the prediction difference with <strong>input</strong> using three cases: scaling, negative reversal and stopping.</p>
  <blockquote>
  <p>What are scaling, negative reversal, and stopping? These three attributes have the combined effect of translating the pure error into the absolute amount you want to change weight.  They do so by addressing three major edge cases where the pure error isn’t sufficient to make a good modification to weight.</p>
  </blockquote>
  <p>These three cases are:</p>
  <ol>
  <li>Negative input,</li>
  <li>zero input and</li>
  <li>the value of input (scaling).</li>
  </ol>
  <p>Negative and zero cases are very obvious. However, I didn't understand scaling. Regarding scaling, there's the following explanation:</p>
  <blockquote>
  <p>Scaling is the third effect on the pure error caused by multiplying it by input. Logically, if the input is big, your weight update should also be big. This is more of a side effect, because it often goes out of control. Later, you’ll use alpha to address when that happens.</p>
  </blockquote>
  <p>But I didn't understand it. Considering the linear regression problem, why weight update should be big if the input is big?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-10T14:33:15.357Z
Comments.Text: Just to be clear, what is the loss function the book is using in that section? What neural network is the book trying to describe the training of? This seems to be just _an interpretation_ of the derivative of the loss function (i.e. back-propagation, aka the application of the chain rule to compute derivatives of compositive functions).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-10T15:14:20.500Z
Comments.Text: Thanks for the response. It's MSE loss. Well, it's just an introduction in the book, single input single output linear function. I also read many loss and transfer/activation functions. In every combination, somehow error is multiplied by the input at the end. I know this multiplication comes from chain rule, but I want to know if there can be an intuition behind it other than calculus definition.
Comments.UserDisplayName: user42191
Tags: <neural-networks><deep-learning><training><backpropagation><gradient-descent>
Title: Why should the weight updates be proportional to input?
UserDisplayName: user42191
ThreadText: # Why should the weight updates be proportional to input?
  
  (Asked by: username_0 on 2020-11-10)
  
  I'm reading the book Grokking Deep Learning. Regarding weight updates during training, it has the following code and explanation:
  direction_and_amount = (pred - goal_pred) * input
  weight = weight - direction_and_amount
  
  It explains the motivation behind multiplying the prediction difference with input using three cases: scaling, negative reversal and stopping.
  
  What are scaling, negative reversal, and stopping? These three attributes have the combined effect of translating the pure error into the absolute amount you want to change weight.  They do so by addressing three major edge cases where the pure error isn’t sufficient to make a good modification to weight.
  
  These three cases are:
  
  Negative input,
  zero input and
  the value of input (scaling).
  
  Negative and zero cases are very obvious. However, I didn't understand scaling. Regarding scaling, there's the following explanation:
  
  Scaling is the third effect on the pure error caused by multiplying it by input. Logically, if the input is big, your weight update should also be big. This is more of a side effect, because it often goes out of control. Later, you’ll use alpha to address when that happens.
  
  But I didn't understand it. Considering the linear regression problem, why weight update should be big if the input is big?
  
  **Tags:** neural-networks, deep-learning, training, backpropagation, gradient-descent
  > **Comments:**
  > **username_1**: Just to be clear, what is the loss function the book is using in that section? What neural network is the book trying to describe the training of? This seems to be just _an interpretation_ of the derivative of the loss function (i.e. back-propagation, aka the application of the chain rule to compute derivatives of compositive functions).  
  > **username_0**: Thanks for the response. It's MSE loss. Well, it's just an introduction in the book, single input single output linear function. I also read many loss and transfer/activation functions. In every combination, somehow error is multiplied by the input at the end. I know this multiplication comes from chain rule, but I want to know if there can be an intuition behind it other than calculus definition.  
  
  ---
  
  ## 0 answers
  
  