Answers.Body: <p>&quot;<em>Rprop is equivalent to using the gradient</em>&quot; means Rprop fundamentally relies on information about the sign of the gradient of each weight to determine the direction of weight updates. Like traditional gradient-based methods, Rprop utilizes information about whether the gradient is positive or negative to guide the weight updates.</p>
  <p>The second part of Hinton's statement &quot;<em>but also dividing by the size of the gradient</em>&quot; roughly draws only an analogy to the concept of adaptive adjustment of the step size based on the magnitude of the gradient change for each weight instead of the usual fixed step size. The net effect is similar to the idea of always normalizing the gradient to have a fixed <em>unit</em> magnitude for weight updates to be consistently controlled and stable which ideally has no or little overshooting or oscillating convergence stability issue if an optimal solution actually exists.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-04T23:57:24.577Z
Answers.LastActivityDate: 2024-02-04T23:57:24.577Z
Answers.UserDisplayName: cinch
Body: <p>Been reviewing some old foundational material and ran into this comment by Hinton on Rprop in his old Coursera class:</p>
  <blockquote>
  <p>Rprop is equivalent to using the gradient, but also dividing by the
  size of the gradient</p>
  </blockquote>
  <p>I don't understand this statement. Dividing by the size of the gradient would to me sound like scaling the gradient to be a unit vector. My understanding of Rprop does nothing like that (i.e. it just starts with small step size and then scales each dimension separately based on adjacent signs of the gradient).</p>
  <p>Can someone elaborate on the details?</p>
  
Tags: <backpropagation><objective-functions><optimization><gradient-descent>
Title: Can you explain the Hinton's comment "Rprop is equivalent to using the gradient, but also dividing by the size of the gradient"?
UserDisplayName: eof
ThreadText: # Can you explain the Hinton's comment "Rprop is equivalent to using the gradient, but also dividing by the size of the gradient"?
  
  (Asked by: username_0 on 2024-02-04)
  
  Been reviewing some old foundational material and ran into this comment by Hinton on Rprop in his old Coursera class:
  
  > Rprop is equivalent to using the gradient, but also dividing by the
  > size of the gradient
  
  I don't understand this statement. Dividing by the size of the gradient would to me sound like scaling the gradient to be a unit vector. My understanding of Rprop does nothing like that (i.e. it just starts with small step size and then scales each dimension separately based on adjacent signs of the gradient).
  
  Can someone elaborate on the details?
  
  **Tags:** backpropagation, objective-functions, optimization, gradient-descent
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-04. Score: 1
  
  "*Rprop is equivalent to using the gradient*" means Rprop fundamentally relies on information about the sign of the gradient of each weight to determine the direction of weight updates. Like traditional gradient-based methods, Rprop utilizes information about whether the gradient is positive or negative to guide the weight updates.
  
  The second part of Hinton's statement "*but also dividing by the size of the gradient*" roughly draws only an analogy to the concept of adaptive adjustment of the step size based on the magnitude of the gradient change for each weight instead of the usual fixed step size. The net effect is similar to the idea of always normalizing the gradient to have a fixed *unit* magnitude for weight updates to be consistently controlled and stable which ideally has no or little overshooting or oscillating convergence stability issue if an optimal solution actually exists.
  
  
  ---
  
  