Answers.Body: <p>as reported in the <a href="https://arxiv.org/pdf/1412.6980.pdf" rel="nofollow noreferrer">original paper</a>:</p>
  <p><a href="https://i.sstatic.net/JGPf0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JGPf0.png" alt="enter image description here" /></a></p>
  <p>You can see that <span class="math-container">$v^t$</span> is a (corrected) exponential moving average of the second moment of the gradient</p>
  <p>This doesn't mean that the gradient is penalized if it has entries very big, but that if during time, there are some entries that are constantly big, then they should be &quot;penalized&quot; (stepsize-wise)</p>
  <p>Moreover, consider that being the second moment, it discards the sign of it. Indeed, this is usually done to reduce zig-zagging behaviors, because the &quot;zipzag&quot; will cause in some directions big gradients with opposite signs (but the second moment will discard such sign, only caring about the magnitude, and thus penalizing such directions)</p>
  <p><a href="https://i.sstatic.net/6mCst.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6mCst.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-08T08:45:28.410Z
Answers.LastActivityDate: 2024-04-08T08:45:28.410Z
Answers.UserDisplayName: Alberto
Body: <p>Hello I'm learning optimizers now, I can understand the momentum part (similar to physics world), but confused about different learning rate of different parameters,</p>
  <p>for Adagrad/Rmsprop, if <span class="math-container">$∂L/∂w_1$</span> is large, then learning rate for <span class="math-container">$w_1$</span> is small, if ∂L/∂w_1 is small, then learning rate for <span class="math-container">$w_1$</span> is large. But mathematically, the -gradient is the steepest direction of value decreasing, for Adagrad/Rmsprop, it essential changes this direction to other direction, essentially changes the update direction more towards those partial derivative is small (if <span class="math-container">$∂L/∂w_1$</span> is small)</p>
  <p>Why is that? My explain is, since Adagrad/Rmsprop essentially changes the update direction more towards those partial derivative is small,say <span class="math-container">$w_1$</span>(if <span class="math-container">$∂L/∂w_1$</span> is small), that equals after take a step at -gradient direction, then take an extra step at <span class="math-container">$w_1$</span> direction since <span class="math-container">$w_1$</span> direction is flatter so it's less risk to take an extra step at <span class="math-container">$w_1$</span>?</p>
  
Tags: <optimization>
Title: Confusion about Adagrad/Rmsprop/Adam about the direction of change
UserDisplayName: femto
ThreadText: # Confusion about Adagrad/Rmsprop/Adam about the direction of change
  
  (Asked by: username_0 on 2024-04-08)
  
  Hello I'm learning optimizers now, I can understand the momentum part (similar to physics world), but confused about different learning rate of different parameters,
  
  for Adagrad/Rmsprop, if $∂L/∂w\_1$ is large, then learning rate for $w\_1$ is small, if ∂L/∂w\_1 is small, then learning rate for $w\_1$ is large. But mathematically, the -gradient is the steepest direction of value decreasing, for Adagrad/Rmsprop, it essential changes this direction to other direction, essentially changes the update direction more towards those partial derivative is small (if $∂L/∂w\_1$ is small)
  
  Why is that? My explain is, since Adagrad/Rmsprop essentially changes the update direction more towards those partial derivative is small,say $w\_1$(if $∂L/∂w\_1$ is small), that equals after take a step at -gradient direction, then take an extra step at $w\_1$ direction since $w\_1$ direction is flatter so it's less risk to take an extra step at $w\_1$?
  
  **Tags:** optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-04-08. Score: 1
  
  as reported in the [original paper](https://arxiv.org/pdf/1412.6980.pdf):
  
  [![enter image description here](https://i.sstatic.net/JGPf0.png)](https://i.sstatic.net/JGPf0.png)
  
  You can see that $v^t$ is a (corrected) exponential moving average of the second moment of the gradient
  
  This doesn't mean that the gradient is penalized if it has entries very big, but that if during time, there are some entries that are constantly big, then they should be "penalized" (stepsize-wise)
  
  Moreover, consider that being the second moment, it discards the sign of it. Indeed, this is usually done to reduce zig-zagging behaviors, because the "zipzag" will cause in some directions big gradients with opposite signs (but the second moment will discard such sign, only caring about the magnitude, and thus penalizing such directions)
  
  [![enter image description here](https://i.sstatic.net/6mCst.png)](https://i.sstatic.net/6mCst.png)
  
  
  ---
  
  