Answers.Body: <p>If you want to use RMSProp for a tabular Q-learning algorithm, then you will have to first identify what the gradient is (the update rule - i.e. the gradient ascent - for Q-learning clearly suggests that the td_error is the gradient). Then, you just apply that to RMSProp by replacing the gradient g by the td_error in all RMSProp equations. So, what you are suggesting seems reasonable to me!</p>
  <p>The only thing I can find to say is that you should be careful to keep a different moving average E[g^2] for each state-action pair (i.e. E[g^2] becomes a table, of the same size as Q)! (Note that the td_error is also state-action-dependent!)
  So, I would guess something like (with E[g^2] as M, for moving average):</p>
  <p>M(s, a) &lt;- beta * M(s, a) + (1-beta) * td_error(s, a)**2</p>
  <p>Q(s,a) &lt;- Q(s,a) + alpha_0 * td_error / np.sqrt(M(s, a)+epsilon).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-11T12:58:56.800Z
Answers.Text: Thanks for your reply. I applied this two equations in my q-update function. However, it didn't work for me. Starting alpha at 0.001, i was expecting its value to increase over time, but still always at 0.0001 with very small variation
Answers.UserDisplayName: Marouane Ben-akka
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-11T09:58:54.803Z
Answers.LastActivityDate: 2024-11-11T09:58:54.803Z
Answers.UserDisplayName: Ahnel
Body: <p>I am new to this group,
  Anybody familiar with Q-learning algorithm and RMSprop approach ?
  i have a question regarding the application of RMSprop approach into Q-Learning to adapt dynamically the learning rate for each Q(s,a).
  I am confused how to compute the average square gradient in tabular Q-Learning. Is it E[g^2] = beta * E[g^2]+ (1-beta) * td_error**2. ?
  and what would be the formula to update Q(s,a) ? Is it Q(s,a) &lt;- Q(s,a) + alpha_0 * td_error/np.sqrt(E[g^2]+epsilon) ?
  Thank you for help in advance ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-11T11:25:37.467Z
Comments.Text: Can you please edit your post to use mathjax to format the math formulas?
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><q-learning><temporal-difference-methods><root-mean-square>
Title: RMSprop approach applied to Q-learning for adaptive dynamic learning rate
UserDisplayName: Marouane Ben-akka
ThreadText: # RMSprop approach applied to Q-learning for adaptive dynamic learning rate
  
  (Asked by: username_0 on 2024-11-10)
  
  I am new to this group,
  Anybody familiar with Q-learning algorithm and RMSprop approach ?
  i have a question regarding the application of RMSprop approach into Q-Learning to adapt dynamically the learning rate for each Q(s,a).
  I am confused how to compute the average square gradient in tabular Q-Learning. Is it E[g^2] = beta \* E[g^2]+ (1-beta) \* td\_error\*\*2. ?
  and what would be the formula to update Q(s,a) ? Is it Q(s,a) <- Q(s,a) + alpha\_0 \* td\_error/np.sqrt(E[g^2]+epsilon) ?
  Thank you for help in advance ?
  
  **Tags:** reinforcement-learning, q-learning, temporal-difference-methods, root-mean-square
  
  > **Comments:**
  > **username_1**: Can you please edit your post to use mathjax to format the math formulas?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-11-11. Score: 0
  
  If you want to use RMSProp for a tabular Q-learning algorithm, then you will have to first identify what the gradient is (the update rule - i.e. the gradient ascent - for Q-learning clearly suggests that the td\_error is the gradient). Then, you just apply that to RMSProp by replacing the gradient g by the td\_error in all RMSProp equations. So, what you are suggesting seems reasonable to me!
  
  The only thing I can find to say is that you should be careful to keep a different moving average E[g^2] for each state-action pair (i.e. E[g^2] becomes a table, of the same size as Q)! (Note that the td\_error is also state-action-dependent!)
  So, I would guess something like (with E[g^2] as M, for moving average):
  
  M(s, a) <- beta \* M(s, a) + (1-beta) \* td\_error(s, a)\*\*2
  
  Q(s,a) <- Q(s,a) + alpha\_0 \* td\_error / np.sqrt(M(s, a)+epsilon).
  
  > **Comments:**
  > **username_0**: Thanks for your reply. I applied this two equations in my q-update function. However, it didn't work for me. Starting alpha at 0.001, i was expecting its value to increase over time, but still always at 0.0001 with very small variation  
  
  ---
  
  