Answers.Body: <p>It's possible for the Q values to grow indefinitely as you continue to train and update them. This can especially happen if the rewards are unbounded for example. But more commonly, Q values will eventually reach an equilibrium where they stop changing significantly.</p>
  <p>Of course the speed and way your Q values change can also depend on the parameters:</p>
  <ul>
  <li><p>the Q values will converge to the true action-value function if your implementation of your algorithm is correct. However if the q values aren't being updated correctly or are initialised poorly they may not converge (at least to the correct values)</p>
  </li>
  <li><p>the size of the learning rate if it's too high it will cause the Q values to change more quickly for example</p>
  </li>
  </ul>
  <p>I can't really say whether it is normal for your Q values to reach 500 without more context on your implementation exactly.</p>
  <p>Generally speaking though, it's possible that the Q values are behaving as expected given the structure of your parameters and environment so you just might have to be patient. But there might also be an issue with your implementation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T14:50:22.477Z
Answers.LastActivityDate: 2023-01-08T14:50:22.477Z
Answers.UserDisplayName: kimo26
Body: <p>In Q-learning,the q values can be updated by the bellman equation. What happens with my Q values is that they keep going up forever, in accordance with the more I train. After 10,000 training episodes, is it normal that my q values will be around 500?  Will there ever be a point where they level off or will they keep going up forever?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-08T10:42:55.090Z
Comments.Text: It would help if you could give some details about your environment and the Q-learning setup you are using (use [edit] to put them in the question). Some variants of Q-learning are inherently unstable and will do this even if everything else is alright. But also if you have a non-episodic problem and have not implemented discounting you would see the same. There may be other causes, depending on details of your environment, the specific Q-learning method or your choice of hyperparameters.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-09T15:35:33.227Z
Comments.Text: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comments.UserDisplayName: Community
Tags: <reinforcement-learning><q-learning><value-functions><reward-functions>
Title: Will my Q values keep going up forever?
UserDisplayName: Kyotiq
ThreadText: # Will my Q values keep going up forever?
  
  (Asked by: username_0 on 2023-01-08)
  
  In Q-learning,the q values can be updated by the bellman equation. What happens with my Q values is that they keep going up forever, in accordance with the more I train. After 10,000 training episodes, is it normal that my q values will be around 500? Will there ever be a point where they level off or will they keep going up forever?
  
  **Tags:** reinforcement-learning, q-learning, value-functions, reward-functions
  > **Comments:**
  > **username_1**: It would help if you could give some details about your environment and the Q-learning setup you are using (use [edit] to put them in the question). Some variants of Q-learning are inherently unstable and will do this even if everything else is alright. But also if you have a non-episodic problem and have not implemented discounting you would see the same. There may be other causes, depending on details of your environment, the specific Q-learning method or your choice of hyperparameters.  
  > **username_2**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-01-08. Score: 1
  
  It's possible for the Q values to grow indefinitely as you continue to train and update them. This can especially happen if the rewards are unbounded for example. But more commonly, Q values will eventually reach an equilibrium where they stop changing significantly.
  
  Of course the speed and way your Q values change can also depend on the parameters:
  
  * the Q values will converge to the true action-value function if your implementation of your algorithm is correct. However if the q values aren't being updated correctly or are initialised poorly they may not converge (at least to the correct values)
  * the size of the learning rate if it's too high it will cause the Q values to change more quickly for example
  
  I can't really say whether it is normal for your Q values to reach 500 without more context on your implementation exactly.
  
  Generally speaking though, it's possible that the Q values are behaving as expected given the structure of your parameters and environment so you just might have to be patient. But there might also be an issue with your implementation.
  
  
  ---
  
  