Answers.Body: <p>It's just zeroed out (not considered), so <span class="math-container">$Q(a,s)$</span> given <span class="math-container">$s’$</span> terminal, should just be the expected immediate reward of the last step, thus the update should just be the one you reported (it's also reported in the original paper <a href="https://link.springer.com/article/10.1007/BF00992698" rel="nofollow noreferrer">https://link.springer.com/article/10.1007/BF00992698</a> in the end notes, but I'm sure to have red it also in the Sutton and Barto book
  )</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-29T08:19:50.477Z
Answers.Text: This is slightly wrong, where the OP is correct. if $s$ is terminal, then $Q(s, *) = 0$, by definition. perhaps you meant $s'$ and not $s$?
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-29T08:55:57.663Z
Answers.Text: @NeilSlater yup
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-29T00:16:31.287Z
Answers.LastActivityDate: 2023-07-29T08:55:39.893Z
Answers.LastEditDate: 2023-07-29T08:55:39.893Z
Answers.UserDisplayName: Alberto
Body: <p>In Q-learning, I know that the Q-values are updated using the Bellman equation.</p>
  <p><span class="math-container">$$
  Q^{new}(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha [R_{t+1} + \gamma \underset{a}{max} Q(S_{t+1},a) - Q(S_t,A_t)]
  $$</span></p>
  <p>However, for the last state in the Q-table, the Q-values cannot be updated using the maximum Q-value of the next state term since the next state itself does not exist.</p>
  <p>So how do we updated the Q-values for the last state? In my mind, the logical answer would be to continue using the Bellman equation but equalling the maximum Q-value for the next state term to 0, which results in:</p>
  <p><span class="math-container">$$
  Q^{new}(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha [R_{t+1} - Q(S_t,A_t)]
  $$</span></p>
  
Tags: <reinforcement-learning><q-learning>
Title: In Q-learning, how are Q values updated for the last state in the Q table?
UserDisplayName: gondorian
ThreadText: # In Q-learning, how are Q values updated for the last state in the Q table?
  
  (Asked by: username_0 on 2023-07-29)
  
  In Q-learning, I know that the Q-values are updated using the Bellman equation.
  
  $$
  Q^{new}(S\_t,A\_t) \leftarrow Q(S\_t,A\_t) + \alpha [R\_{t+1} + \gamma \underset{a}{max} Q(S\_{t+1},a) - Q(S\_t,A\_t)]
  $$
  
  However, for the last state in the Q-table, the Q-values cannot be updated using the maximum Q-value of the next state term since the next state itself does not exist.
  
  So how do we updated the Q-values for the last state? In my mind, the logical answer would be to continue using the Bellman equation but equalling the maximum Q-value for the next state term to 0, which results in:
  
  $$
  Q^{new}(S\_t,A\_t) \leftarrow Q(S\_t,A\_t) + \alpha [R\_{t+1} - Q(S\_t,A\_t)]
  $$
  
  **Tags:** reinforcement-learning, q-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-29. Score: 2 | Accepted answer
  
  It's just zeroed out (not considered), so $Q(a,s)$ given $s’$ terminal, should just be the expected immediate reward of the last step, thus the update should just be the one you reported (it's also reported in the original paper <https://link.springer.com/article/10.1007/BF00992698> in the end notes, but I'm sure to have red it also in the Sutton and Barto book
  )
  
  > **Comments:**
  > **username_2**: This is slightly wrong, where the OP is correct. if $s$ is terminal, then $Q(s, *) = 0$, by definition. perhaps you meant $s'$ and not $s$?  
  > **username_1**: @NeilSlater yup  
  
  ---
  
  