Answers.Body: <p>If the state not changing is a valid transition for the environment (e.g. due to an action result such as running into a wall in a maze, or just random chance), then yes it should be accounted for as normal in updates. After all, time is still ticking, and it will impact mean reward or discounted return calculations.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-11T16:06:33.293Z
Answers.LastActivityDate: 2023-05-11T16:06:33.293Z
Answers.UserDisplayName: Neil Slater
Body: <p>Assuming the agent in my environment does an action, however the agent's state does not change. Does that mean the Q-Table gets updated regardless of respective states (current and next) being the same, or do we skip calculating a new Q-Value in this instance?</p>
  
Tags: <reinforcement-learning><q-learning>
Title: Should the Q-Value of a state-action tuple be updated, if $s_{t} == s_{t+1}$
UserDisplayName: Ralph
ThreadText: # Should the Q-Value of a state-action tuple be updated, if $s_{t} == s_{t+1}$
  
  (Asked by: username_0 on 2023-05-11)
  
  Assuming the agent in my environment does an action, however the agent's state does not change. Does that mean the Q-Table gets updated regardless of respective states (current and next) being the same, or do we skip calculating a new Q-Value in this instance?
  
  **Tags:** reinforcement-learning, q-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-05-11. Score: 3 | Accepted answer
  
  If the state not changing is a valid transition for the environment (e.g. due to an action result such as running into a wall in a maze, or just random chance), then yes it should be accounted for as normal in updates. After all, time is still ticking, and it will impact mean reward or discounted return calculations.
  
  
  ---
  
  