Answers.Body: <p>You do not store a terminal state as <span class="math-container">$s$</span> in the replay table because by definition its value is always <span class="math-container">$0$</span>, and there is no action, reward or next state. There is literally nothing to learn.</p>
  <p>However, you may find it useful to store information that <span class="math-container">$s'$</span> is actually a terminal state, in case this is not obvious. That is typically achieved by storing an additional <strong>done</strong> boolean component. This is useful, because it allows you to branch when calculating the TD target <strong>g</strong>:</p>
  <pre><code>s, a, r, next_s, done = replay_memory_sample()
  if done:
    g = r
  else:
    g = r + gamma * max( q(s') )
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T07:04:57.743Z
Answers.LastActivityDate: 2020-06-24T11:50:39.207Z
Answers.LastEditDate: 2020-06-24T11:50:39.207Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm using the DQN algorithm to train my agent to play a turn-based game. The memory replay buffer stores tuples of experiences <span class="math-container">$(s, a, r, s')$</span>, where <span class="math-container">$s$</span> and <span class="math-container">$s'$</span> are consecutive states. At the last turn, the game ends, and the non-zero reward is given to the agent. There are no more observations to be made and there is no next state <span class="math-container">$s'$</span> to store in the experience tuple. How should the final states be handled?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-01T19:14:19.537Z
Comments.Text: This question seems to be a duplicate of [this](https://ai.stackexchange.com/q/15918/2444).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><dqn><experience-replay>
Title: How to handle the final state in experience replay?
UserDisplayName: mark mark
ThreadText: # How to handle the final state in experience replay?
  
  (Asked by: username_0 on 2020-06-24)
  
  I'm using the DQN algorithm to train my agent to play a turn-based game. The memory replay buffer stores tuples of experiences $(s, a, r, s')$, where $s$ and $s'$ are consecutive states. At the last turn, the game ends, and the non-zero reward is given to the agent. There are no more observations to be made and there is no next state $s'$ to store in the experience tuple. How should the final states be handled?
  
  **Tags:** reinforcement-learning, dqn, experience-replay
  > **Comments:**
  > **username_1**: This question seems to be a duplicate of [this](https://ai.stackexchange.com/q/15918/2444).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-06-24. Score: 2 | Accepted answer
  
  You do not store a terminal state as $s$ in the replay table because by definition its value is always $0$, and there is no action, reward or next state. There is literally nothing to learn.
  However, you may find it useful to store information that $s'$ is actually a terminal state, in case this is not obvious. That is typically achieved by storing an additional done boolean component. This is useful, because it allows you to branch when calculating the TD target g:
  s, a, r, next_s, done = replay_memory_sample()
  if done:
    g = r
  else:
    g = r + gamma * max( q(s') )
  
  
  ---
  
  