Answers.Body: <p>You do not store a terminal state as <span class="math-container">$s$</span> in the replay table because by definition its value is always <span class="math-container">$0$</span>, and there is no action, reward or next state. There is literally nothing to learn.</p>
  <p>However, you may find it useful to store information that <span class="math-container">$s'$</span> is actually a terminal state, in case this is not obvious. That is typically achieved by storing an additional <strong>done</strong> boolean component. This is useful, because it allows you to branch when calculating the TD target <strong>g</strong>:</p>
  <pre><code>s, a, r, next_s, done = replay_memory_sample()
  if done:
    g = r
  else:
    g = r + gamma * max( q(s') )
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T07:04:57.743Z
Answers.LastActivityDate: 2020-06-24T11:50:39.207Z
Answers.LastEditDate: 2020-06-24T11:50:39.207Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm using the DQN algorithm to train my agent to play a turn-based game. The memory replay buffer stores tuples of experiences <span class="math-container">$(s, a, r, s')$</span>, where <span class="math-container">$s$</span> and <span class="math-container">$s'$</span> are consecutive states. At the last turn, the game ends, and the non-zero reward is given to the agent. There are no more observations to be made and there is no next state <span class="math-container">$s'$</span> to store in the experience tuple. How should the final states be handled?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-01T19:14:19.537Z
Comments.Text: This question seems to be a duplicate of [this](https://ai.stackexchange.com/q/15918/2444).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><dqn><experience-replay>
Title: How to handle the final state in experience replay?
UserDisplayName: mark mark
ThreadText: # How to handle the final state in experience replay?
  
  (Asked by: username_0 on 2020-06-24)
  
  I'm using the DQN algorithm to train my agent to play a turn-based game. The memory replay buffer stores tuples of experiences $(s, a, r, s')$, where $s$ and $s'$ are consecutive states. At the last turn, the game ends, and the non-zero reward is given to the agent. There are no more observations to be made and there is no next state $s'$ to store in the experience tuple. How should the final states be handled?
  
  **Tags:** reinforcement-learning, dqn, experience-replay
  
  > **Comments:**
  > **username_1**: This question seems to be a duplicate of [this](https://ai.stackexchange.com/q/15918/2444).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-06-24. Score: 2 | Accepted answer
  
  You do not store a terminal state as $s$ in the replay table because by definition its value is always $0$, and there is no action, reward or next state. There is literally nothing to learn.
  
  However, you may find it useful to store information that $s'$ is actually a terminal state, in case this is not obvious. That is typically achieved by storing an additional **done** boolean component. This is useful, because it allows you to branch when calculating the TD target **g**:
  
  ```
  s, a, r, next_s, done = replay_memory_sample()
  if done:
    g = r
  else:
    g = r + gamma * max( q(s') )
  
  ```
  
  
  ---
  
  