Answers.Body: <p>If the episode does not terminate naturally, then if you are breaking it up into pseudo-episodes for training purposes, the one thing you should not do is use the TD target <span class="math-container">$G_{T-1} = R_T$</span> used for an end of episode, which assumes a return of 0 from any terminal state <span class="math-container">$S_{T}$</span>. Of course that is because it is <em>not</em> the end of the episode.</p>
  <p>You have two &quot;natural&quot; options to tweak DQN to match to theory at the end of a pseudo-episode:</p>
  <ul>
  <li><p>Store the state, action, reward, next_state tuple as normal and use the standard one step TD target <span class="math-container">$G_{t:t+1} = R_{t+1} + \gamma \text{max}_{a'} Q(S_{t+1}, a')$</span></p>
  </li>
  <li><p>Completely ignore the last step and don't store it in memory. There is no benefit to this as opposed to the above option, but it might be simpler to implement if you are using a pre-built RL library.</p>
  </li>
  </ul>
  <p>Both these involve ignoring any <code>done</code> flag returned by the environment for the purposes of calculating TD targets. You still can use that flag to trigger the end of a loop and a reset to new starting state.</p>
  <p>You should also take this approach if you terminate an episodic problem early after hitting a time step limit, in order to reset for training purposes.</p>
  <hr />
  <p>As an aside (and mentioned in comment by Swakshar Deb), you can also look into the <a href="https://link.springer.com/article/10.1007/BF00114727" rel="nofollow noreferrer">average reward setting for non-episodic environments</a>. This solves the problem of needing to pick a value for <span class="math-container">$\gamma$</span>. If you have no reason to pick a specific <span class="math-container">$\gamma$</span> in a continuing problem, then it is common to pick a value close to 1 such as 0.99 or 0.999 in DQN - this is basically an approximation to average reward.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-19T09:03:54.630Z
Answers.Text: So can I use Deep Q-Learning with the differential reward for a non-episodic task with γ=1? Why if γ=0.99 as you said then this approximates the differential reward?
Answers.UserDisplayName: ddaedalus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-19T09:32:12.427Z
Answers.Text: @ddaedalus: If you use differential reward you should not use $\gamma$ at all. Mathematically $\gamma = 1$ is the same, but better to think of average reward as not having any discount factor at all. When $\gamma \sim 1$ in a discounted return, then the measure is very similar to average reward in expectation, just different by a scale factor. So a policy that maximises expected discounted return with a high $\gamma$ and no terminal states will also be very similar to a policy that maximises average reward.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-19T08:20:42.750Z
Answers.LastActivityDate: 2020-08-19T09:35:39.410Z
Answers.LastEditDate: 2020-08-19T09:35:39.410Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm training a DQN in a real environment where I do not have a natural terminal state, so I've built the episode in an artificial way (i.e. it starts in a random condition and after T steps it ends). My question is about the terminal state: should I consider it when I have to compute <span class="math-container">$y$</span> (so using only the reward) or not?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-19T07:11:11.450Z
Comments.Text: I think you can use Deep Q learning with the average reward for a non-episodic task,  so you don't need to terminate the environment artificially.
Comments.UserDisplayName: Swakshar Deb
Tags: <reinforcement-learning><dqn><environment>
Title: How should I compute the target for updating in a DQN at the terminal state if I have pseudo-episodes?
UserDisplayName: unter_983
ThreadText: # How should I compute the target for updating in a DQN at the terminal state if I have pseudo-episodes?
  
  (Asked by: username_0 on 2020-08-19)
  
  I'm training a DQN in a real environment where I do not have a natural terminal state, so I've built the episode in an artificial way (i.e. it starts in a random condition and after T steps it ends). My question is about the terminal state: should I consider it when I have to compute $y$ (so using only the reward) or not?
  
  **Tags:** reinforcement-learning, dqn, environment
  
  > **Comments:**
  > **username_1**: I think you can use Deep Q learning with the average reward for a non-episodic task,  so you don't need to terminate the environment artificially.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-08-19. Score: 3 | Accepted answer
  
  If the episode does not terminate naturally, then if you are breaking it up into pseudo-episodes for training purposes, the one thing you should not do is use the TD target $G\_{T-1} = R\_T$ used for an end of episode, which assumes a return of 0 from any terminal state $S\_{T}$. Of course that is because it is *not* the end of the episode.
  
  You have two "natural" options to tweak DQN to match to theory at the end of a pseudo-episode:
  
  * Store the state, action, reward, next\_state tuple as normal and use the standard one step TD target $G\_{t:t+1} = R\_{t+1} + \gamma \text{max}\_{a'} Q(S\_{t+1}, a')$
  * Completely ignore the last step and don't store it in memory. There is no benefit to this as opposed to the above option, but it might be simpler to implement if you are using a pre-built RL library.
  
  Both these involve ignoring any `done` flag returned by the environment for the purposes of calculating TD targets. You still can use that flag to trigger the end of a loop and a reset to new starting state.
  
  You should also take this approach if you terminate an episodic problem early after hitting a time step limit, in order to reset for training purposes.
  
  ---
  
  As an aside (and mentioned in comment by username_1), you can also look into the [average reward setting for non-episodic environments](https://link.springer.com/article/10.1007/BF00114727). This solves the problem of needing to pick a value for $\gamma$. If you have no reason to pick a specific $\gamma$ in a continuing problem, then it is common to pick a value close to 1 such as 0.99 or 0.999 in DQN - this is basically an approximation to average reward.
  
  > **Comments:**
  > **username_3**: So can I use Deep Q-Learning with the differential reward for a non-episodic task with γ=1? Why if γ=0.99 as you said then this approximates the differential reward?  
  > **username_2**: @username_3: If you use differential reward you should not use $\gamma$ at all. Mathematically $\gamma = 1$ is the same, but better to think of average reward as not having any discount factor at all. When $\gamma \sim 1$ in a discounted return, then the measure is very similar to average reward in expectation, just different by a scale factor. So a policy that maximises expected discounted return with a high $\gamma$ and no terminal states will also be very similar to a policy that maximises average reward.  
  
  ---
  
  