Answers.Body: <p>It's an important feature, and you drop it at the risk of the agent failing to learn successfully.</p>
  <p>The difference between the TD target without the terminal flag</p>
  <p><span class="math-container">$$G_t = R_{t+1} + \gamma \text{max}_{a'} Q(S_{t+1}, a')$$</span></p>
  <p>and with the terminal flag applied to <span class="math-container">$S_{t+1} = S_T$</span></p>
  <p><span class="math-container">$$G_t = R_{t+1}$$</span></p>
  <p>is important whenever <span class="math-container">$Q(S_{T}, a')$</span> might be evaluated as non-zero. The true action value of the terminal state is always defined as zero.</p>
  <p>In some cicumstances, values based on a function approximator estimate could be far from zero - if at any point the estimator moves significantly away from zero then this could lead to value estimates diverging. This can be a problem at any time, because the estimator is usually not trained on actions taken from the terminal state - due to them never being observed. In principle you could add training data to keep the function approximator in line with this estimate, but as it is an absolute value it is far more common (when using approximators such as neural networks) to change the TD target as above.</p>
  <p>Double Q learning improves estimates of bootstrap values by removing some of the maximisation bias from the next action value. This may help in your case, if the expected future rewards from the terminal state remain close to zero.</p>
  <p>I would expect Double Q learning to work OK without using terminal state informaton, and without adding &quot;mitigating&quot; training to keep terminal state estimates close to zero, <em>if rewards are not sparse, and do not increase close to the terminal states</em>. That would mean the estimator should recognise expected future reward becomes closer to zero as the agent approaches a terminal state, thus would not take much extrapolation to predict close to zero for the terminal state - close enough that it does not create runaway feedback or influence the optimal policy much.</p>
  <p>With sparse rewards, or with significantly high rewards close to the terminal state (e.g. a goal state which the agent must reach to gain all the reward), then using the terminal flag in the normal way becomes more important. It is unlikely that double Q learning would help much in that case. However, it may still be possible to find the optimal policy but with highly inaccurate action values (e.g. action values could all be double what they should be).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-02T09:47:41.083Z
Answers.LastActivityDate: 2021-10-02T09:55:27.870Z
Answers.LastEditDate: 2021-10-02T09:55:27.870Z
Answers.UserDisplayName: Neil Slater
Body: <p>I just got my agent training, and I'm wondering if the terminal flags are necessary when sampling from the replay buffer. The game I'm implementing the agent in has two different ways the game can end, and so far my agent seems to be learning without terminal flags. I was wondering how important this feature is, as it's in all the pseudocode but doesn't seem to be necessary in my implementation.</p>
  
Tags: <reinforcement-learning><deep-learning><deep-rl><keras><monte-carlo-tree-search>
Title: Do you need a terminal state when using double deep q networks?
UserDisplayName: Arlo Rostirolla
ThreadText: # Do you need a terminal state when using double deep q networks?
  
  (Asked by: username_0 on 2021-10-02)
  
  I just got my agent training, and I'm wondering if the terminal flags are necessary when sampling from the replay buffer. The game I'm implementing the agent in has two different ways the game can end, and so far my agent seems to be learning without terminal flags. I was wondering how important this feature is, as it's in all the pseudocode but doesn't seem to be necessary in my implementation.
  
  **Tags:** reinforcement-learning, deep-learning, deep-rl, keras, monte-carlo-tree-search
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-10-02. Score: 1
  
  It's an important feature, and you drop it at the risk of the agent failing to learn successfully.
  
  The difference between the TD target without the terminal flag
  
  $$G\_t = R\_{t+1} + \gamma \text{max}\_{a'} Q(S\_{t+1}, a')$$
  
  and with the terminal flag applied to $S\_{t+1} = S\_T$
  
  $$G\_t = R\_{t+1}$$
  
  is important whenever $Q(S\_{T}, a')$ might be evaluated as non-zero. The true action value of the terminal state is always defined as zero.
  
  In some cicumstances, values based on a function approximator estimate could be far from zero - if at any point the estimator moves significantly away from zero then this could lead to value estimates diverging. This can be a problem at any time, because the estimator is usually not trained on actions taken from the terminal state - due to them never being observed. In principle you could add training data to keep the function approximator in line with this estimate, but as it is an absolute value it is far more common (when using approximators such as neural networks) to change the TD target as above.
  
  Double Q learning improves estimates of bootstrap values by removing some of the maximisation bias from the next action value. This may help in your case, if the expected future rewards from the terminal state remain close to zero.
  
  I would expect Double Q learning to work OK without using terminal state informaton, and without adding "mitigating" training to keep terminal state estimates close to zero, *if rewards are not sparse, and do not increase close to the terminal states*. That would mean the estimator should recognise expected future reward becomes closer to zero as the agent approaches a terminal state, thus would not take much extrapolation to predict close to zero for the terminal state - close enough that it does not create runaway feedback or influence the optimal policy much.
  
  With sparse rewards, or with significantly high rewards close to the terminal state (e.g. a goal state which the agent must reach to gain all the reward), then using the terminal flag in the normal way becomes more important. It is unlikely that double Q learning would help much in that case. However, it may still be possible to find the optimal policy but with highly inaccurate action values (e.g. action values could all be double what they should be).
  
  
  ---
  
  