Answers.Body: <p><code>done[t]</code> indicates whether the state at timestep <code>t</code> was terminal or not.</p>
  <p>It is used to say when or not to <em>bootstrap</em> (i.e. estimate) the value <code>V(s)</code> of a state: we bootstrap when <code>done[t] = False</code>, providing an estimate for the future discounted reward that the agent should get in expectation (this is also useful when the episode is artificially truncated but not really terminated.) If true, the state at time <code>t</code> is terminal and so its value <code>V(state[t])</code> should be zero, because the agent landed in a terminal state that &quot;absorbs&quot; the future rewards.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T16:12:39.053Z
Answers.Text: This question and corresponding answers (https://ai.stackexchange.com/q/17890/37607) add some detail to your last sentence.
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-04T04:19:14.813Z
Answers.Text: Thank you both this clears it up for me :)
Answers.UserDisplayName: Jacob B
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T08:21:46.243Z
Answers.LastActivityDate: 2023-04-03T08:21:46.243Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>In PPO a common pattern I see in calcualting advantages is:</p>
  <p><span class="math-container">$$ delta = reward[t] + (gamma * valueNewState[t] * done[t]) - valueOldState[t]$$</span></p>
  <p>Such as in this <a href="https://towardsdatascience.com/generalized-advantage-estimate-maths-and-code-b5d5bd3ce737" rel="nofollow noreferrer">article</a>. I am wondering why we multiply by <code>done[t]</code> indicating for the last step in the epoch we take reward only and not the value of the current state.</p>
  
Tags: <reinforcement-learning><proximal-policy-optimization>
Title: Why do we not calculate value for the done state in PPO?
UserDisplayName: Jacob B
ThreadText: # Why do we not calculate value for the done state in PPO?
  
  (Asked by: username_0 on 2023-04-03)
  
  In PPO a common pattern I see in calcualting advantages is:
  
  $$ delta = reward[t] + (gamma \* valueNewState[t] \* done[t]) - valueOldState[t]$$
  
  Such as in this [article](https://towardsdatascience.com/generalized-advantage-estimate-maths-and-code-b5d5bd3ce737). I am wondering why we multiply by `done[t]` indicating for the last step in the epoch we take reward only and not the value of the current state.
  
  **Tags:** reinforcement-learning, proximal-policy-optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-03. Score: 1 | Accepted answer
  
  `done[t]` indicates whether the state at timestep `t` was terminal or not.
  
  It is used to say when or not to *bootstrap* (i.e. estimate) the value `V(s)` of a state: we bootstrap when `done[t] = False`, providing an estimate for the future discounted reward that the agent should get in expectation (this is also useful when the episode is artificially truncated but not really terminated.) If true, the state at time `t` is terminal and so its value `V(state[t])` should be zero, because the agent landed in a terminal state that "absorbs" the future rewards.
  
  > **Comments:**
  > **username_2**: This question and corresponding answers (https://ai.stackexchange.com/q/17890/37607) add some detail to your last sentence.  
  > **username_0**: Thank you both this clears it up for me :)  
  
  ---
  
  