Answers.Body: <p>So the loss function that you want to optimize to improve your policy is given by:</p>
  <p><span class="math-container">$$ L^{CLIP} = \mathbb{E} \Big[ \frac{\pi_\theta (a|s)}{\pi_{\theta_{old}}(a|s)} \Big( R(s,a) - V_\phi(s) \Big) \Big]. $$</span></p>
  <p>And to improve the value network you optimize a squared-error loss between the output of your value network and the return:</p>
  <p><span class="math-container">$$ L^{VF} = 0.5\Big(V_\phi(s) - R(s,a)\Big)^2.$$</span></p>
  <p>And really the question is: <em>How do you compute the return <span class="math-container">$R(s,a)$</span>?</em></p>
  <p>If you use a Monte-Carlo estimate of the return, i.e.
  <span class="math-container">$$ R(s_t,a_t) = \sum_{i=t}^{T} r_{i+1},$$</span>
  well, then you have to rollout the episode until a terminal state is reached. Only then you can calculate the returns for each state that was visited during the rollout, and update the policy and the value networks.</p>
  <p>If, instead, you use a one-step bootstrapped estimate (also called TD), i.e.
  <span class="math-container">$$ R(s_t,a_t) = r_{t+1} + V_\phi(s_{t+1}),$$</span>
  then you can actually update the networks after every single step.</p>
  <p>What is usually done in practice is to use an n-step bootstrapped estimate:
  <span class="math-container">$$ R(s_t, a_t) = r_{t+1} + r_{t+2} + \cdots + r_{t+n} + V(s_{t+n}).$$</span>
  Now you rollout <span class="math-container">$n$</span> steps and then you update.</p>
  <p>Even better than the n-step bootstrapped estimate would be to use the Generalized Advantage Estimate (GAE).</p>
  <p>If you want to read more about all of these please check out:<br />
  <a href="https://pi-tau.github.io/posts/actor-critic/" rel="nofollow noreferrer">https://pi-tau.github.io/posts/actor-critic/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-26T22:48:44.117Z
Answers.LastActivityDate: 2023-08-26T22:48:44.117Z
Answers.UserDisplayName: pi-tau
Body: <p>In many places, it says PPO and Actor-Critic methods in general use TD-updates, but in the loss function for PPO, the Value function loss component uses the difference between output of the value function and the value target, which I can only assume is the discounted sum of rewards that can only be obtained at the END of the episode?</p>
  <p>So this might be a moment of stupidity for me, but</p>
  <ol>
  <li><p>Is the value target in PPO set only at the end of the episode using the discounted sum of rewards? or is there a secret way of setting these value targets that I am missing?</p>
  </li>
  <li><p>If a learning update indeed takes place every learning step (before the end of the episode), then how does this TD-learning happen - does it use some other approximate of the value target?</p>
  </li>
  </ol>
  <p>Thank you.
  Please help.</p>
  <p>Sincerely,
  a frustrated student</p>
  
Tags: <reinforcement-learning><deep-rl><actor-critic-methods><value-functions><proximal-policy-optimization>
Title: PPO when does the update happen?
UserDisplayName: hridayns
ThreadText: # PPO when does the update happen?
  
  (Asked by: username_0 on 2021-09-01)
  
  In many places, it says PPO and Actor-Critic methods in general use TD-updates, but in the loss function for PPO, the Value function loss component uses the difference between output of the value function and the value target, which I can only assume is the discounted sum of rewards that can only be obtained at the END of the episode?
  
  So this might be a moment of stupidity for me, but
  
  1. Is the value target in PPO set only at the end of the episode using the discounted sum of rewards? or is there a secret way of setting these value targets that I am missing?
  2. If a learning update indeed takes place every learning step (before the end of the episode), then how does this TD-learning happen - does it use some other approximate of the value target?
  
  Thank you.
  Please help.
  
  Sincerely,
  a frustrated student
  
  **Tags:** reinforcement-learning, deep-rl, actor-critic-methods, value-functions, proximal-policy-optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-26. Score: 0
  
  So the loss function that you want to optimize to improve your policy is given by:
  
  $$ L^{CLIP} = \mathbb{E} \Big[ \frac{\pi\_\theta (a|s)}{\pi\_{\theta\_{old}}(a|s)} \Big( R(s,a) - V\_\phi(s) \Big) \Big]. $$
  
  And to improve the value network you optimize a squared-error loss between the output of your value network and the return:
  
  $$ L^{VF} = 0.5\Big(V\_\phi(s) - R(s,a)\Big)^2.$$
  
  And really the question is: *How do you compute the return $R(s,a)$?*
  
  If you use a Monte-Carlo estimate of the return, i.e.
  $$ R(s\_t,a\_t) = \sum\_{i=t}^{T} r\_{i+1},$$
  well, then you have to rollout the episode until a terminal state is reached. Only then you can calculate the returns for each state that was visited during the rollout, and update the policy and the value networks.
  
  If, instead, you use a one-step bootstrapped estimate (also called TD), i.e.
  $$ R(s\_t,a\_t) = r\_{t+1} + V\_\phi(s\_{t+1}),$$
  then you can actually update the networks after every single step.
  
  What is usually done in practice is to use an n-step bootstrapped estimate:
  $$ R(s\_t, a\_t) = r\_{t+1} + r\_{t+2} + \cdots + r\_{t+n} + V(s\_{t+n}).$$
  Now you rollout $n$ steps and then you update.
  
  Even better than the n-step bootstrapped estimate would be to use the Generalized Advantage Estimate (GAE).
  
  If you want to read more about all of these please check out:  
  <https://pi-tau.github.io/posts/actor-critic/>
  
  
  ---
  
  