Answers.Body: <p>When you say simple policy gradient, I assume you mean something like REINFORCE. </p>
  
  <p>The main difference between actor-critic and REINFORCE-like algorithms is in how they estimate the reward to go: </p>
  
  <ul>
  <li><p>in REINFORCE, you wait until a trajectory terminates to make any updates, and your estimator of the reward to go is the actual reward to go that was observed in the trajectory. </p></li>
  <li><p>in actor critic algorithms, instead of using an entire trajectory to estimate a reward to go, you use the action value function (note that this way, you can compute updates at every time step). The reason why you use the action-value function is because at each transition, you have already committed to some action. </p></li>
  </ul>
  
  <p>The benefit of the actor critic estimator is that it exhibits much less variance. REINFORCE estimators are Monte Carlo estimators, which are known to exhibit extremely high variance. </p>
  
  <p>Now, the actor critic method in the slides you posted takes variance reduction one step further. Instead of estimating reward to go, it's estimating <em>advantage</em> -- that is, the difference between the current action value and your estimated state value. The state value function term doesn't depend on the action, so it doesn't affect the expected value of the policy gradient. However, it serves as a 'baseline', which helps reduce the variance of the reward to go estimator even further.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-15T16:06:49.667Z
Answers.Text: I see. Just to be clear, in REINFORCE is there a single update after a trajectory terminates or are there T updates, where for each timestep the gradient is approximated as grad log prob of policy at that timestep multiplied by reward to go from that timestep? Also then in the case of on policy actor critic, the advantage is Q-V is given as r + gamma*V'. But then we also try to get V to converge to r+gamma*V'. Could you please elaborate a bit more on the mathematical justification in using Q and V?
Answers.UserDisplayName: pranav
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-15T16:26:04.403Z
Answers.Text: Firstly, in REINFORCE, there should be T updates after each trajectory as you suggested
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-15T16:29:56.193Z
Answers.Text: For your second question, especially in the on-policy setting, you need to account for the action you took (V doesn't do this). Let's say you just observed the following transition, (s, a, r, s'). You want to use this transition to approximate the reward to go. To do this with the data you have, you're assuming the next state is s', but this is correlated with the fact that you took action a. That's why you need to use Q instead of, V, otherwise r + gamma * V(s') wouldn't necessarily be a valid estimate of the reward to go.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T03:27:48.273Z
Answers.Text: Okay. But I am still unable to understand why the advantage goes to zero once the critic gets better. We seem to set the target for V as r + gamma*V'. But the difference between the two is what we call Advantage
Answers.UserDisplayName: pranav
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T03:46:00.933Z
Answers.Text: Right, because once the critic and policy have converged, the best actions are chosen by the policy, so the Q value should equal V (so advantage is 0).
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-16T12:08:44.277Z
Answers.Text: "To do this with the data you have, you're assuming the next state is s', but this is correlated with the fact that you took action a. That's why you need to use Q instead of, V, otherwise r + gamma * V(s') wouldn't necessarily be a valid estimate of the reward to go." I understand V(s') is not the reward to go, but r + gamma * V(s') should be the reward to go on convergence? Is your point that we need to use Q(s,a) - V(s) (approximated as r + V(s')) rather than V(s') - V(s) when training the policy?
Answers.UserDisplayName: bill
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-15T15:55:51.393Z
Answers.LastActivityDate: 2020-06-15T15:55:51.393Z
Answers.UserDisplayName: harwiltz
Body: <p>While transitioning from simple policy gradient to the actor-critic algorithm, most sources begin by replacing the "reward to go" with the state-action value function (see <a href="http://rail.eecs.berkeley.edu/deeprlcourse-fa17/f17docs/lecture_5_actor_critic_pdf.pdf" rel="nofollow noreferrer">this slide 5</a>). </p>
  
  <p>I am not able to understand how this is mathematically justified. It seems intuitive to me that the "reward to go" when sampled through multiple trajectories should be estimated by the state-value function. </p>
  
  <p>I feel this way since nowhere in the objective function formulation or resulting gradient expression do we tie down the first action after reaching a state. Alternatively, when we sample a bunch of trajectories, these trajectories might include different actions being taken from the state reached in timestep <span class="math-container">$t$</span>. </p>
  
  <p>So, why isn't the estimation/approximation for the "reward to go" the state value function, in which the expectation is also over all the actions that may be taken from that state as well?</p>
  
Tags: <reinforcement-learning><actor-critic-methods><reinforce><advantage-actor-critic><reward-to-go>
Title: Why is the "reward to go" replaced by Q instead of V, when transitioning from PG to actor critic methods?
UserDisplayName: pranav
ThreadText: # Why is the "reward to go" replaced by Q instead of V, when transitioning from PG to actor critic methods?
  
  (Asked by: username_0 on 2020-06-15)
  
  While transitioning from simple policy gradient to the actor-critic algorithm, most sources begin by replacing the "reward to go" with the state-action value function (see [this slide 5](http://rail.eecs.berkeley.edu/deeprlcourse-fa17/f17docs/lecture_5_actor_critic_pdf.pdf)).
  
  I am not able to understand how this is mathematically justified. It seems intuitive to me that the "reward to go" when sampled through multiple trajectories should be estimated by the state-value function.
  
  I feel this way since nowhere in the objective function formulation or resulting gradient expression do we tie down the first action after reaching a state. Alternatively, when we sample a bunch of trajectories, these trajectories might include different actions being taken from the state reached in timestep $t$.
  
  So, why isn't the estimation/approximation for the "reward to go" the state value function, in which the expectation is also over all the actions that may be taken from that state as well?
  
  **Tags:** reinforcement-learning, actor-critic-methods, reinforce, advantage-actor-critic, reward-to-go
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-15. Score: 0 | Accepted answer
  
  When you say simple policy gradient, I assume you mean something like REINFORCE.
  
  The main difference between actor-critic and REINFORCE-like algorithms is in how they estimate the reward to go:
  
  * in REINFORCE, you wait until a trajectory terminates to make any updates, and your estimator of the reward to go is the actual reward to go that was observed in the trajectory.
  * in actor critic algorithms, instead of using an entire trajectory to estimate a reward to go, you use the action value function (note that this way, you can compute updates at every time step). The reason why you use the action-value function is because at each transition, you have already committed to some action.
  
  The benefit of the actor critic estimator is that it exhibits much less variance. REINFORCE estimators are Monte Carlo estimators, which are known to exhibit extremely high variance.
  
  Now, the actor critic method in the slides you posted takes variance reduction one step further. Instead of estimating reward to go, it's estimating *advantage* -- that is, the difference between the current action value and your estimated state value. The state value function term doesn't depend on the action, so it doesn't affect the expected value of the policy gradient. However, it serves as a 'baseline', which helps reduce the variance of the reward to go estimator even further.
  
  > **Comments:**
  > **username_0**: I see. Just to be clear, in REINFORCE is there a single update after a trajectory terminates or are there T updates, where for each timestep the gradient is approximated as grad log prob of policy at that timestep multiplied by reward to go from that timestep? Also then in the case of on policy actor critic, the advantage is Q-V is given as r + gamma*V'. But then we also try to get V to converge to r+gamma*V'. Could you please elaborate a bit more on the mathematical justification in using Q and V?  
  > **username_1**: Firstly, in REINFORCE, there should be T updates after each trajectory as you suggested  
  > **username_1**: For your second question, especially in the on-policy setting, you need to account for the action you took (V doesn't do this). Let's say you just observed the following transition, (s, a, r, s'). You want to use this transition to approximate the reward to go. To do this with the data you have, you're assuming the next state is s', but this is correlated with the fact that you took action a. That's why you need to use Q instead of, V, otherwise r + gamma * V(s') wouldn't necessarily be a valid estimate of the reward to go.  
  > **username_0**: Okay. But I am still unable to understand why the advantage goes to zero once the critic gets better. We seem to set the target for V as r + gamma*V'. But the difference between the two is what we call Advantage  
  > **username_1**: Right, because once the critic and policy have converged, the best actions are chosen by the policy, so the Q value should equal V (so advantage is 0).  
  > **username_2**: "To do this with the data you have, you're assuming the next state is s', but this is correlated with the fact that you took action a. That's why you need to use Q instead of, V, otherwise r + gamma * V(s') wouldn't necessarily be a valid estimate of the reward to go." I understand V(s') is not the reward to go, but r + gamma * V(s') should be the reward to go on convergence? Is your point that we need to use Q(s,a) - V(s) (approximated as r + V(s')) rather than V(s') - V(s) when training the policy?  
  
  ---
  
  