Answers.Body: <p>Given the formula for the policy gradient with baseline:</p>
  <p><span class="math-container">$$ \nabla J(\theta) = \mathbb{E}_{a,s \sim \pi_\theta} \bigg[ \nabla \log \pi_\theta(a|s) \Big(R(s, a) - V_\phi(s) \Big) \bigg] $$</span></p>
  <p>How do you compute the return <span class="math-container">$R(s,a)$</span> ?<br />
  If you use a simple Monte-Carlo estimate, i.e. <span class="math-container">$R = \sum_{t=t'}^{T} r_{t+1}$</span>, then you get the &quot;vpg with baseline&quot; as it is called in the spinning up documentation. Note that in this case you have to rollout episode until it is finished, otherwise you can't compute the return !<br />
  If you use a one-step bootstrapped estimate, i.e. <span class="math-container">$ R(s,a) = r + V_\phi(s')$</span>, then you would get the actor-critic setup, where we have an actor (the policy network) that selects actions to perform the rollout and a critic (the value network) that is used to compute the returns, i.e. it grades the performance. And since you are baseline-ing with the value function you actually get an advantage actor-critic. Note that now you can calculate the return without the episode being finished. Thus, you could step the environment for a few steps, then update the policy, and then continue stepping.</p>
  <p>Its all about how you calculate the return.</p>
  <p>As a side note: I really don't like the fact that people use &quot;vanilla policy gradient&quot; to mean that they are using a Monte-Carlo estimate for the return. In my opinion &quot;vanilla&quot; policy gradient means that you perform a &quot;vanilla&quot; update of the policy using the calculated gradient, i.e.:
  <span class="math-container">$$ \theta_{new} = \theta - \alpha \nabla J(\theta).$$</span>
  Instead of a &quot;vanilla&quot; update you could update the weights using the natural gradient (TRPO) or you could perform multiple clipped updates (PPO) (see <a href="https://pi-tau.github.io/posts/actor-critic/#ppo" rel="nofollow noreferrer">here</a> for more). There are of course other types of policy gradient algorithms, but the idea is that once you have the gradient estimate you do not perform a simple update in the direction of the gradient, but instead do something more sophisticated with it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-26T22:32:43.793Z
Answers.LastActivityDate: 2023-08-26T22:32:43.793Z
Answers.UserDisplayName: pi-tau
Answers.Body: <p>The difference between Vanilla Policy Gradient (VPG) with a baseline as value function and Advantage Actor-Critic (A2C) is very similar to the difference between Monte Carlo Control and SARSA:</p>
  <ul>
  <li><p>The value estimates used in updates for VPG are based on full sampled returns, calculated at the end of episodes.</p>
  </li>
  <li><p>The value estimates used in updates for A2C are based on temporal difference <em>bootstrapped</em> from e.g. a single step difference, and the Bellman function.</p>
  </li>
  </ul>
  <p>This leads to the following practical differences:</p>
  <ul>
  <li><p>A2C can learn <em>during</em> an episode which can lead to faster refinements in policy than with VPG.</p>
  </li>
  <li><p>A2C can learn in continuing environments, whilst VPG cannot.</p>
  </li>
  <li><p>A2C relies on initially biased value estimates, so can take more tuning to find hyperparameters for the agent that allows for stable learning. Whilst VPG typically has higher variance and can require more samples to achieve the same degree of learning.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-05T17:27:40.487Z
Answers.Text: Thanks for your answer! What does "continuing environments" mean?
Answers.UserDisplayName: Iyeeke
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-05T17:37:39.693Z
Answers.Text: @lyeeke: It's another way of saying "non-episodic" i.e. the environment has no terminal states and keeps going with no natural break points.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-08T07:17:31.873Z
Answers.Text: It there anything inbetween? Like having this bootstrapped critic evaluation after every step but being able to adjust in the end of the episode if/when it turns out the bootstrapped value was wrong?
Answers.UserDisplayName: hypers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-08T08:38:43.623Z
Answers.Text: @hypers: You could use eligibility traces with the critic, it can be trained in almost any value-based way. Unfortunately you cannot easily go back and adjust the advice it gave to the actor in previous steps.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-08T10:57:45.677Z
Answers.Text: I see from this TF Actor Critic article (https://www.tensorflow.org/tutorials/reinforcement_learning/actor_critic) that A2C can also be trained in the end of the episode (as done in that tutorial).
  So my reckoning is that A2C is not necessarily different from VPG by this criteria (when the training is happening).
Answers.UserDisplayName: hypers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-04T13:56:54.757Z
Answers.Text: If I understand correctly, Neil's answer is correct because VPG uses rollouts/trajectories in place of the traditional episodes. A rollout has a max length, so that it can either end when max length is reached, or when a terminal state is reached. 
  
  VPG learns at the end of each rollout, which can also be during an episode. correct me if im wrong!
Answers.UserDisplayName: Jakob G
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-27T10:07:14.377Z
Answers.LastActivityDate: 2022-01-12T21:06:39.597Z
Answers.LastEditDate: 2022-01-12T21:06:39.597Z
Answers.UserDisplayName: Neil Slater
Body: <p>What is the difference between vanilla policy gradient (VPG) with a baseline as value function and advantage actor-critic (A2C)?</p>
  <p>By vanilla policy gradient I am specifically referring to spinning up's explanation of VPG.</p>
  
Tags: <reinforcement-learning><comparison><policy-gradients><actor-critic-methods><advantage-actor-critic>
Title: What is the difference between vanilla policy gradient with a baseline as value function and advantage actor-critic?
UserDisplayName: Vedant Shah
ThreadText: # What is the difference between vanilla policy gradient with a baseline as value function and advantage actor-critic?
  
  (Asked by: username_0 on 2020-07-27)
  
  What is the difference between vanilla policy gradient (VPG) with a baseline as value function and advantage actor-critic (A2C)?
  
  By vanilla policy gradient I am specifically referring to spinning up's explanation of VPG.
  
  **Tags:** reinforcement-learning, comparison, policy-gradients, actor-critic-methods, advantage-actor-critic
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-07-27. Score: 7 | Accepted answer
  
  The difference between Vanilla Policy Gradient (VPG) with a baseline as value function and Advantage Actor-Critic (A2C) is very similar to the difference between Monte Carlo Control and SARSA:
  
  * The value estimates used in updates for VPG are based on full sampled returns, calculated at the end of episodes.
  * The value estimates used in updates for A2C are based on temporal difference *bootstrapped* from e.g. a single step difference, and the Bellman function.
  
  This leads to the following practical differences:
  
  * A2C can learn *during* an episode which can lead to faster refinements in policy than with VPG.
  * A2C can learn in continuing environments, whilst VPG cannot.
  * A2C relies on initially biased value estimates, so can take more tuning to find hyperparameters for the agent that allows for stable learning. Whilst VPG typically has higher variance and can require more samples to achieve the same degree of learning.
  
  > **Comments:**
  > **username_3**: Thanks for your answer! What does "continuing environments" mean?  
  > **username_2**: @lyeeke: It's another way of saying "non-episodic" i.e. the environment has no terminal states and keeps going with no natural break points.  
  > **username_4**: It there anything inbetween? Like having this bootstrapped critic evaluation after every step but being able to adjust in the end of the episode if/when it turns out the bootstrapped value was wrong?  
  > **username_2**: @username_4: You could use eligibility traces with the critic, it can be trained in almost any value-based way. Unfortunately you cannot easily go back and adjust the advice it gave to the actor in previous steps.  
  > **username_4**: I see from this TF Actor Critic article (https://www.tensorflow.org/tutorials/reinforcement_learning/actor_critic) that A2C can also be trained in the end of the episode (as done in that tutorial). So my reckoning is that A2C is not necessarily different from VPG by this criteria (when the training is happening).  
  > **username_5**: If I understand correctly, username_2's answer is correct because VPG uses rollouts/trajectories in place of the traditional episodes. A rollout has a max length, so that it can either end when max length is reached, or when a terminal state is reached.   VPG learns at the end of each rollout, which can also be during an episode. correct me if im wrong!  
  
  ---
  
  ### Answer by username_1 on 2023-08-26. Score: 0
  
  Given the formula for the policy gradient with baseline:
  
  $$ \nabla J(\theta) = \mathbb{E}\_{a,s \sim \pi\_\theta} \bigg[ \nabla \log \pi\_\theta(a|s) \Big(R(s, a) - V\_\phi(s) \Big) \bigg] $$
  
  How do you compute the return $R(s,a)$ ?  
  If you use a simple Monte-Carlo estimate, i.e. $R = \sum\_{t=t'}^{T} r\_{t+1}$, then you get the "vpg with baseline" as it is called in the spinning up documentation. Note that in this case you have to rollout episode until it is finished, otherwise you can't compute the return !  
  If you use a one-step bootstrapped estimate, i.e. $ R(s,a) = r + V\_\phi(s')$, then you would get the actor-critic setup, where we have an actor (the policy network) that selects actions to perform the rollout and a critic (the value network) that is used to compute the returns, i.e. it grades the performance. And since you are baseline-ing with the value function you actually get an advantage actor-critic. Note that now you can calculate the return without the episode being finished. Thus, you could step the environment for a few steps, then update the policy, and then continue stepping.
  
  Its all about how you calculate the return.
  
  As a side note: I really don't like the fact that people use "vanilla policy gradient" to mean that they are using a Monte-Carlo estimate for the return. In my opinion "vanilla" policy gradient means that you perform a "vanilla" update of the policy using the calculated gradient, i.e.:
  $$ \theta\_{new} = \theta - \alpha \nabla J(\theta).$$
  Instead of a "vanilla" update you could update the weights using the natural gradient (TRPO) or you could perform multiple clipped updates (PPO) (see [here](https://username_1.github.io/posts/actor-critic/#ppo) for more). There are of course other types of policy gradient algorithms, but the idea is that once you have the gradient estimate you do not perform a simple update in the direction of the gradient, but instead do something more sophisticated with it.
  
  
  ---
  
  