Answers.Body: <p>According to Sutton and Barto, they are the same thing. Note 13.5-6 (page 338) of their <em>Reinforcement Learning: An Introduction, 2nd Edition</em> book:</p>
  
  <blockquote>
    <p>Actor-critic methods are sometimes referred to as advantage actor-critic methods in the literature</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-04T12:23:58.057Z
Answers.LastActivityDate: 2020-05-14T10:22:43.137Z
Answers.LastEditDate: 2020-05-14T10:22:43.137Z
Answers.UserDisplayName: Budi Kurniawan
Answers.Body: <p>Though the word "Advantage" in the actor-critic realm has been used to refer to the difference between the state value and the state action value, A2C brings in the ideas of A3C. In A3C, several worker networks interact with different copies of the environment (asynchronous learning) and update a master network after a set if steps. This was meant to solve instability issues associated with both temporal difference update method and correlations within neural network generated prediction and target values. However it was noticed by OpenAI that there was no need of the asynchrony, i.e. there was no practical benefit of having different worker networks. Instead, they had the same copy of the network that interacted with different copies of the environment (one works from the beginning, another working backwards from the end) and they update at once without the master lagging behind like in A3C. The removal of asynchrony gave rise to A2C.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-14T09:11:50.747Z
Answers.LastActivityDate: 2020-05-14T09:11:50.747Z
Answers.UserDisplayName: EArwa
Answers.Body: <p>So, this is the formula for updating the weights <span class="math-container">$\theta$</span> of your policy network using the policy gradient theorem:
  <span class="math-container">$$ \nabla_\theta J(\theta) = E_{a, s \sim \pi_\theta} \Big[ \nabla \log \pi_\theta(a|s) R(s, a) \Big]. $$</span>
  Obviously your policy network is the actor. The question is how do you evaluate <span class="math-container">$R$</span>.<br />
  You could do a simple Monte-Carlo estimate:
  <span class="math-container">$$ R(s_t, a_t) = \sum_{i=t}^{T} r_{i+1} $$</span>
  This is <strong>not</strong> an actor-critic algorithm.<br />
  Or, you could do for example a one-step bootstrap using a value network with weights <span class="math-container">$\phi$</span>:
  <span class="math-container">$$ R(s_t, a_t) = r_{t+1} + V_\phi(s_{t+1}) $$</span>
  This is an actor-critic algorithm. In this case the value network is the critic. It essentially assigns a score on each of the actions taken by the actor.<br />
  Or, you could learn a q-value network with weights <span class="math-container">$\psi$</span> and use that:
  <span class="math-container">$$ R(s_t, a_t) = Q_\psi(s_t, a_t) $$</span>
  This is again an actor-critic algorithm and your q-value network is the critic. Same reasoning as above.<br />
  So, actor-critic algorithms are a set of algorithms where you have an actor (the policy network) that selects actions for the rollout and a critic (a second model), that is used to compute the returns.<br />
  But instead of using the return, you could estimate the policy gradient using the advantage:
  <span class="math-container">$$ \nabla_\theta J(\theta) = E_{a, s \sim \pi_\theta} \Big[ \nabla \log \pi_\theta(a|s) A(s, a) \Big], $$</span>
  where <span class="math-container">$A(s,a) = R(s,a) - V_\phi(s)$</span>. Usually you will have a value network for estimating the value of the state.<br />
  Now, for the Monte-Carlo estimate <span class="math-container">$A(s_t, a_t) = \sum_{i=t}^{T} r_{i+1} - V_\phi(s_t)$</span> you get what is known as &quot;policy gradient with baseline&quot; - not an actor-critic algorithm. In the other two cases:</p>
  <ul>
  <li>one-step bootstrap <span class="math-container">$A(s_t, a_t) = r_{t+1} + V_\phi(s_{t+1}) - V_\phi(s_{t})$</span>,</li>
  <li>q-value network <span class="math-container">$A(s_t, a_t) = Q_\psi(s_t, a_t) - V_\phi(s_t)$</span>,</li>
  </ul>
  <p>you get an Advantage Actor-Critic algorithm.<br />
  You could also use a neural network that models the advantage directly:
  <span class="math-container">$ A(s_t, a_t) = A_\zeta(s_t, a_t)$</span>. Obviously, you also get an advantage actor-critic algorithm.<br />
  So, advantage actor-critic algorithms are a set of algorithms where again you have an actor and a critic, but now you estimate the gradient using the advantage instead of the return.</p>
  <p>I recommend reading this <a href="https://pi-tau.github.io/posts/actor-critic/" rel="nofollow noreferrer">blog post</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-25T22:55:15.940Z
Answers.LastActivityDate: 2023-08-25T22:55:15.940Z
Answers.UserDisplayName: pi-tau
Answers.Body: <p>Actor-Critic is not just a single algorithm, it should be viewed as a &quot;family&quot; of related techniques. They're all techniques based on the policy gradient theorem, which train some form of critic that computes some form of value estimate to plug into the update rule as a lower-variance replacement for the returns at the end of an episode. They all perform &quot;bootstrapping&quot; by using some sort of prediction of value.</p>
  <p><strong>Advantage Actor-Critic</strong> specifically uses estimates of the advantage function <span class="math-container">$A(s, a) = Q(s, a) - V(s)$</span> for its bootstrapping, whereas &quot;actor-critic&quot; without the &quot;advantage&quot; qualifier is not specific; it could be a trained <span class="math-container">$V(s)$</span> function, it could be some sort of estimate of <span class="math-container">$Q(s, a)$</span>, it could be a variety of things.</p>
  <p>In practice, the critic of Advantage Actor-Critic methods actually can just be trained to predict <span class="math-container">$V(s)$</span>. Combined with an empirically observed reward <span class="math-container">$r$</span>, they can then compute the advantage estimate <span class="math-container">$A(s, a) = r + \gamma V(s') - V(s)$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-18T10:06:54.913Z
Answers.Text: Nice clarification. Do you know an example of an actor-critic method that is not advantage actor-critic? Also, would it be fair to say that PPO is a type of advantage actor-critic method?
Answers.UserDisplayName: Mei Zhang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-18T10:35:53.740Z
Answers.Text: @MeiZhang Some options are listed [here](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#other-forms-of-the-policy-gradient), but I don't think they have clear names really. Advantages are certainly the most commonly-used. Sure, PPO could be viewed as an advantage actor-critic method.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-12T22:17:31.500Z
Answers.Text: Can advantage actor critic use the loss gradient $$-\sum_t \left( Q_\phi(s_t, a_t) - \sum_a \pi_\theta(s_t, a) Q_\phi(s_t, a) \right) \nabla_\theta \log \pi_\theta(s_t, a_t) + \nabla_\phi \frac{1}{T} \sum_t (R_t - Q_\phi(s_t, a_t))^2$$ (where $R_t$ are the discounted returns), as described in the [Mean Actor Critic paper](https://arxiv.org/pdf/1709.00503.pdf#page=3)? It's not doing well in my implementation, for some reason. I can post a separate question if you'd like.
Answers.UserDisplayName: user76284
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-13T09:30:14.330Z
Answers.Text: @user76284 I didn't carefully inspect the equation you wrote there, but I'm familiar with the paper and yes, I see no reason why that wouldn't work. I've used something with similar intuition (summing up over all the actions, multiplying predicted values with policy's probabilities) in https://arxiv.org/abs/1905.05809. That was a very different setting though, games instead of single-agent RL, with value estimates produced by MCTS instead of a network. That seemed to work fine though.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-14T20:02:40.240Z
Answers.Text: @DennisSoemers I posted a question about this [here](https://ai.stackexchange.com/questions/24077/problems-with-gradient-biased-actor-critic-methods).
Answers.UserDisplayName: user76284
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-25T22:16:42.803Z
Answers.Text: @MeiZhang Please refer to this blog post: https://pi-tau.github.io/posts/actor-critic/
Answers.UserDisplayName: pi-tau
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-04T17:24:43.520Z
Answers.LastActivityDate: 2022-07-07T09:49:07.537Z
Answers.LastEditDate: 2022-07-07T09:49:07.537Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I'm struggling to understand the difference between actor-critic and advantage actor-critic.</p>
  
  <p>At least, I know they are different from asynchronous advantage actor-critic (A3C), as A3C adds an asynchronous mechanism that uses multiple worker agents interacting with their own copy of the environment and reports the gradient to the global agent. </p>
  
  <p>But what is the difference between the actor-critic and advantage actor-critic (A2C)? Is it simply with or without <em>advantage</em> function? But, then, does the actor-critic have any other implementation except for the use of advantage function?</p>
  
  <p>Or maybe are they synonyms and actor-critic is just a shorthand for A2C?</p>
  
Tags: <reinforcement-learning><terminology><actor-critic-methods><comparison><advantage-actor-critic>
Title: What is the difference between actor-critic and advantage actor-critic?
UserDisplayName: Blaszard
ThreadText: # What is the difference between actor-critic and advantage actor-critic?
  
  (Asked by: username_0 on 2018-08-02)
  
  I'm struggling to understand the difference between actor-critic and advantage actor-critic.
  
  At least, I know they are different from asynchronous advantage actor-critic (A3C), as A3C adds an asynchronous mechanism that uses multiple worker agents interacting with their own copy of the environment and reports the gradient to the global agent.
  
  But what is the difference between the actor-critic and advantage actor-critic (A2C)? Is it simply with or without *advantage* function? But, then, does the actor-critic have any other implementation except for the use of advantage function?
  
  Or maybe are they synonyms and actor-critic is just a shorthand for A2C?
  
  **Tags:** reinforcement-learning, terminology, actor-critic-methods, comparison, advantage-actor-critic
  
  ---
  
  ## 4 answers
  
  ### Answer by username_4 on 2018-08-04. Score: 18 | Accepted answer
  
  Actor-Critic is not just a single algorithm, it should be viewed as a "family" of related techniques. They're all techniques based on the policy gradient theorem, which train some form of critic that computes some form of value estimate to plug into the update rule as a lower-variance replacement for the returns at the end of an episode. They all perform "bootstrapping" by using some sort of prediction of value.
  
  **Advantage Actor-Critic** specifically uses estimates of the advantage function $A(s, a) = Q(s, a) - V(s)$ for its bootstrapping, whereas "actor-critic" without the "advantage" qualifier is not specific; it could be a trained $V(s)$ function, it could be some sort of estimate of $Q(s, a)$, it could be a variety of things.
  
  In practice, the critic of Advantage Actor-Critic methods actually can just be trained to predict $V(s)$. Combined with an empirically observed reward $r$, they can then compute the advantage estimate $A(s, a) = r + \gamma V(s') - V(s)$.
  
  > **Comments:**
  > **username_5**: Nice clarification. Do you know an example of an actor-critic method that is not advantage actor-critic? Also, would it be fair to say that PPO is a type of advantage actor-critic method?  
  > **username_4**: @MeiZhang Some options are listed [here](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#other-forms-of-the-policy-gradient), but I don't think they have clear names really. Advantages are certainly the most commonly-used. Sure, PPO could be viewed as an advantage actor-critic method.  
  > **username_6**: Can advantage actor critic use the loss gradient $$-\sum_t \left( Q_\phi(s_t, a_t) - \sum_a \pi_\theta(s_t, a) Q_\phi(s_t, a) \right) \nabla_\theta \log \pi_\theta(s_t, a_t) + \nabla_\phi \frac{1}{T} \sum_t (R_t - Q_\phi(s_t, a_t))^2$$ (where $R_t$ are the discounted returns), as described in the [Mean Actor Critic paper](https://arxiv.org/pdf/1709.00503.pdf#page=3)? It's not doing well in my implementation, for some reason. I can post a separate question if you'd like.  
  > **username_4**: @user76284 I didn't carefully inspect the equation you wrote there, but I'm familiar with the paper and yes, I see no reason why that wouldn't work. I've used something with similar intuition (summing up over all the actions, multiplying predicted values with policy's probabilities) in https://arxiv.org/abs/1905.05809. That was a very different setting though, games instead of single-agent RL, with value estimates produced by MCTS instead of a network. That seemed to work fine though.  
  > **username_6**: @DennisSoemers I posted a question about this [here](https://ai.stackexchange.com/questions/24077/problems-with-gradient-biased-actor-critic-methods).  
  > **username_3**: @MeiZhang Please refer to this blog post: https://pi-tau.github.io/posts/actor-critic/  
  
  ---
  
  ### Answer by username_1 on 2019-09-04. Score: 2
  
  According to Sutton and Barto, they are the same thing. Note 13.5-6 (page 338) of their *Reinforcement Learning: An Introduction, 2nd Edition* book:
  
  > Actor-critic methods are sometimes referred to as advantage actor-critic methods in the literature
  
  
  ---
  
  ### Answer by username_2 on 2020-05-14. Score: 0
  
  Though the word "Advantage" in the actor-critic realm has been used to refer to the difference between the state value and the state action value, A2C brings in the ideas of A3C. In A3C, several worker networks interact with different copies of the environment (asynchronous learning) and update a master network after a set if steps. This was meant to solve instability issues associated with both temporal difference update method and correlations within neural network generated prediction and target values. However it was noticed by OpenAI that there was no need of the asynchrony, i.e. there was no practical benefit of having different worker networks. Instead, they had the same copy of the network that interacted with different copies of the environment (one works from the beginning, another working backwards from the end) and they update at once without the master lagging behind like in A3C. The removal of asynchrony gave rise to A2C.
  
  
  ---
  
  ### Answer by username_3 on 2023-08-25. Score: 0
  
  So, this is the formula for updating the weights $\theta$ of your policy network using the policy gradient theorem:
  $$ \nabla\_\theta J(\theta) = E\_{a, s \sim \pi\_\theta} \Big[ \nabla \log \pi\_\theta(a|s) R(s, a) \Big]. $$
  Obviously your policy network is the actor. The question is how do you evaluate $R$.  
  You could do a simple Monte-Carlo estimate:
  $$ R(s\_t, a\_t) = \sum\_{i=t}^{T} r\_{i+1} $$
  This is **not** an actor-critic algorithm.  
  Or, you could do for example a one-step bootstrap using a value network with weights $\phi$:
  $$ R(s\_t, a\_t) = r\_{t+1} + V\_\phi(s\_{t+1}) $$
  This is an actor-critic algorithm. In this case the value network is the critic. It essentially assigns a score on each of the actions taken by the actor.  
  Or, you could learn a q-value network with weights $\psi$ and use that:
  $$ R(s\_t, a\_t) = Q\_\psi(s\_t, a\_t) $$
  This is again an actor-critic algorithm and your q-value network is the critic. Same reasoning as above.  
  So, actor-critic algorithms are a set of algorithms where you have an actor (the policy network) that selects actions for the rollout and a critic (a second model), that is used to compute the returns.  
  But instead of using the return, you could estimate the policy gradient using the advantage:
  $$ \nabla\_\theta J(\theta) = E\_{a, s \sim \pi\_\theta} \Big[ \nabla \log \pi\_\theta(a|s) A(s, a) \Big], $$
  where $A(s,a) = R(s,a) - V\_\phi(s)$. Usually you will have a value network for estimating the value of the state.  
  Now, for the Monte-Carlo estimate $A(s\_t, a\_t) = \sum\_{i=t}^{T} r\_{i+1} - V\_\phi(s\_t)$ you get what is known as "policy gradient with baseline" - not an actor-critic algorithm. In the other two cases:
  
  * one-step bootstrap $A(s\_t, a\_t) = r\_{t+1} + V\_\phi(s\_{t+1}) - V\_\phi(s\_{t})$,
  * q-value network $A(s\_t, a\_t) = Q\_\psi(s\_t, a\_t) - V\_\phi(s\_t)$,
  
  you get an Advantage Actor-Critic algorithm.  
  You could also use a neural network that models the advantage directly:
  $ A(s\_t, a\_t) = A\_\zeta(s\_t, a\_t)$. Obviously, you also get an advantage actor-critic algorithm.  
  So, advantage actor-critic algorithms are a set of algorithms where again you have an actor and a critic, but now you estimate the gradient using the advantage instead of the return.
  
  I recommend reading this [blog post](https://pi-tau.github.io/posts/actor-critic/).
  
  
  ---
  
  