Answers.Body: <p>Let's look at how they derived the equation for that. Recall that <span class="math-container">$A(s,a) = Q(s,a) - V(s)$</span>, so in the first equation, it is <span class="math-container">$Q(s,a) = \sum_t \gamma^tr_t$</span>. We should noted that, the <em>original</em> Policy Gradient uses <span class="math-container">$A(s,a) = \sum_t \gamma^tr_t - b$</span>, where <span class="math-container">$b$</span> is any baseline.</p>
  <p>The first equation <span class="math-container">$A(s,a) = \sum_t \gamma^t r_t - b$</span>, the original Policy Gradient, is <em>unbiased</em> because it is a single-sample estimate, but it has <em>high variance</em> because it sums all rewards, thus requires a good learning rate.</p>
  <p>The second equation <span class="math-container">$A(s,a) = r + \gamma V(s') - V(s)$</span> has the advantage that it dramatically lowers the <em>variance</em> since it relies on all possibilities that can occur (Policy Gradient has the term &quot;sum of the Advantage since step 0&quot;). However, it is not <em>unbiased</em> because if the approximation of <span class="math-container">$V$</span> is incorrect, the policy gradient will fail to converge. Now this happens regularly with Neural Networks (NN) in Deep RL, because random initialization of NN already makes incorrect approximation of <span class="math-container">$V$</span>.</p>
  <p>In the end, what we want is an <em>unbiased</em> Advantage estimator, but needs to have <em>low variance</em> as well. We then use a critic as a the state-dependent baselines <span class="math-container">$V(s)$</span> for the Advantage calculation, that is the equation <span class="math-container">$A(s,a) =  \sum_t \gamma^t r_t - V(s)$</span> you see in A2C.</p>
  <p>You can read more details in the <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#baselines-in-policy-gradients" rel="nofollow noreferrer">Spinning Up docs</a>, <a href="http://rail.eecs.berkeley.edu/deeprlcourse-fa21/static/slides/lec-6.pdf" rel="nofollow noreferrer">RAIL Lecture 6</a>, and <a href="http://rail.eecs.berkeley.edu/deeprlcourse-fa21/static/homeworks/hw2.pdf" rel="nofollow noreferrer">RAIL homework 2</a> if you want to know why they come up with such equation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-29T07:13:39.043Z
Answers.Text: Another way to balance the bias-variance trade-off is to use the Generalized Advantage Estimation (https://arxiv.org/abs/1506.02438)
Answers.UserDisplayName: entropy07
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-28T17:20:23.187Z
Answers.LastActivityDate: 2023-01-28T17:20:23.187Z
Answers.UserDisplayName: Minh-Long Luu
Body: <p>Why does A2C use the actual returns from an episode in calculating the advantage instead of using a bellman equation style estimate of the value?</p>
  <p>Basically, why this:</p>
  <p><span class="math-container">$A(s,a) = \sum_t\gamma^tr_t - V(s)$</span></p>
  <p>instead of</p>
  <p><span class="math-container">$A(s,a) = r + \gamma V(s') - V(s)$</span></p>
  <p>This blog post seems to describe it one way, but their code is the other way:
  <a href="https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f" rel="nofollow noreferrer">https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f</a></p>
  <p>I feel there's some big gap in my understanding, but would appreciate some insight.</p>
  
Tags: <reinforcement-learning><actor-critic-methods><bellman-equations>
Title: Why does A2C use the actual returns from an episode in calculating the advantage?
UserDisplayName: JacKeown
ThreadText: # Why does A2C use the actual returns from an episode in calculating the advantage?
  
  (Asked by: username_0 on 2023-01-25)
  
  Why does A2C use the actual returns from an episode in calculating the advantage instead of using a bellman equation style estimate of the value?
  
  Basically, why this:
  
  $A(s,a) = \sum\_t\gamma^tr\_t - V(s)$
  
  instead of
  
  $A(s,a) = r + \gamma V(s') - V(s)$
  
  This blog post seems to describe it one way, but their code is the other way:
  <https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f>
  
  I feel there's some big gap in my understanding, but would appreciate some insight.
  
  **Tags:** reinforcement-learning, actor-critic-methods, bellman-equations
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-28. Score: 2 | Accepted answer
  
  Let's look at how they derived the equation for that. Recall that $A(s,a) = Q(s,a) - V(s)$, so in the first equation, it is $Q(s,a) = \sum\_t \gamma^tr\_t$. We should noted that, the *original* Policy Gradient uses $A(s,a) = \sum\_t \gamma^tr\_t - b$, where $b$ is any baseline.
  
  The first equation $A(s,a) = \sum\_t \gamma^t r\_t - b$, the original Policy Gradient, is *unbiased* because it is a single-sample estimate, but it has *high variance* because it sums all rewards, thus requires a good learning rate.
  
  The second equation $A(s,a) = r + \gamma V(s') - V(s)$ has the advantage that it dramatically lowers the *variance* since it relies on all possibilities that can occur (Policy Gradient has the term "sum of the Advantage since step 0"). However, it is not *unbiased* because if the approximation of $V$ is incorrect, the policy gradient will fail to converge. Now this happens regularly with Neural Networks (NN) in Deep RL, because random initialization of NN already makes incorrect approximation of $V$.
  
  In the end, what we want is an *unbiased* Advantage estimator, but needs to have *low variance* as well. We then use a critic as a the state-dependent baselines $V(s)$ for the Advantage calculation, that is the equation $A(s,a) = \sum\_t \gamma^t r\_t - V(s)$ you see in A2C.
  
  You can read more details in the [Spinning Up docs](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#baselines-in-policy-gradients), [RAIL Lecture 6](http://rail.eecs.berkeley.edu/deeprlcourse-fa21/static/slides/lec-6.pdf), and [RAIL homework 2](http://rail.eecs.berkeley.edu/deeprlcourse-fa21/static/homeworks/hw2.pdf) if you want to know why they come up with such equation.
  
  > **Comments:**
  > **username_2**: Another way to balance the bias-variance trade-off is to use the Generalized Advantage Estimation (https://arxiv.org/abs/1506.02438)  
  
  ---
  
  