Answers.Body: <p>In policy gradient methods reducing bias is crucial to obtain more accurate gradient estimates for updating the policy in a stochastic gradient ascent fashion. The usual advantage function as you expressed only implicitly addresses credit assignment for the current state and the specific on-policy's action at time step <span class="math-container">$t$</span>, while the generalized advantage estimation (GAE) expressed by equation 16 explicitly addresses credit assignment by capturing the temporal consistency of advantages and considering the cumulative effects of the on-policy's actions from the initial state to the current state and beyond until the end of its episode during each parameters update time step <span class="math-container">$t$</span>. Also from equation 16 you see GAE has a flexible bias-variance trade-off parameter <span class="math-container">$λ$</span> to further fine tune for each problem, and even if it's biased when it's not <span class="math-container">$γ$</span>-just, the bias must be very small as reflected in equation 14.</p>
  <p>Therefore it's clear that the advantage estimator <span class="math-container">$\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$</span> referenced in equation 7 is a function of the entire trajectory of an episode in general (the 2 semicolons mean the independent variables could be all states and their on-policy actions of any episode), and if it's of the form expressed in Proposition 1 then it's <span class="math-container">$γ$</span>-just. The usual discounted advantage function is a special <span class="math-container">$γ$</span>-just case of the advantage estimator <span class="math-container">$\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$</span> as mentioned immediately after Proposition 1, and the GAE expressed in equation 16 is the generalized case of <span class="math-container">$\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$</span> considering the on-policy's advantage over the entire episode to become completely unbiased when <span class="math-container">$λ=1$</span> in theory.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-18T08:13:06.453Z
Answers.Text: I think you mean that in policy gradient methods we are interested in reducing variance, not bias? Since with this method explained in the paper, we are introducing bias, we are not reducing it.
Answers.UserDisplayName: kklaw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-19T00:21:59.423Z
Answers.Text: See equation 18 and the explanation after it: *GAE(γ, 1) is γ-just regardless of the accuracy of V, but it has high variance due to the sum of terms*. So when λ=1 GAE is unbiased estimator for *γ-discounted* advantage function. But as the footnote on page 3 notes that here we are concerned with obtaining an unbiased estimate of $g^γ$, which is a biased estimate of the policy gradient of the undiscounted MDP. PG method has to take discounted and undiscounted MDP separately as one of its complications compared to value based methods.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-19T00:26:20.443Z
Answers.Text: Anyway the flexible bias-variance trade-off capability of GAE is its feature compared to the usual advantage function which typically has low variance but high bias which is also mentioned in the same paper and this has little to do with your question about $\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$, and equation 16 is the constructive way to estimate the generalized advantage at time step $t$ using all actions and states starting from the starting state as you requested.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-16T06:40:49.293Z
Answers.Text: @cinch I have a question. You say "cumulative effects of the on-policy's actions from the initial state to the current state", but the formula seems to look from the time step forward (running variable l) not from the beginning somehow (t=0). Could you clarify where my misunderstanding is?
Answers.UserDisplayName: craaaft
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-18T07:16:33.447Z
Answers.LastActivityDate: 2024-02-18T07:16:33.447Z
Answers.UserDisplayName: cinch
Body: <p>In general, the advantage function is defined as:</p>
  <p><span class="math-container">$A^\pi\left(s_t, a_t\right):=Q^\pi\left(s_t, a_t\right)-V^\pi\left(s_t\right)$</span></p>
  <p>So far, I understand this formular like this:</p>
  <p>With the advantage function, we can compare, how good (positive sign, i.e. <span class="math-container">$A^\pi &gt; 0$</span>) or bad (negative sign , i.e. <span class="math-container">$A^\pi &lt; 0$</span>) a specific action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span> is compared against the average of all actions at time step <span class="math-container">$t$</span>.</p>
  <p>However, in the paper High-Dimensional Continuous Control Using Generalized Advantage Estimation (<a href="https://arxiv.org/pdf/1506.02438.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1506.02438.pdf</a>), they introduce the following notation:</p>
  <p><span class="math-container">$E_{s_{0: \infty} \\ a_{0: \infty}}\left[\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right) \nabla_\theta \log \pi_\theta(a_t \mid s_t)\right]$</span> (see equation <span class="math-container">$7$</span>)</p>
  <p>I am a little bit confused by the term</p>
  <p><span class="math-container">$\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$</span></p>
  <p>What does this expression mean?
  How can we estimate the advantage at time step <span class="math-container">$t$</span>, using all actions and states starting from the starting state?</p>
  
Tags: <reinforcement-learning><papers><notation><advantage-actor-critic>
Title: What does the notation $\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$ appearing in Generalized Advantage Estimation mean?
UserDisplayName: kklaw
ThreadText: # What does the notation $\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$ appearing in Generalized Advantage Estimation mean?
  
  (Asked by: username_0 on 2024-02-17)
  
  In general, the advantage function is defined as:
  
  $A^\pi\left(s\_t, a\_t\right):=Q^\pi\left(s\_t, a\_t\right)-V^\pi\left(s\_t\right)$
  
  So far, I understand this formular like this:
  
  With the advantage function, we can compare, how good (positive sign, i.e. $A^\pi > 0$) or bad (negative sign , i.e. $A^\pi < 0$) a specific action $a$ in state $s$ is compared against the average of all actions at time step $t$.
  
  However, in the paper High-Dimensional Continuous Control Using Generalized Advantage Estimation (<https://arxiv.org/pdf/1506.02438.pdf>), they introduce the following notation:
  
  $E\_{s\_{0: \infty} \\ a\_{0: \infty}}\left[\hat{A}\_t\left(s\_{0: \infty}, a\_{0: \infty}\right) \nabla\_\theta \log \pi\_\theta(a\_t \mid s\_t)\right]$ (see equation $7$)
  
  I am a little bit confused by the term
  
  $\hat{A}\_t\left(s\_{0: \infty}, a\_{0: \infty}\right)$
  
  What does this expression mean?
  How can we estimate the advantage at time step $t$, using all actions and states starting from the starting state?
  
  **Tags:** reinforcement-learning, papers, notation, advantage-actor-critic
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-18. Score: 1
  
  In policy gradient methods reducing bias is crucial to obtain more accurate gradient estimates for updating the policy in a stochastic gradient ascent fashion. The usual advantage function as you expressed only implicitly addresses credit assignment for the current state and the specific on-policy's action at time step $t$, while the generalized advantage estimation (GAE) expressed by equation 16 explicitly addresses credit assignment by capturing the temporal consistency of advantages and considering the cumulative effects of the on-policy's actions from the initial state to the current state and beyond until the end of its episode during each parameters update time step $t$. Also from equation 16 you see GAE has a flexible bias-variance trade-off parameter $λ$ to further fine tune for each problem, and even if it's biased when it's not $γ$-just, the bias must be very small as reflected in equation 14.
  
  Therefore it's clear that the advantage estimator $\hat{A}\_t\left(s\_{0: \infty}, a\_{0: \infty}\right)$ referenced in equation 7 is a function of the entire trajectory of an episode in general (the 2 semicolons mean the independent variables could be all states and their on-policy actions of any episode), and if it's of the form expressed in Proposition 1 then it's $γ$-just. The usual discounted advantage function is a special $γ$-just case of the advantage estimator $\hat{A}\_t\left(s\_{0: \infty}, a\_{0: \infty}\right)$ as mentioned immediately after Proposition 1, and the GAE expressed in equation 16 is the generalized case of $\hat{A}\_t\left(s\_{0: \infty}, a\_{0: \infty}\right)$ considering the on-policy's advantage over the entire episode to become completely unbiased when $λ=1$ in theory.
  
  > **Comments:**
  > **username_0**: I think you mean that in policy gradient methods we are interested in reducing variance, not bias? Since with this method explained in the paper, we are introducing bias, we are not reducing it.  
  > **username_1**: See equation 18 and the explanation after it: *GAE(γ, 1) is γ-just regardless of the accuracy of V, but it has high variance due to the sum of terms*. So when λ=1 GAE is unbiased estimator for *γ-discounted* advantage function. But as the footnote on page 3 notes that here we are concerned with obtaining an unbiased estimate of $g^γ$, which is a biased estimate of the policy gradient of the undiscounted MDP. PG method has to take discounted and undiscounted MDP separately as one of its complications compared to value based methods.  
  > **username_1**: Anyway the flexible bias-variance trade-off capability of GAE is its feature compared to the usual advantage function which typically has low variance but high bias which is also mentioned in the same paper and this has little to do with your question about $\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$, and equation 16 is the constructive way to estimate the generalized advantage at time step $t$ using all actions and states starting from the starting state as you requested.  
  > **username_2**: @username_1 I have a question. You say "cumulative effects of the on-policy's actions from the initial state to the current state", but the formula seems to look from the time step forward (running variable l) not from the beginning somehow (t=0). Could you clarify where my misunderstanding is?  
  
  ---
  
  