Answers.Body: <blockquote>
  <p>Bootstrapped estimate is biased because it based on <span class="math-container">$V(s_{t+1})$</span> which is usually a biased estimate by some estimator such as neural network.</p>
  </blockquote>
  <p>I don't think this statement is totally correct in the context of the paper. Quoting the paper:</p>
  <blockquote>
  <p>Taking <span class="math-container">$\gamma &lt; 1$</span> introduces bias into the policy gradient estimate, regardless of the value function’s accuracy. On the other hand, <span class="math-container">$ \lambda &lt; 1$</span> introduces bias only when the value function is inaccurate.</p>
  </blockquote>
  <p>So the initial <span class="math-container">$\gamma$</span>-discounting introduces bias regardless of the accuracy of the <span class="math-container">$V(s)$</span>.</p>
  <p>Same point is reiterated in the footnote on the page 3:</p>
  <blockquote>
  <p>Note, that we have already introduced bias by using <span class="math-container">$A^{\pi,\gamma}$</span> in place of <span class="math-container">$A^\pi$</span>; here we are concerned with obtaining an unbiased estimate of <span class="math-container">$g^\gamma$</span>, which is a biased estimate of the policy gradient of the undiscounted MDP.</p>
  </blockquote>
  <p>To summarize the above:</p>
  <ol>
  <li>the original MDP under consideration is undiscounted</li>
  <li>authors introduce <span class="math-container">$\gamma$</span>-discounting to reduce variance (at cost of bias) of the PG estimate - this happens even if we had accurate <span class="math-container">$V(s)$</span></li>
  <li>later, authors introduce another <span class="math-container">$\lambda$</span>-discounting which also controls bias-variance tradeoff - now due to inaccuracy in <span class="math-container">$V(s)$</span></li>
  </ol>
  <p>As for why lower <span class="math-container">$\gamma$</span> reduces variance (p.2 above) - your explanation is on point: smaller <span class="math-container">$\gamma$</span> <span class="math-container">$\Rightarrow$</span> smaller weights for distant rewards <span class="math-container">$\Rightarrow$</span> less variance (but more bias).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-15T23:48:22.287Z
Answers.LastActivityDate: 2021-05-15T23:48:22.287Z
Answers.UserDisplayName: Kostya
Body: <p>I was reading the paper on <a href="https://arxiv.org/abs/1506.02438" rel="nofollow noreferrer">Generalized Advantage Estimate</a>. It first introduces a generalized form of policy gradient equation without involving <span class="math-container">$\gamma$</span> and then it says the following:</p>
  <blockquote>
  <p>We will introduce a parameter <span class="math-container">$\gamma$</span> that allows us to reduce variance by downweighting rewards corresponding to delayed effects, at the cost of introducing bias. This parameter corresponds to the discount factor used in discounted formulations of MDPs, but we treat it as a variance reduction parameter in an undiscounted problem.</p>
  </blockquote>
  <p>I know the Monte Carlo estimate of value function is given as:</p>
  <p><span class="math-container">$$V(s_t)=\sum_{l=t}^\infty \gamma^tr_t$$</span></p>
  <p>The bootstrapped estimate of value function is given as:</p>
  <p><span class="math-container">$$V(s_t)=r_t+\gamma V(s_{t+1})$$</span></p>
  <p>(In both equations, <span class="math-container">$\gamma$</span> is a discount factor.)</p>
  <p>The bootstrapped estimate is biased because it based on <span class="math-container">$V(s_{t+1})$</span> which is usually a biased estimate by some estimator such as a neural network. The Monte Carlo estimate is unbiased because it contains all rewards sampled from the environment. In this case, however, because the agent might take a lot of actions over the course of an episode, it's hard to assign credit to the right action, which means that a Monte Carlo estimate will have a high variance.</p>
  <p>Does this contradict what the paper says: &quot;a parameter <span class="math-container">$\gamma$</span> that allows us to reduce variance&quot;? Or does it simply mean the following: lower <span class="math-container">$\gamma$</span> gives smaller weights to distant future rewards, thus making value estimate less dependent on them, reducing variance in comparison to larger <span class="math-container">$\gamma$</span>, which make distant future rewards contribute significantly to value estimate. So introduction/existence of <span class="math-container">$\gamma$</span> itself does not reduce the variance but gives way to increase or decrease the variance.</p>
  
Tags: <reinforcement-learning><deep-rl><papers>
Title: Understanding Generalized Advantage Estimate in reinforcement learning
UserDisplayName: Mahesha999
ThreadText: # Understanding Generalized Advantage Estimate in reinforcement learning
  
  (Asked by: username_0 on 2021-05-15)
  
  I was reading the paper on [Generalized Advantage Estimate](https://arxiv.org/abs/1506.02438). It first introduces a generalized form of policy gradient equation without involving $\gamma$ and then it says the following:
  
  > We will introduce a parameter $\gamma$ that allows us to reduce variance by downweighting rewards corresponding to delayed effects, at the cost of introducing bias. This parameter corresponds to the discount factor used in discounted formulations of MDPs, but we treat it as a variance reduction parameter in an undiscounted problem.
  
  I know the Monte Carlo estimate of value function is given as:
  
  $$V(s\_t)=\sum\_{l=t}^\infty \gamma^tr\_t$$
  
  The bootstrapped estimate of value function is given as:
  
  $$V(s\_t)=r\_t+\gamma V(s\_{t+1})$$
  
  (In both equations, $\gamma$ is a discount factor.)
  
  The bootstrapped estimate is biased because it based on $V(s\_{t+1})$ which is usually a biased estimate by some estimator such as a neural network. The Monte Carlo estimate is unbiased because it contains all rewards sampled from the environment. In this case, however, because the agent might take a lot of actions over the course of an episode, it's hard to assign credit to the right action, which means that a Monte Carlo estimate will have a high variance.
  
  Does this contradict what the paper says: "a parameter $\gamma$ that allows us to reduce variance"? Or does it simply mean the following: lower $\gamma$ gives smaller weights to distant future rewards, thus making value estimate less dependent on them, reducing variance in comparison to larger $\gamma$, which make distant future rewards contribute significantly to value estimate. So introduction/existence of $\gamma$ itself does not reduce the variance but gives way to increase or decrease the variance.
  
  **Tags:** reinforcement-learning, deep-rl, papers
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-05-15. Score: 2
  
  > Bootstrapped estimate is biased because it based on $V(s\_{t+1})$ which is usually a biased estimate by some estimator such as neural network.
  
  I don't think this statement is totally correct in the context of the paper. Quoting the paper:
  
  > Taking $\gamma < 1$ introduces bias into the policy gradient estimate, regardless of the value function’s accuracy. On the other hand, $ \lambda < 1$ introduces bias only when the value function is inaccurate.
  
  So the initial $\gamma$-discounting introduces bias regardless of the accuracy of the $V(s)$.
  
  Same point is reiterated in the footnote on the page 3:
  
  > Note, that we have already introduced bias by using $A^{\pi,\gamma}$ in place of $A^\pi$; here we are concerned with obtaining an unbiased estimate of $g^\gamma$, which is a biased estimate of the policy gradient of the undiscounted MDP.
  
  To summarize the above:
  
  1. the original MDP under consideration is undiscounted
  2. authors introduce $\gamma$-discounting to reduce variance (at cost of bias) of the PG estimate - this happens even if we had accurate $V(s)$
  3. later, authors introduce another $\lambda$-discounting which also controls bias-variance tradeoff - now due to inaccuracy in $V(s)$
  
  As for why lower $\gamma$ reduces variance (p.2 above) - your explanation is on point: smaller $\gamma$ $\Rightarrow$ smaller weights for distant rewards $\Rightarrow$ less variance (but more bias).
  
  
  ---
  
  