Answers.Body: <p><em>Theoretically</em>, nothing precludes the use of <span class="math-container">$\lambda$</span>-returns in actor-critic methods. The <span class="math-container">$\lambda$</span>-return is an unbiased estimator of the Monte Carlo (MC) return, which means they are essentially interchangeable. In fact, as discussed in <a href="https://arxiv.org/pdf/1506.02438.pdf" rel="nofollow noreferrer">High-Dimensional Continuous Control Using Generalized Advantage Estimation</a>, using the <span class="math-container">$\lambda$</span>-return instead of the MC return can actually help reduce the variance of gradient updates.</p>
  
  <p>The above is similar to <a href="https://ai.stackexchange.com/a/17095/32070">my answer in the other question you linked</a>, so let me try to answer your question more specifically. <em>Even though we can use <span class="math-container">$\lambda$</span>-returns, why are they not too common in practice?</em> I suspect there might be a few reasons:</p>
  
  <ol>
  <li><p>Empirically, faster credit assignment might be more desirable than lower variance. Sometimes the learning speed of your algorithm is constrained simply by how quickly you can learn about the consequences of certain actions. In this case, it is faster to use the MC return, even if it theoretically has higher variance than the <span class="math-container">$\lambda$</span>-return.</p></li>
  <li><p>When proposing a new algorithm, adding <span class="math-container">$\lambda$</span>-returns to it might give it an "unfair" advantage if the other baseline methods do not use them (the reviewers would not like this), so researchers tend to favor simpler 1-step or MC returns for the sake of consistency. I would guess this is why you don't typically see <span class="math-container">$\lambda$</span>-returns in papers that propose new actor-critic methods. In some sense, it is generally assumed that you could always add <span class="math-container">$\lambda$</span>-returns to them later and probably get better performance.<span class="math-container">$^*$</span></p></li>
  <li><p>A decent number of deep RL researchers don't know what <span class="math-container">$\lambda$</span>-returns are. This is especially true if they come from a pure deep learning background; they may have never read <a href="http://www.incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">Reinforcement Learning: An Introduction</a> which is where most people are introduced to TD(<span class="math-container">$\lambda$</span>) and <span class="math-container">$\lambda$</span>-returns.</p></li>
  </ol>
  
  <p><span class="math-container">$^*$</span>Exceptions to this are papers like mine (<a href="https://arxiv.org/abs/1810.09967" rel="nofollow noreferrer">Reconciling <span class="math-container">$\lambda$</span>-Returns with Experience Replay</a>) where the contribution <em>is</em> the use of <span class="math-container">$\lambda$</span>-returns in methods that previously could not use them. But actor-critic methods are pretty straightforward to combine with <span class="math-container">$\lambda$</span>-returns.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-20T12:04:59.650Z
Answers.Text: Why are $\lambda$ returns unbiased estimators of MC returns, given that they are an average of returns that go from TD(0) to TD(1)? Maybe this could be a question on the main site. Anyway, you say that $\lambda$ returns are unbiased estimators of MC returns, but then you say "In fact, they help to reduce the variance", while the use of MC returns usually leads to high variance. In other words, the "In fact" isn't probably the best way of starting that phrase. Also, I think you should better distinguish between "faster credit assignment" and "faster implementation".
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-20T20:56:10.300Z
Answers.Text: The $\lambda$-return is a statistically unbiased estimator of the MC return because it's a weighted average of $n$-step returns, each of which are themselves unbiased estimators of the MC return on average. This should not be confused with the "bias" due to bootstrapping; the terminology is a bit confusing because there are two notions of bias here. I am referring to the former notion: each $n$-step return, in expectation, is equal to the MC return and therefore so is the $\lambda$-return.
Answers.UserDisplayName: Brett Daley
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-20T21:08:10.623Z
Answers.Text: This also explains why the $\lambda$-return should reduce variance. Each $n$-step return has some variance, but they all have the same expectation. The $\lambda$-return is a convex combination of the $n$-step returns, where the weights are positive and sum to 1. Therefore, the $\lambda$-return must also have the same expectation. But from the variance formula $Var(aX+bY)=a^2 Var(X)+b^2 Var(Y)$ (assuming independence), you can see how this should reduce the variance because $a^2 + b^2 < 1$. This can be viewed as an [ensemble method](https://en.wikipedia.org/wiki/Ensemble_learning) of sorts.
Answers.UserDisplayName: Brett Daley
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-20T21:11:56.523Z
Answers.Text: Finally, I did intend to say "faster credit assignment". The MC return propagates reward information back in time to all states in the episode. The $\lambda$-return does too, technically/mathematically, but the exponential decay means that it practically stops updating states after some finite number of timesteps.
Answers.UserDisplayName: Brett Daley
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-20T03:16:59.083Z
Answers.LastActivityDate: 2020-02-20T03:16:59.083Z
Answers.UserDisplayName: Brett Daley
Body: <p>Is there either an empirical or theoretical reason that actor-critic algorithms with eligibility traces have not been more fully explored? I was hoping to find a paper or implementation or both for continuous tasks (not episodic) in continuous state-action spaces.</p>
  
  <p>This has been the only related question on SE-AI that I have been able to find <a href="https://ai.stackexchange.com/q/10049/2444">Why are lambda returns so rarely used in policy gradients?</a>.</p>
  
  <p>Although I appreciated the dialog and found it useful, I was wondering if there was any further detail or reasoning that might help explain the void.</p>
  
Tags: <reinforcement-learning><actor-critic-methods><temporal-difference-methods><eligibility-traces><td-lambda>
Title: Why not more TD() in actor-critic algorithms?
UserDisplayName: Nick Kunz
ThreadText: # Why not more TD() in actor-critic algorithms?
  
  (Asked by: username_0 on 2020-02-17)
  
  Is there either an empirical or theoretical reason that actor-critic algorithms with eligibility traces have not been more fully explored? I was hoping to find a paper or implementation or both for continuous tasks (not episodic) in continuous state-action spaces.
  
  This has been the only related question on SE-AI that I have been able to find [Why are lambda returns so rarely used in policy gradients?](https://ai.stackexchange.com/q/10049/2444).
  
  Although I appreciated the dialog and found it useful, I was wondering if there was any further detail or reasoning that might help explain the void.
  
  **Tags:** reinforcement-learning, actor-critic-methods, temporal-difference-methods, eligibility-traces, td-lambda
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-20. Score: 2 | Accepted answer
  
  *Theoretically*, nothing precludes the use of $\lambda$-returns in actor-critic methods. The $\lambda$-return is an unbiased estimator of the Monte Carlo (MC) return, which means they are essentially interchangeable. In fact, as discussed in [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://arxiv.org/pdf/1506.02438.pdf), using the $\lambda$-return instead of the MC return can actually help reduce the variance of gradient updates.
  
  The above is similar to [my answer in the other question you linked](https://ai.stackexchange.com/a/17095/32070), so let me try to answer your question more specifically. *Even though we can use $\lambda$-returns, why are they not too common in practice?* I suspect there might be a few reasons:
  
  1. Empirically, faster credit assignment might be more desirable than lower variance. Sometimes the learning speed of your algorithm is constrained simply by how quickly you can learn about the consequences of certain actions. In this case, it is faster to use the MC return, even if it theoretically has higher variance than the $\lambda$-return.
  2. When proposing a new algorithm, adding $\lambda$-returns to it might give it an "unfair" advantage if the other baseline methods do not use them (the reviewers would not like this), so researchers tend to favor simpler 1-step or MC returns for the sake of consistency. I would guess this is why you don't typically see $\lambda$-returns in papers that propose new actor-critic methods. In some sense, it is generally assumed that you could always add $\lambda$-returns to them later and probably get better performance.$^\*$
  3. A decent number of deep RL researchers don't know what $\lambda$-returns are. This is especially true if they come from a pure deep learning background; they may have never read [Reinforcement Learning: An Introduction](http://www.incompleteideas.net/book/the-book-2nd.html) which is where most people are introduced to TD($\lambda$) and $\lambda$-returns.
  
  $^\*$Exceptions to this are papers like mine ([Reconciling $\lambda$-Returns with Experience Replay](https://arxiv.org/abs/1810.09967)) where the contribution *is* the use of $\lambda$-returns in methods that previously could not use them. But actor-critic methods are pretty straightforward to combine with $\lambda$-returns.
  
  > **Comments:**
  > **username_2**: Why are $\lambda$ returns unbiased estimators of MC returns, given that they are an average of returns that go from TD(0) to TD(1)? Maybe this could be a question on the main site. Anyway, you say that $\lambda$ returns are unbiased estimators of MC returns, but then you say "In fact, they help to reduce the variance", while the use of MC returns usually leads to high variance. In other words, the "In fact" isn't probably the best way of starting that phrase. Also, I think you should better distinguish between "faster credit assignment" and "faster implementation".  
  > **username_1**: The $\lambda$-return is a statistically unbiased estimator of the MC return because it's a weighted average of $n$-step returns, each of which are themselves unbiased estimators of the MC return on average. This should not be confused with the "bias" due to bootstrapping; the terminology is a bit confusing because there are two notions of bias here. I am referring to the former notion: each $n$-step return, in expectation, is equal to the MC return and therefore so is the $\lambda$-return.  
  > **username_1**: This also explains why the $\lambda$-return should reduce variance. Each $n$-step return has some variance, but they all have the same expectation. The $\lambda$-return is a convex combination of the $n$-step returns, where the weights are positive and sum to 1. Therefore, the $\lambda$-return must also have the same expectation. But from the variance formula $Var(aX+bY)=a^2 Var(X)+b^2 Var(Y)$ (assuming independence), you can see how this should reduce the variance because $a^2 + b^2 < 1$. This can be viewed as an [ensemble method](https://en.wikipedia.org/wiki/Ensemble_learning) of sorts.  
  > **username_1**: Finally, I did intend to say "faster credit assignment". The MC return propagates reward information back in time to all states in the episode. The $\lambda$-return does too, technically/mathematically, but the exponential decay means that it practically stops updating states after some finite number of timesteps.  
  
  ---
  
  