Answers.Body: <p>Recent actor-critic algorithms <em>do</em> use <span class="math-container">$\lambda$</span>-returns, but they are disguised as something called the <a href="https://arxiv.org/pdf/1506.02438.pdf" rel="nofollow noreferrer">Generalized Advantage Estimator</a> defined as <span class="math-container">$A^{GAE}_t = \sum_{i=0}^{\infty} (\gamma\lambda)^i \delta_{t+i}$</span> where <span class="math-container">$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$</span>. This turns out to be identically equal to <span class="math-container">$[G^\lambda_t - V(s_t)]$</span>, <em>i.e.</em> the <span class="math-container">$\lambda$</span>-return with a value-function baseline subtracted from it. Theoretically, any actor-critic gradient method could use this quite easily; it was combined with TRPO in the GAE paper, and later used for PPO. Similarly, ACER uses an off-policy variant known as Retrace(<span class="math-container">$\lambda$</span>).</p>
  <p>For replay methods like DQN or DDPG, it is harder to implement <span class="math-container">$\lambda$</span>-returns. This is why they have historically defaulted to <span class="math-container">$n$</span>-step returns as @DennisSoemers mentioned. I recently <a href="https://papers.nips.cc/paper/8397-reconciling-returns-with-experience-replay" rel="nofollow noreferrer">published a paper</a> that describes a way to efficiently combine <span class="math-container">$\lambda$</span>-returns with experience replay, which I hope will increase the popularity of <span class="math-container">$\lambda$</span>-returns for these methods.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-14T08:08:53.580Z
Answers.LastActivityDate: 2020-08-11T16:59:36.413Z
Answers.LastEditDate: 2020-08-11T16:59:36.413Z
Answers.UserDisplayName: Brett Daley
Answers.Body: <p>That can be done. For example, Chapter 13 of the 2nd edition of Sutton and Barto's Reinforcement Learning book (page 332) has pseudocode for "Actor Critic with Eligibility Traces". It's using <span class="math-container">$G_t^{\lambda}$</span> returns for the critic (value function estimator), but also for the actor's policy gradients. </p>
  
  <p>Note that you do not <em>explicitly</em> see the <span class="math-container">$G_t^{\lambda}$</span> returns mentioned in the pseudocode. They are being used <em>implicitly</em> through eligibility traces, which allow for an efficient online implementation (the "backward view").</p>
  
  <hr>
  
  <p>I do indeed have the impression that such uses are fairly rare in recent research though. I haven't personally played around with policy gradient methods to tell from personal experience why that would be. My guess would be that it is because policy gradient methods are almost always combined with Deep Neural Networks, and <strong>variance</strong> is already a big enough problem in training these things without starting to involve long-trajectory returns. </p>
  
  <p>If you use large <span class="math-container">$\lambda$</span> with <span class="math-container">$\lambda$</span>-returns, you get low bias, but high variance. For <span class="math-container">$\lambda = 1$</span>, you basically get REINFORCE again, which isn't really used much in practice, and has very high variance. For <span class="math-container">$\lambda = 0$</span>, you just get one-step returns again. Higher values for <span class="math-container">$\lambda$</span> (such as <span class="math-container">$\lambda = 0.8$</span>) tend to work very well in my experience with tabular methods or linear function approximation, but I suspect the variance may simply be too much when using DNNs.</p>
  
  <p>Note that it is quite popular to use <span class="math-container">$n$</span>-step returns with a fixed, generally fairly small, <span class="math-container">$n$</span> in Deep RL approaches. For instance, I believe the original A3C paper used <span class="math-container">$5$</span>-step returns, and Rainbow uses <span class="math-container">$3$</span>-step returns. These often work better in practice than <span class="math-container">$1$</span>-step returns, but still have reasonably low variance due to using small <span class="math-container">$n$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-18T21:29:27.490Z
Answers.Text: Thanks for the answer! It makes sense that lambda returns would add more variance compared to the more common TD(small n) rewards, so if variance reduction is a priority then one would use TD(0) or TD(small n).
Answers.UserDisplayName: jhinGhin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-18T14:29:37.537Z
Answers.LastActivityDate: 2019-01-18T14:29:37.537Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I've seen the Monte Carlo return <span class="math-container">$G_{t}$</span> being used in REINFORCE and the TD(<span class="math-container">$0$</span>) target <span class="math-container">$r_t + \gamma Q(s', a')$</span> in vanilla actor-critic. However, I've never seen someone use the lambda return <span class="math-container">$G^{\lambda}_{t}$</span> in these situations, nor in any other algorithms.</p>
  <p>Is there a specific reason for this? Could there be performance improvements if we used <span class="math-container">$G^{\lambda}_{t}$</span>?</p>
  
Tags: <reinforcement-learning><policy-gradients><reinforce><return><td-lambda>
Title: Why are lambda returns so rarely used in policy gradients?
UserDisplayName: jhinGhin
ThreadText: # Why are lambda returns so rarely used in policy gradients?
  
  (Asked by: username_0 on 2019-01-17)
  
  I've seen the Monte Carlo return $G\_{t}$ being used in REINFORCE and the TD($0$) target $r\_t + \gamma Q(s', a')$ in vanilla actor-critic. However, I've never seen someone use the lambda return $G^{\lambda}\_{t}$ in these situations, nor in any other algorithms.
  
  Is there a specific reason for this? Could there be performance improvements if we used $G^{\lambda}\_{t}$?
  
  **Tags:** reinforcement-learning, policy-gradients, reinforce, return, td-lambda
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-01-18. Score: 11 | Accepted answer
  
  That can be done. For example, Chapter 13 of the 2nd edition of Sutton and Barto's Reinforcement Learning book (page 332) has pseudocode for "Actor Critic with Eligibility Traces". It's using $G\_t^{\lambda}$ returns for the critic (value function estimator), but also for the actor's policy gradients.
  
  Note that you do not *explicitly* see the $G\_t^{\lambda}$ returns mentioned in the pseudocode. They are being used *implicitly* through eligibility traces, which allow for an efficient online implementation (the "backward view").
  
  ---
  
  I do indeed have the impression that such uses are fairly rare in recent research though. I haven't personally played around with policy gradient methods to tell from personal experience why that would be. My guess would be that it is because policy gradient methods are almost always combined with Deep Neural Networks, and **variance** is already a big enough problem in training these things without starting to involve long-trajectory returns.
  
  If you use large $\lambda$ with $\lambda$-returns, you get low bias, but high variance. For $\lambda = 1$, you basically get REINFORCE again, which isn't really used much in practice, and has very high variance. For $\lambda = 0$, you just get one-step returns again. Higher values for $\lambda$ (such as $\lambda = 0.8$) tend to work very well in my experience with tabular methods or linear function approximation, but I suspect the variance may simply be too much when using DNNs.
  
  Note that it is quite popular to use $n$-step returns with a fixed, generally fairly small, $n$ in Deep RL approaches. For instance, I believe the original A3C paper used $5$-step returns, and Rainbow uses $3$-step returns. These often work better in practice than $1$-step returns, but still have reasonably low variance due to using small $n$.
  
  > **Comments:**
  > **username_0**: Thanks for the answer! It makes sense that lambda returns would add more variance compared to the more common TD(small n) rewards, so if variance reduction is a priority then one would use TD(0) or TD(small n).  
  
  ---
  
  ### Answer by username_1 on 2019-12-14. Score: 9
  
  Recent actor-critic algorithms *do* use $\lambda$-returns, but they are disguised as something called the [Generalized Advantage Estimator](https://arxiv.org/pdf/1506.02438.pdf) defined as $A^{GAE}\_t = \sum\_{i=0}^{\infty} (\gamma\lambda)^i \delta\_{t+i}$ where $\delta\_t = r\_t + \gamma V(s\_{t+1}) - V(s\_t)$. This turns out to be identically equal to $[G^\lambda\_t - V(s\_t)]$, *i.e.* the $\lambda$-return with a value-function baseline subtracted from it. Theoretically, any actor-critic gradient method could use this quite easily; it was combined with TRPO in the GAE paper, and later used for PPO. Similarly, ACER uses an off-policy variant known as Retrace($\lambda$).
  
  For replay methods like DQN or DDPG, it is harder to implement $\lambda$-returns. This is why they have historically defaulted to $n$-step returns as @username_2 mentioned. I recently [published a paper](https://papers.nips.cc/paper/8397-reconciling-returns-with-experience-replay) that describes a way to efficiently combine $\lambda$-returns with experience replay, which I hope will increase the popularity of $\lambda$-returns for these methods.
  
  
  ---
  
  