Body: <p>The pseudocode below is taken from Barto and Sutton's &quot;Reinforcement Learning: an introduction&quot;. It shows an actor-critic implementation with eligibility traces. My question is: if I set <span class="math-container">$\lambda^{\theta}=1$</span> and replace <span class="math-container">$\delta$</span> with the immediate reward <span class="math-container">$R_t$</span>, do I get a backwards implementation of REINFORCE?</p>
  <p><a href="https://i.sstatic.net/dSgKV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/dSgKV.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T14:33:28.643Z
Comments.Text: Well, I think, if you set $\lambda^{\theta}$ to 1, there will be no sense in eligibility traces. In this case you are going to update $\theta$ directly based on policy gradient multiplied by TD-error $\delta$
Comments.UserDisplayName: Georgii Firsov
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T15:04:28.240Z
Comments.Text: Well, not quite. $z^{\theta}$ will still hold $\nabla \pi(A|S, \theta)$ from previous steps, and will decay them based on $\gamma$. That means past updates will be credited in a decaying manner going forward. Since ET implement a backward view of the $\lambda$-return, and setting $\gamma=1$ there should result in MC updates. Is there anything wrong in this reasoning?
Comments.UserDisplayName: Javier Ventajas Hernández
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T15:07:57.510Z
Comments.Text: Now that I think about it, setting both $\lambda^{\theta}=1$ and $\lambda^w=1$ should make this method become REINFORCE with baseline, shouldn't it?
Comments.UserDisplayName: Javier Ventajas Hernández
Tags: <reinforcement-learning><actor-critic-methods><reinforce><eligibility-traces>
Title: How to implement REINFORCE with eligibility traces?
UserDisplayName: Javier Ventajas Hernández
ThreadText: # How to implement REINFORCE with eligibility traces?
  
  (Asked by: username_0 on 2021-01-20)
  
  The pseudocode below is taken from Barto and Sutton's "Reinforcement Learning: an introduction". It shows an actor-critic implementation with eligibility traces. My question is: if I set $\lambda^{\theta}=1$ and replace $\delta$ with the immediate reward $R\_t$, do I get a backwards implementation of REINFORCE?
  
  [![enter image description here](https://i.sstatic.net/dSgKV.png)](https://i.sstatic.net/dSgKV.png)
  
  **Tags:** reinforcement-learning, actor-critic-methods, reinforce, eligibility-traces
  > **Comments:**
  > **username_1**: Well, I think, if you set $\lambda^{\theta}$ to 1, there will be no sense in eligibility traces. In this case you are going to update $\theta$ directly based on policy gradient multiplied by TD-error $\delta$  
  > **username_0**: Well, not quite. $z^{\theta}$ will still hold $\nabla \pi(A|S, \theta)$ from previous steps, and will decay them based on $\gamma$. That means past updates will be credited in a decaying manner going forward. Since ET implement a backward view of the $\lambda$-return, and setting $\gamma=1$ there should result in MC updates. Is there anything wrong in this reasoning?  
  > **username_0**: Now that I think about it, setting both $\lambda^{\theta}=1$ and $\lambda^w=1$ should make this method become REINFORCE with baseline, shouldn't it?  
  
  ---
  
  ## 0 answers
  
  