Answers.Body: <blockquote>
    <p>When lambda = 0 as in TD(0), how does the method learn? As it appears, with lambda = 0, there will never be a change in weight and hence no learning. </p>
  </blockquote>
  
  <p>I think the detail that you're missing is that one of the terms in the sum (the final "iteration" of the sum, the case where <span class="math-container">$k = t$</span>) has <span class="math-container">$\lambda$</span> raised to the power <span class="math-container">$0$</span>, and anything raised to the power <span class="math-container">$0$</span> (even <span class="math-container">$0$</span>) is equal to <span class="math-container">$1$</span>. So, for <span class="math-container">$\lambda = 0$</span>, your update equation becomes</p>
  
  <p><span class="math-container">$$\Delta w_t = \alpha \left( P_{t+1} - P_t \right) \nabla_w P_t,$$</span></p>
  
  <p>which is basically a one-step update (just like Sarsa).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-01T16:51:36.850Z
Answers.Text: At page $16$ of the same paper [Learning to Predict by the Methods
  of Temporal Differences](http://incompleteideas.net/papers/sutton-88-with-erratum.pdf) (1988), Sutton actually states that $\Delta w_t = \alpha \left( P_{t+1} - P_t \right) \nabla_w P_t$ is the learning rule when $\lambda = 0$.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-01T16:58:09.543Z
Answers.Text: He starts with the supervised setting and then derives the Widrow-Hoff (or delta) rule. The TD rule is then a special case of the delta rule, where the errors $z - P_t$ are replaced with a summation of the successive temporal-difference predictions. However, how is that specific 1-step TD learning rule _exactly_ related to the usual learning rules of (tabular) temporal difference methods, where apparently no _gradient_ is needed?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-01T17:33:29.070Z
Answers.Text: @nbro You can view tabular methods as methods using linear function "approximation", where there is a single binary feature for every possible state-action pair. Then there would be a gradient needed, but the gradient would simply be $1$ for the "binary feature" corresponding to the state-action pair, and $0$ everywhere else.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-01T15:07:06.217Z
Answers.LastActivityDate: 2019-06-01T15:07:06.217Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>In the paper <a href="http://incompleteideas.net/papers/sutton-88-with-erratum.pdf" rel="nofollow noreferrer">Learning to predict by the methods of temporal differences</a> (p. 15), the weights in the <a href="https://en.wikipedia.org/wiki/Temporal_difference_learning" rel="nofollow noreferrer">temporal difference learning</a> are updated as given by the equation
  <span class="math-container">$$
  \Delta w_t
  = \alpha \left(P_{t+1} - P_t\right) \sum_{k=1}^{t}{\lambda^{t-k} \nabla_w P_k}
  \tag{4}
  \,.$$</span>
  When <span class="math-container">$\lambda = 0$</span>, as in TD(0), how does the method learn? As it appears, with <span class="math-container">$\lambda = 0$</span>, there will never be a change in weight and hence no learning. </p>
  
  <p>Am I missing anything?</p>
  
Tags: <reinforcement-learning><temporal-difference-methods><notation>
Title: Understanding the equation of TD(0) in the paper "Learning to predict by the methods of temporal differences"
UserDisplayName: Amanda
ThreadText: # Understanding the equation of TD(0) in the paper "Learning to predict by the methods of temporal differences"
  
  (Asked by: username_0 on 2019-06-01)
  
  In the paper [Learning to predict by the methods of temporal differences](http://incompleteideas.net/papers/sutton-88-with-erratum.pdf) (p. 15), the weights in the [temporal difference learning](https://en.wikipedia.org/wiki/Temporal_difference_learning) are updated as given by the equation
  $$
  \Delta w\_t
  = \alpha \left(P\_{t+1} - P\_t\right) \sum\_{k=1}^{t}{\lambda^{t-k} \nabla\_w P\_k}
  \tag{4}
  \,.$$
  When $\lambda = 0$, as in TD(0), how does the method learn? As it appears, with $\lambda = 0$, there will never be a change in weight and hence no learning.
  
  Am I missing anything?
  
  **Tags:** reinforcement-learning, temporal-difference-methods, notation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-01. Score: 5 | Accepted answer
  
  > When lambda = 0 as in TD(0), how does the method learn? As it appears, with lambda = 0, there will never be a change in weight and hence no learning.
  
  I think the detail that you're missing is that one of the terms in the sum (the final "iteration" of the sum, the case where $k = t$) has $\lambda$ raised to the power $0$, and anything raised to the power $0$ (even $0$) is equal to $1$. So, for $\lambda = 0$, your update equation becomes
  
  $$\Delta w\_t = \alpha \left( P\_{t+1} - P\_t \right) \nabla\_w P\_t,$$
  
  which is basically a one-step update (just like Sarsa).
  
  > **Comments:**
  > **username_2**: At page $16$ of the same paper [Learning to Predict by the Methods of Temporal Differences](http://incompleteideas.net/papers/sutton-88-with-erratum.pdf) (1988), Sutton actually states that $\Delta w_t = \alpha \left( P_{t+1} - P_t \right) \nabla_w P_t$ is the learning rule when $\lambda = 0$.  
  > **username_2**: He starts with the supervised setting and then derives the Widrow-Hoff (or delta) rule. The TD rule is then a special case of the delta rule, where the errors $z - P_t$ are replaced with a summation of the successive temporal-difference predictions. However, how is that specific 1-step TD learning rule _exactly_ related to the usual learning rules of (tabular) temporal difference methods, where apparently no _gradient_ is needed?  
  > **username_1**: @username_2 You can view tabular methods as methods using linear function "approximation", where there is a single binary feature for every possible state-action pair. Then there would be a gradient needed, but the gradient would simply be $1$ for the "binary feature" corresponding to the state-action pair, and $0$ everywhere else.  
  
  ---
  
  