Body: <p>I was reading the book &quot;Reinforcement Learning: An Introduction&quot; by Sutton and Barto. In section 7.3, they write the formula for n-step off-policy TD as</p>
  <p><span class="math-container">$$V(S_t) = V(S_{t-1}) + \alpha \rho_{t:t+n-1}[G_{t:t+n} - V(S_{t-1})],$$</span></p>
  <p>where <span class="math-container">$V(S_{t})$</span> is state value function of the state <span class="math-container">$S$</span> at time <span class="math-container">$t$</span> and <span class="math-container">$ G_{t:t+n} \doteq \sum_{i=t}^{t+n-1}\gamma^{i-t}R_{i+1} + \gamma^n V(S_{t+n})$</span> and <span class="math-container">$\rho_{t:t+n-1}$</span> is the importance sampling ratio.</p>
  <p>I tried to prove this equation for <span class="math-container">$n = 1$</span> using the incremental update of the value function. Now I end up with this formula:
  <span class="math-container">$$V(S_t) = \frac{1}{t} \sum_{j=1}^{t} \rho_{j}G_{j} $$</span>
  <span class="math-container">$$V(S_t)= \frac{1}{t}(\rho_{t}G_{t} + \sum_{j=1}^{t-1}\rho_{j}G_{j}) $$</span>
  <span class="math-container">$$V(S_t) = \frac{1}{t}(\rho_t G_t + (t-1)V(S_{t-1}))$$</span>
  <span class="math-container">$$V(S_t)=V(S_{t-1}) + \frac{1}{t}(\rho_{t}G_{t} - V(S_{t-1}))$$</span>
  I know I'm wrong because this does not match with the above equation. But can anyone please show me where I am wrong?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-05T22:40:14.103Z
Comments.Text: Do you remember which version of the book you were reading at the time?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-07T14:26:44.910Z
Comments.Text: It was a draft version
Comments.UserDisplayName: Swakshar Deb
Tags: <reinforcement-learning><proofs><temporal-difference-methods><off-policy-methods><importance-sampling>
Title: How can I derive n-step off-policy temporal difference formula?
UserDisplayName: Swakshar Deb
ThreadText: # How can I derive n-step off-policy temporal difference formula?
  
  (Asked by: username_0 on 2020-08-10)
  
  I was reading the book "Reinforcement Learning: An Introduction" by Sutton and Barto. In section 7.3, they write the formula for n-step off-policy TD as
  
  $$V(S\_t) = V(S\_{t-1}) + \alpha \rho\_{t:t+n-1}[G\_{t:t+n} - V(S\_{t-1})],$$
  
  where $V(S\_{t})$ is state value function of the state $S$ at time $t$ and $ G\_{t:t+n} \doteq \sum\_{i=t}^{t+n-1}\gamma^{i-t}R\_{i+1} + \gamma^n V(S\_{t+n})$ and $\rho\_{t:t+n-1}$ is the importance sampling ratio.
  
  I tried to prove this equation for $n = 1$ using the incremental update of the value function. Now I end up with this formula:
  $$V(S\_t) = \frac{1}{t} \sum\_{j=1}^{t} \rho\_{j}G\_{j} $$
  $$V(S\_t)= \frac{1}{t}(\rho\_{t}G\_{t} + \sum\_{j=1}^{t-1}\rho\_{j}G\_{j}) $$
  $$V(S\_t) = \frac{1}{t}(\rho\_t G\_t + (t-1)V(S\_{t-1}))$$
  $$V(S\_t)=V(S\_{t-1}) + \frac{1}{t}(\rho\_{t}G\_{t} - V(S\_{t-1}))$$
  I know I'm wrong because this does not match with the above equation. But can anyone please show me where I am wrong?
  
  **Tags:** reinforcement-learning, proofs, temporal-difference-methods, off-policy-methods, importance-sampling
  
  > **Comments:**
  > **username_1**: Do you remember which version of the book you were reading at the time?  
  > **username_0**: It was a draft version  
  
  ---
  
  ## 0 answers
  
  