Answers.Body: <p>The important part, where you can see a single reward value is used for <span class="math-container">$n$</span> different updates, is the part where a sum of <span class="math-container">$R_i$</span> values with <span class="math-container">$i$</span> ranging from <span class="math-container">$\tau + 1$</span> to <span class="math-container">$\tau + n$</span> is assigned to <span class="math-container">$G$</span>.</p>
  
  <p>So yes, the outer loop of the algorithm always does at most one update per iteration, but for that update it uses multiple previously observed <span class="math-container">$R_i$</span> values. Each of those <span class="math-container">$R_i$</span> values is used for multiple updates (not multiple updates at the same time, but multiple updates spread out over different iterations).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-01-03T14:27:18.747Z
Answers.LastActivityDate: 2018-11-26T15:20:23.920Z
Answers.LastEditDate: 2018-11-26T15:20:23.920Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I am trying to understand the algorithm for n-step SARSA from <a href="http://incompleteideas.net/book/RLbook2018.pdf" rel="nofollow noreferrer">Sutton and Barto (2nd Edition)</a>. As I understand it, this algorithm should update n state-action values, but I cannot see where it is propagated backward. Can someone explain to me how that works?</p>
  
Tags: <reinforcement-learning><sutton-barto><sarsa>
Title: When does backward propagation occur in n-step SARSA?
UserDisplayName: nehalem
ThreadText: # When does backward propagation occur in n-step SARSA?
  
  (Asked by: username_0 on 2017-02-15)
  
  I am trying to understand the algorithm for n-step SARSA from [Sutton and Barto (2nd Edition)](http://incompleteideas.net/book/RLbook2018.pdf). As I understand it, this algorithm should update n state-action values, but I cannot see where it is propagated backward. Can someone explain to me how that works?
  
  **Tags:** reinforcement-learning, sutton-barto, sarsa
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-01-03. Score: 1
  
  The important part, where you can see a single reward value is used for $n$ different updates, is the part where a sum of $R\_i$ values with $i$ ranging from $\tau + 1$ to $\tau + n$ is assigned to $G$.
  
  So yes, the outer loop of the algorithm always does at most one update per iteration, but for that update it uses multiple previously observed $R\_i$ values. Each of those $R\_i$ values is used for multiple updates (not multiple updates at the same time, but multiple updates spread out over different iterations).
  
  
  ---
  
  