<p>The important part, where you can see a single reward value is used for <span class="math-container">$n$</span> different updates, is the part where a sum of <span class="math-container">$R_i$</span> values with <span class="math-container">$i$</span> ranging from <span class="math-container">$\tau + 1$</span> to <span class="math-container">$\tau + n$</span> is assigned to <span class="math-container">$G$</span>.</p>

<p>So yes, the outer loop of the algorithm always does at most one update per iteration, but for that update it uses multiple previously observed <span class="math-container">$R_i$</span> values. Each of those <span class="math-container">$R_i$</span> values is used for multiple updates (not multiple updates at the same time, but multiple updates spread out over different iterations).</p>

CC BY-SA 4.0
2018-01-03T14:27:18.747Z
2018-11-26T15:20:23.920Z
2018-11-26T15:20:23.920Z
Dennis Soemers
<p>I am trying to understand the algorithm for n-step SARSA from <a href="http://incompleteideas.net/book/RLbook2018.pdf" rel="nofollow noreferrer">Sutton and Barto (2nd Edition)</a>. As I understand it, this algorithm should update n state-action values, but I cannot see where it is propagated backward. Can someone explain to me how that works?</p>

<reinforcement-learning><sutton-barto><sarsa>
When does backward propagation occur in n-step SARSA?
nehalem
# When does backward propagation occur in n-step SARSA?

(Asked by: username_0 on 2017-02-15)

I am trying to understand the algorithm for n-step SARSA from [Sutton and Barto (2nd Edition)](http://incompleteideas.net/book/RLbook2018.pdf). As I understand it, this algorithm should update n state-action values, but I cannot see where it is propagated backward. Can someone explain to me how that works?

**Tags:** reinforcement-learning, sutton-barto, sarsa


---

## 1 answers

### Answer by username_1 on 2018-01-03. Score: 1

The important part, where you can see a single reward value is used for $n$ different updates, is the part where a sum of $R\_i$ values with $i$ ranging from $\tau + 1$ to $\tau + n$ is assigned to $G$.

So yes, the outer loop of the algorithm always does at most one update per iteration, but for that update it uses multiple previously observed $R\_i$ values. Each of those $R\_i$ values is used for multiple updates (not multiple updates at the same time, but multiple updates spread out over different iterations).


---

