<p>Your table is almost correct. It is a minor difference, you should not have a $R_0$, the top row, leftmost column of numbers should be empty. That is because the first reward is $R_1$ (a result of taking action $A_0$ in state $S_0$). The alignment of the columns on the right hand side <em>is</em> correct though.</p>

<p>It might help to add the time step number at the top. But the important detail is that $G_t$ is a measure of all <em>future</em> rewards. </p>

<p>For instance it should always be zero when you reach a terminal state, which is what your example shows. Whilst it is quite common to receive a reward at the end of an episode (i.e. whilst arriving at a terminal state), also as your example shows.</p>

<p>The decision to have reward time step match that of <em>next state</em> is a convention that can be altered. A few RL sources, but not Sutton &amp; Barto, will have reward on same time step as state and action that decided it, and thus $R_0$ will exist. The reward of 1 for reaching the terminal state would then be 1 time step earlier in your table, and there would be no $R_4$. The definition of $G_t$ would need to change to match ($G_t = R_t + \gamma G_{t+1})$, as well as other equations. That would change your table also - the reward sequence (top row) would shift to the left.</p>

CC BY-SA 4.0
2018-08-24T19:11:08.903Z
2018-08-25T07:02:20.637Z
2018-08-25T07:02:20.637Z
Neil Slater
<p>Sutton and Barto 2018 define the discounted return <span class="math-container">$G_t$</span> the following way (<a href="https://drive.google.com/file/d/1xeUDVGWGUUv1-ccUMAZHJLej2C7aAFWY/view" rel="nofollow noreferrer">p 55</a>):</p>

<p><a href="https://i.sstatic.net/ilEt7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ilEt7.png" alt="enter image description here"></a></p>

<p>Is my interpretation correct?</p>

<p><a href="https://i.sstatic.net/yfxBu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yfxBu.png" alt="enter image description here"></a></p>

<p>Or should all "1" be in the same column?</p>

<reinforcement-learning><rewards><sutton-barto><return>
Is my interpretation of the return correct?
LaTeXian
# Is my interpretation of the return correct?

(Asked by: username_0 on 2018-08-24)

Sutton and Barto 2018 define the discounted return $G\_t$ the following way ([p 55](https://drive.google.com/file/d/1xeUDVGWGUUv1-ccUMAZHJLej2C7aAFWY/view)):

[![enter image description here](https://i.sstatic.net/ilEt7.png)](https://i.sstatic.net/ilEt7.png)

Is my interpretation correct?

[![enter image description here](https://i.sstatic.net/yfxBu.png)](https://i.sstatic.net/yfxBu.png)

Or should all "1" be in the same column?

**Tags:** reinforcement-learning, rewards, sutton-barto, return


---

## 1 answers

### Answer by username_1 on 2018-08-24. Score: 3

Your table is almost correct. It is a minor difference, you should not have a $R\_0$, the top row, leftmost column of numbers should be empty. That is because the first reward is $R\_1$ (a result of taking action $A\_0$ in state $S\_0$). The alignment of the columns on the right hand side *is* correct though.

It might help to add the time step number at the top. But the important detail is that $G\_t$ is a measure of all *future* rewards.

For instance it should always be zero when you reach a terminal state, which is what your example shows. Whilst it is quite common to receive a reward at the end of an episode (i.e. whilst arriving at a terminal state), also as your example shows.

The decision to have reward time step match that of *next state* is a convention that can be altered. A few RL sources, but not Sutton & Barto, will have reward on same time step as state and action that decided it, and thus $R\_0$ will exist. The reward of 1 for reaching the terminal state would then be 1 time step earlier in your table, and there would be no $R\_4$. The definition of $G\_t$ would need to change to match ($G\_t = R\_t + \gamma G\_{t+1})$, as well as other equations. That would change your table also - the reward sequence (top row) would shift to the left.


---

