Answers.Body: <p>Your table is almost correct. It is a minor difference, you should not have a $R_0$, the top row, leftmost column of numbers should be empty. That is because the first reward is $R_1$ (a result of taking action $A_0$ in state $S_0$). The alignment of the columns on the right hand side <em>is</em> correct though.</p>
  
  <p>It might help to add the time step number at the top. But the important detail is that $G_t$ is a measure of all <em>future</em> rewards. </p>
  
  <p>For instance it should always be zero when you reach a terminal state, which is what your example shows. Whilst it is quite common to receive a reward at the end of an episode (i.e. whilst arriving at a terminal state), also as your example shows.</p>
  
  <p>The decision to have reward time step match that of <em>next state</em> is a convention that can be altered. A few RL sources, but not Sutton &amp; Barto, will have reward on same time step as state and action that decided it, and thus $R_0$ will exist. The reward of 1 for reaching the terminal state would then be 1 time step earlier in your table, and there would be no $R_4$. The definition of $G_t$ would need to change to match ($G_t = R_t + \gamma G_{t+1})$, as well as other equations. That would change your table also - the reward sequence (top row) would shift to the left.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-24T19:11:08.903Z
Answers.LastActivityDate: 2018-08-25T07:02:20.637Z
Answers.LastEditDate: 2018-08-25T07:02:20.637Z
Answers.UserDisplayName: Neil Slater
Body: <p>Sutton and Barto 2018 define the discounted return <span class="math-container">$G_t$</span> the following way (<a href="https://drive.google.com/file/d/1xeUDVGWGUUv1-ccUMAZHJLej2C7aAFWY/view" rel="nofollow noreferrer">p 55</a>):</p>
  
  <p><a href="https://i.sstatic.net/ilEt7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ilEt7.png" alt="enter image description here"></a></p>
  
  <p>Is my interpretation correct?</p>
  
  <p><a href="https://i.sstatic.net/yfxBu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yfxBu.png" alt="enter image description here"></a></p>
  
  <p>Or should all "1" be in the same column?</p>
  
Tags: <reinforcement-learning><rewards><sutton-barto><return>
Title: Is my interpretation of the return correct?
UserDisplayName: LaTeXian
ThreadText: # Is my interpretation of the return correct?
  
  (Asked by: username_0 on 2018-08-24)
  
  Sutton and Barto 2018 define the discounted return $G\_t$ the following way ([p 55](https://drive.google.com/file/d/1xeUDVGWGUUv1-ccUMAZHJLej2C7aAFWY/view)):
  
  [![enter image description here](https://i.sstatic.net/ilEt7.png)](https://i.sstatic.net/ilEt7.png)
  
  Is my interpretation correct?
  
  [![enter image description here](https://i.sstatic.net/yfxBu.png)](https://i.sstatic.net/yfxBu.png)
  
  Or should all "1" be in the same column?
  
  **Tags:** reinforcement-learning, rewards, sutton-barto, return
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-08-24. Score: 3
  
  Your table is almost correct. It is a minor difference, you should not have a $R\_0$, the top row, leftmost column of numbers should be empty. That is because the first reward is $R\_1$ (a result of taking action $A\_0$ in state $S\_0$). The alignment of the columns on the right hand side *is* correct though.
  
  It might help to add the time step number at the top. But the important detail is that $G\_t$ is a measure of all *future* rewards.
  
  For instance it should always be zero when you reach a terminal state, which is what your example shows. Whilst it is quite common to receive a reward at the end of an episode (i.e. whilst arriving at a terminal state), also as your example shows.
  
  The decision to have reward time step match that of *next state* is a convention that can be altered. A few RL sources, but not Sutton & Barto, will have reward on same time step as state and action that decided it, and thus $R\_0$ will exist. The reward of 1 for reaching the terminal state would then be 1 time step earlier in your table, and there would be no $R\_4$. The definition of $G\_t$ would need to change to match ($G\_t = R\_t + \gamma G\_{t+1})$, as well as other equations. That would change your table also - the reward sequence (top row) would shift to the left.
  
  
  ---
  
  