Answers.Body: <p>Perfect. </p>
  
  <p>To back up your intuition about there not being a <span class="math-container">$G_5$</span>, refer to the definition of discounted return in the periodic case (3.11). 
  <span class="math-container">$$G_t \doteq \sum_{k=t+1}^T \gamma^{k-t-1} R_k$$</span></p>
  
  <p>You'll see that <span class="math-container">$G_5$</span> would be written as a sum with no terms in it, since <span class="math-container">$T=5$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-07T19:48:32.480Z
Answers.LastActivityDate: 2019-03-07T19:48:32.480Z
Answers.UserDisplayName: Philip Raeisghasem
Body: <p>Let's use Excercise 3.8 from Sutton, Barto - Introduction to RL:</p>
  
  <blockquote>
    <p>Suppose <span class="math-container">$\gamma = 0.5$</span> and following sequence of rewards is received
    <span class="math-container">$R_1=-1$</span> , <span class="math-container">$R_2=2$</span> , <span class="math-container">$R_3=6$</span> , <span class="math-container">$R_4=3$</span> , <span class="math-container">$R_5=2$</span> , with <span class="math-container">$T=5$</span> . What
    are <span class="math-container">$G_0, G_1, ..., G_5?$</span></p>
  </blockquote>
  
  <p>There isn't <span class="math-container">$G_5$</span> because <span class="math-container">$R_5$</span> is last reward. Am I understanding it right? </p>
  
  <p>So:</p>
  
  <p><span class="math-container">$G_4 = 2$</span></p>
  
  <p><span class="math-container">$G_3 = 3 + 0.5*2 = 4$</span></p>
  
  <p><span class="math-container">$G_2 = 6+0.5*4 = 8$</span></p>
  
  <p><span class="math-container">$G_1 = 2+0.5*8 = 6$</span></p>
  
  <p><span class="math-container">$G_0 = -1 +0.5*6 = 2$</span></p>
  
Tags: <reinforcement-learning><rewards><return>
Title: Given specific rewards, how can I calculate the returns for each time step?
UserDisplayName: Jakub Bielan
ThreadText: # Given specific rewards, how can I calculate the returns for each time step?
  
  (Asked by: username_0 on 2019-03-07)
  
  Let's use Excercise 3.8 from Sutton, Barto - Introduction to RL:
  
  > Suppose $\gamma = 0.5$ and following sequence of rewards is received
  > $R\_1=-1$ , $R\_2=2$ , $R\_3=6$ , $R\_4=3$ , $R\_5=2$ , with $T=5$ . What
  > are $G\_0, G\_1, ..., G\_5?$
  
  There isn't $G\_5$ because $R\_5$ is last reward. Am I understanding it right?
  
  So:
  
  $G\_4 = 2$
  
  $G\_3 = 3 + 0.5\*2 = 4$
  
  $G\_2 = 6+0.5\*4 = 8$
  
  $G\_1 = 2+0.5\*8 = 6$
  
  $G\_0 = -1 +0.5\*6 = 2$
  
  **Tags:** reinforcement-learning, rewards, return
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-07. Score: 1 | Accepted answer
  
  Perfect.
  
  To back up your intuition about there not being a $G\_5$, refer to the definition of discounted return in the periodic case (3.11).
  $$G\_t \doteq \sum\_{k=t+1}^T \gamma^{k-t-1} R\_k$$
  
  You'll see that $G\_5$ would be written as a sum with no terms in it, since $T=5$.
  
  
  ---
  
  