Answers.Body: <p>You know all the rewards. They're 5, 7, 7, 7, and 7s forever. The problem now boils down to essentially a geometric series computation.</p>
  <p><span class="math-container">$$ 
  G_0 = R_0 + \gamma G_1
  $$</span></p>
  <p><span class="math-container">$$ 
  G_0 = 5 + \gamma\sum_{k=0}^\infty 7\gamma^k 
  $$</span></p>
  <p><span class="math-container">$$ 
  G_0 = 5 + 7\gamma\sum_{k=0}^\infty\gamma^k 
  $$</span></p>
  <p><span class="math-container">$$ 
  G_0 = 5 + \frac{7\gamma}{1-\gamma} = \frac{5 + 2\gamma}{1-\gamma}
  $$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-29T14:42:41.703Z
Answers.LastActivityDate: 2020-06-29T14:42:41.703Z
Answers.UserDisplayName: harwiltz
Answers.Body: <p>There are a few ways to resolve values of infinite sums. In this case, we can use a simple technique of self-reference to create a solvable equation.</p>
  <p>I will show how to do it for the generic case here of an MDP with same reward <span class="math-container">$r$</span> on each timestep:</p>
  <p><span class="math-container">$$G_t = \sum_{k=0}^{\infty} \gamma^k r$$</span></p>
  <p>We can &quot;pop off&quot; the first item:</p>
  <p><span class="math-container">$$G_t = r + \sum_{k=1}^{\infty} \gamma^k r$$</span></p>
  <p>Then we can note that the second term is just <span class="math-container">$\gamma$</span> times the orginal term:</p>
  <p><span class="math-container">$$G_t = r + \gamma G_t$$</span></p>
  <p>(There are situations where this won't work, such as when <span class="math-container">$\gamma \ge 1$</span> - essentially we are taking advantage that the high order terms are arbitrarily close to zero, so can be ignored)</p>
  <p>Re-arrange it again:</p>
  <p><span class="math-container">$$G_t = \frac{r}{1 - \gamma}$$</span></p>
  <p>This means that you can get the value for a return for the discounted sum of repeating rewards. Which allows you to calculate your <span class="math-container">$G_1$</span>. I will leave that last part as an execrise for you, as you already figured the first part out.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-29T14:44:20.283Z
Answers.LastActivityDate: 2020-06-29T14:44:20.283Z
Answers.UserDisplayName: Neil Slater
Body: <p>I know that <span class="math-container">$G_t = R_{t+1} + G_{t+1}$</span>.</p>
  <p>Suppose <span class="math-container">$\gamma = 0.9$</span> and the reward sequence is <span class="math-container">$R_1 = 2$</span> followed by an infinite sequence of <span class="math-container">$7$</span>s. What is the value of <span class="math-container">$G_0$</span>?</p>
  <p>As it's infinite, how can we deduce the value of <span class="math-container">$G_0$</span>? I don't see the solution. It's just <span class="math-container">$G_0 = 5 + 0.9*G_1$</span>. And we don't know <span class="math-container">$G_1$</span> value, and we don't know <span class="math-container">$R_2, R_3, R_4, ...$</span></p>
  
Tags: <reinforcement-learning><markov-decision-process><rewards><return><discount-factor>
Title: How do I calculate the return given the discount factor and a sequence of rewards?
UserDisplayName: Thom
ThreadText: # How do I calculate the return given the discount factor and a sequence of rewards?
  
  (Asked by: username_0 on 2020-06-29)
  
  I know that $G\_t = R\_{t+1} + G\_{t+1}$.
  
  Suppose $\gamma = 0.9$ and the reward sequence is $R\_1 = 2$ followed by an infinite sequence of $7$s. What is the value of $G\_0$?
  
  As it's infinite, how can we deduce the value of $G\_0$? I don't see the solution. It's just $G\_0 = 5 + 0.9\*G\_1$. And we don't know $G\_1$ value, and we don't know $R\_2, R\_3, R\_4, ...$
  
  **Tags:** reinforcement-learning, markov-decision-process, rewards, return, discount-factor
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-06-29. Score: 2
  
  You know all the rewards. They're 5, 7, 7, 7, and 7s forever. The problem now boils down to essentially a geometric series computation.
  
  $$
  G\_0 = R\_0 + \gamma G\_1
  $$
  
  $$
  G\_0 = 5 + \gamma\sum\_{k=0}^\infty 7\gamma^k
  $$
  
  $$
  G\_0 = 5 + 7\gamma\sum\_{k=0}^\infty\gamma^k
  $$
  
  $$
  G\_0 = 5 + \frac{7\gamma}{1-\gamma} = \frac{5 + 2\gamma}{1-\gamma}
  $$
  
  
  ---
  
  ### Answer by username_2 on 2020-06-29. Score: 1
  
  There are a few ways to resolve values of infinite sums. In this case, we can use a simple technique of self-reference to create a solvable equation.
  
  I will show how to do it for the generic case here of an MDP with same reward $r$ on each timestep:
  
  $$G\_t = \sum\_{k=0}^{\infty} \gamma^k r$$
  
  We can "pop off" the first item:
  
  $$G\_t = r + \sum\_{k=1}^{\infty} \gamma^k r$$
  
  Then we can note that the second term is just $\gamma$ times the orginal term:
  
  $$G\_t = r + \gamma G\_t$$
  
  (There are situations where this won't work, such as when $\gamma \ge 1$ - essentially we are taking advantage that the high order terms are arbitrarily close to zero, so can be ignored)
  
  Re-arrange it again:
  
  $$G\_t = \frac{r}{1 - \gamma}$$
  
  This means that you can get the value for a return for the discounted sum of repeating rewards. Which allows you to calculate your $G\_1$. I will leave that last part as an execrise for you, as you already figured the first part out.
  
  
  ---
  
  