Answers.Body: <p><span class="math-container">$T = \infty$</span> and <span class="math-container">$\gamma = 1$</span> cannot be both true at the same time because the return defined in equation 3.11 is supposed to be a <em>unified</em> definition of the return for <em>both</em> continuing and episodic tasks. In the case of continuing tasks, <span class="math-container">$T = \infty$</span> and <span class="math-container">$\gamma = 1$</span> cannot be true at the same time, because the return may not be finite in that case (as I think you already understood).</p>
  <p>Moreover, note that, in that <em>specific</em> example of the book, they assume that the agent ends up in an absorbing state, so this specific sum is finite, no matter whether <span class="math-container">$T$</span> is finite or <span class="math-container">$\infty$</span>, given that, once you enter the absorbing state, you will always get a reward of <span class="math-container">$0$</span>. Of course, if you discount those specific rewards, the sum will still be finite. However, in general, if you had a different MDP where the absorbing state is not reachable (i.e. the episode never ends), then the return could not be finite.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-01T01:42:14.213Z
Answers.LastActivityDate: 2020-12-01T01:42:14.213Z
Answers.UserDisplayName: nbro
Body: <p>For episodic tasks with an absorbing state, why can't <span class="math-container">$\gamma=1$</span> and <span class="math-container">$T= \infty$</span>?</p>
  <p>In Sutton and Barto's book, they say that, for episodic tasks with absorbing states that becomes an infinite sequence, then the return  is defined by:</p>
  <p><span class="math-container">$$G_t=\sum_{k=t+1}^{T}\gamma^{k-t-1}R_k$$</span></p>
  <p>This allows the return to be the same whether the sum is over the first <span class="math-container">$T$</span> rewards, where <span class="math-container">$T$</span> is the time of termination or over the full infinite sequence, with <span class="math-container">$T=\infty$</span> xor <span class="math-container">$\gamma=1$</span>.</p>
  <p>Why can't we have both? I don't see how they can both be set to those parameters. It seems like, if you have an absorbing state, the rewards from terminal onward will just be 0 and not be affected by <span class="math-container">$\gamma$</span> or <span class="math-container">$T$</span>.</p>
  <p>Here's the full section of the book on page 57 in the 2nd edition</p>
  <p><a href="https://i.sstatic.net/cI2ih.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/cI2ih.png" alt="enter image description here" /></a></p>
  <p>I think the reasoning behind this also leads to why for policy evaluation where</p>
  <p><span class="math-container">$$v_\pi(s)=\sum_a\pi(a|s)\sum_{s',r}p(s',r|s,a)[r+\gamma v_\pi(s')]$$</span></p>
  <p>&quot;Has an existence and uniqueness guarantee only if <span class="math-container">$\gamma &lt; 1$</span> or termination is guaranteed under <span class="math-container">$\pi$</span>&quot;(page 74). This part I'm also a bit confused by, but seems related.</p>
  
Tags: <reinforcement-learning><hyper-parameters><return><discount-factor>
Title: For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?
UserDisplayName: user8714896
ThreadText: # For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?
  
  (Asked by: username_0 on 2020-11-30)
  
  For episodic tasks with an absorbing state, why can't $\gamma=1$ and $T= \infty$?
  
  In Sutton and Barto's book, they say that, for episodic tasks with absorbing states that becomes an infinite sequence, then the return is defined by:
  
  $$G\_t=\sum\_{k=t+1}^{T}\gamma^{k-t-1}R\_k$$
  
  This allows the return to be the same whether the sum is over the first $T$ rewards, where $T$ is the time of termination or over the full infinite sequence, with $T=\infty$ xor $\gamma=1$.
  
  Why can't we have both? I don't see how they can both be set to those parameters. It seems like, if you have an absorbing state, the rewards from terminal onward will just be 0 and not be affected by $\gamma$ or $T$.
  
  Here's the full section of the book on page 57 in the 2nd edition
  
  [![enter image description here](https://i.sstatic.net/cI2ih.png)](https://i.sstatic.net/cI2ih.png)
  
  I think the reasoning behind this also leads to why for policy evaluation where
  
  $$v\_\pi(s)=\sum\_a\pi(a|s)\sum\_{s',r}p(s',r|s,a)[r+\gamma v\_\pi(s')]$$
  
  "Has an existence and uniqueness guarantee only if $\gamma < 1$ or termination is guaranteed under $\pi$"(page 74). This part I'm also a bit confused by, but seems related.
  
  **Tags:** reinforcement-learning, hyper-parameters, return, discount-factor
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-01. Score: 4 | Accepted answer
  
  $T = \infty$ and $\gamma = 1$ cannot be both true at the same time because the return defined in equation 3.11 is supposed to be a *unified* definition of the return for *both* continuing and episodic tasks. In the case of continuing tasks, $T = \infty$ and $\gamma = 1$ cannot be true at the same time, because the return may not be finite in that case (as I think you already understood).
  
  Moreover, note that, in that *specific* example of the book, they assume that the agent ends up in an absorbing state, so this specific sum is finite, no matter whether $T$ is finite or $\infty$, given that, once you enter the absorbing state, you will always get a reward of $0$. Of course, if you discount those specific rewards, the sum will still be finite. However, in general, if you had a different MDP where the absorbing state is not reachable (i.e. the episode never ends), then the return could not be finite.
  
  
  ---
  
  