Answers.Body: <p>It wouldn't make sense to define the return as you propose, from time 0 to <span class="math-container">$t$</span>. Once we are in a state at time <span class="math-container">$t$</span> we don't care what the returns <em>have been</em>, rather what they will be in the future, thus returns are defined as the total sum of discounted rewards from the current time step onwards. This allows the agent to make decisions about which actions to take based on how valuable taking said action is in the current state at time <span class="math-container">$t$</span> -- clearly the rewards previous to this have no effect upon that.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-30T23:38:43.333Z
Answers.Text: You say "Once we are in state $t$", but the return does not necessarily depend on any state. I think you're referring to the definition of the state-action (or state) value function, which is the expected return for taking action $a$ in state $s$ and then following a certain policy, so you should clarify that (although the OP accepted this answer, so I guess he's happy with it).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-01T00:52:09.993Z
Answers.Text: I meant to say a state. Once we are in state $s$ at time $t$ then we are interested in future returns from this time step onwards, regardless of what has happened.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-01T01:11:31.287Z
Answers.Text: Yes, we want to predict the future, but maybe the original doubt arouse because the OP was thinking something like "Shouldn't we also use the past to predict the future?".
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-30T23:35:40.440Z
Answers.LastActivityDate: 2020-12-01T00:50:05.607Z
Answers.LastEditDate: 2020-12-01T00:50:05.607Z
Answers.UserDisplayName: David
Body: <p>Why is it useful to define the return as the sum of the rewards from time <span class="math-container">$t$</span> onward rather than up to <span class="math-container">$t$</span>?</p>
  <p>The return for an MDP is usually defined as</p>
  <p><span class="math-container">$$G_t=R_{t+1}+R_{t+2}+ \dots +R_T$$</span></p>
  <p>Why is this defined as the return? Is there anything useful about this?</p>
  <p>It seems like it's more useful to define the return as <span class="math-container">$$G_t=R_0+ \dots+R_t,$$</span> because your &quot;return&quot;, so to speak, is the &quot;profit from investment&quot; so it seems like your return will be your accumulated reward from taking actions up to that point.</p>
  
Tags: <reinforcement-learning><definitions><markov-decision-process><return>
Title: Why is it useful to define the return as the sum of the rewards from time $t$ onward rather than up to $t$?
UserDisplayName: user8714896
ThreadText: # Why is it useful to define the return as the sum of the rewards from time $t$ onward rather than up to $t$?
  
  (Asked by: username_0 on 2020-11-30)
  
  Why is it useful to define the return as the sum of the rewards from time $t$ onward rather than up to $t$?
  
  The return for an MDP is usually defined as
  
  $$G\_t=R\_{t+1}+R\_{t+2}+ \dots +R\_T$$
  
  Why is this defined as the return? Is there anything useful about this?
  
  It seems like it's more useful to define the return as $$G\_t=R\_0+ \dots+R\_t,$$ because your "return", so to speak, is the "profit from investment" so it seems like your return will be your accumulated reward from taking actions up to that point.
  
  **Tags:** reinforcement-learning, definitions, markov-decision-process, return
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-30. Score: 3 | Accepted answer
  
  It wouldn't make sense to define the return as you propose, from time 0 to $t$. Once we are in a state at time $t$ we don't care what the returns *have been*, rather what they will be in the future, thus returns are defined as the total sum of discounted rewards from the current time step onwards. This allows the agent to make decisions about which actions to take based on how valuable taking said action is in the current state at time $t$ -- clearly the rewards previous to this have no effect upon that.
  
  > **Comments:**
  > **username_2**: You say "Once we are in state $t$", but the return does not necessarily depend on any state. I think you're referring to the definition of the state-action (or state) value function, which is the expected return for taking action $a$ in state $s$ and then following a certain policy, so you should clarify that (although the OP accepted this answer, so I guess he's happy with it).  
  > **username_1**: I meant to say a state. Once we are in state $s$ at time $t$ then we are interested in future returns from this time step onwards, regardless of what has happened.  
  > **username_2**: Yes, we want to predict the future, but maybe the original doubt arouse because the OP was thinking something like "Shouldn't we also use the past to predict the future?".  
  
  ---
  
  