Answers.Body: <blockquote>
    <p>shouldn't the expected return be calculated for some faraway time in the future (+) instead of current time <span class="math-container">$t$</span>?</p>
  </blockquote>
  
  <p>This is partly a notation issue, but <span class="math-container">$G_t$</span> is <em>already</em> the future sum of rewards as seen by the first (and correct) equation in your question. You don't actually know the value of any individual return <span class="math-container">$g_t$</span>* until after <span class="math-container">$t+n$</span>. However, you can predict the expected value <span class="math-container">$\mathbb{E}[G_t]$</span> provided the environment and policy remain consistent.</p>
  
  <p>Your second equation is pretty similar to the first one, which is why I say it is partly a notation issue. However, the point of calculating the forward-looking expectation for the return is to allow for assessment at time <span class="math-container">$t$</span>. This is to either predict likely outcome for future rewards knowing the system is in state <span class="math-container">$s_t$</span>, or in control scenarios to choose an action <span class="math-container">$a_t$</span>.</p>
  
  <p>In addition, in both prediction and control scenarios, the past (from time steps <span class="math-container">$0$</span> to <span class="math-container">$t-1$</span>) has already happened. Measurements of how well a system did in terms of gathering reward, looking backwards, might be useful metrics for e.g. "how good is this agent?". However, they are not in general a guide to the future. In many sparse environments (e.g. a board game scoring +1 for a win), this data is basically useless for predicting the future and all you want to know is summarised by the current state and the acting policy.</p>
  
  <p>Regardless of when you get to finally calculate a return, the start time step - the part of the trajectory that the return is calculated <em>from</em>, is a key parameter. The end time step is a practical concern for implementation, but can often be considered to be at infinity for theoretical purposes (i.e. we are interested in measuring or optimising all future rewards). So if you are only going to display/use one parameter in the notation, the start time <span class="math-container">$t$</span> is the one to use.</p>
  
  <p>There are variants of notation, to show how the return is calculated, where the calculation <em>horizon</em> is made explicit, e.g. <span class="math-container">$G_{t:t+n}$</span> for a <em>truncated</em> return or <span class="math-container">$G_{t:t+1}$</span> when calculating a one step temporal difference target. All the ones I have seen still maintain the forward view that defines a value associated with a current timestep <span class="math-container">$t$</span>, for the same reason as explained above - it is at timestep <span class="math-container">$t$</span> where this value is of most interest as a prediction. </p>
  
  <p>In practice during training you often wait until <span class="math-container">$t+n$</span> before you know the correct value of <span class="math-container">$g_t$</span>* to apply as a training value - which is then used to update value estimates for <span class="math-container">$\hat{v}(s_t)$</span> or <span class="math-container">$\hat{q}(s_t, a_t)$</span>. It is possible to make partial updates before that ending time step using techniques such as <a href="https://towardsdatascience.com/eligibility-traces-in-reinforcement-learning-a6b458c019d6" rel="nofollow noreferrer">eligibility traces</a>.</p>
  
  <hr>
  
  <p>* Using notation of uppercase <span class="math-container">$G$</span> for random variable, and lowercase <span class="math-container">$g$</span> for a measured value.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-03T08:08:52.047Z
Answers.LastActivityDate: 2020-05-04T07:15:07.057Z
Answers.LastEditDate: 2020-05-04T07:15:07.057Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am learning RL for the first time. It may be naive, but it is a bit odd to grasp this idea that, if the goal of RL is to maximize the expected return, then shouldn't the expected return be calculated for some faraway time in the future (<span class="math-container">$t+n$</span>) instead of current time <span class="math-container">$t$</span>? It is because we are building our system for the future using current information. ( I am coming from machine learning background and this makes more sense to me).</p>
  
  <p>Generally, the expected return is:
  <span class="math-container">$$\mathbb{E}[G_t] = \mathbb{E}[ R_{t+1} + R_{t+2} + R_{t+3}+... R_{t+n}]$$</span></p>
  
  <p>However, shouldn't the expected return be:
  <span class="math-container">$$\mathbb{E}[G_{t+n}] = \mathbb{E}[R_{t+1} + R_{t+2} + R_{t+3}+... R_{t+n-1}]$$</span></p>
  
Tags: <reinforcement-learning><rewards><expectation><return>
Title: Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$?
UserDisplayName: SJa
ThreadText: # Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$?
  
  (Asked by: username_0 on 2020-05-03)
  
  I am learning RL for the first time. It may be naive, but it is a bit odd to grasp this idea that, if the goal of RL is to maximize the expected return, then shouldn't the expected return be calculated for some faraway time in the future ($t+n$) instead of current time $t$? It is because we are building our system for the future using current information. ( I am coming from machine learning background and this makes more sense to me).
  
  Generally, the expected return is:
  $$\mathbb{E}[G\_t] = \mathbb{E}[ R\_{t+1} + R\_{t+2} + R\_{t+3}+... R\_{t+n}]$$
  
  However, shouldn't the expected return be:
  $$\mathbb{E}[G\_{t+n}] = \mathbb{E}[R\_{t+1} + R\_{t+2} + R\_{t+3}+... R\_{t+n-1}]$$
  
  **Tags:** reinforcement-learning, rewards, expectation, return
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-03. Score: 1 | Accepted answer
  
  > shouldn't the expected return be calculated for some faraway time in the future (+) instead of current time $t$?
  
  This is partly a notation issue, but $G\_t$ is *already* the future sum of rewards as seen by the first (and correct) equation in your question. You don't actually know the value of any individual return $g\_t$\* until after $t+n$. However, you can predict the expected value $\mathbb{E}[G\_t]$ provided the environment and policy remain consistent.
  
  Your second equation is pretty similar to the first one, which is why I say it is partly a notation issue. However, the point of calculating the forward-looking expectation for the return is to allow for assessment at time $t$. This is to either predict likely outcome for future rewards knowing the system is in state $s\_t$, or in control scenarios to choose an action $a\_t$.
  
  In addition, in both prediction and control scenarios, the past (from time steps $0$ to $t-1$) has already happened. Measurements of how well a system did in terms of gathering reward, looking backwards, might be useful metrics for e.g. "how good is this agent?". However, they are not in general a guide to the future. In many sparse environments (e.g. a board game scoring +1 for a win), this data is basically useless for predicting the future and all you want to know is summarised by the current state and the acting policy.
  
  Regardless of when you get to finally calculate a return, the start time step - the part of the trajectory that the return is calculated *from*, is a key parameter. The end time step is a practical concern for implementation, but can often be considered to be at infinity for theoretical purposes (i.e. we are interested in measuring or optimising all future rewards). So if you are only going to display/use one parameter in the notation, the start time $t$ is the one to use.
  
  There are variants of notation, to show how the return is calculated, where the calculation *horizon* is made explicit, e.g. $G\_{t:t+n}$ for a *truncated* return or $G\_{t:t+1}$ when calculating a one step temporal difference target. All the ones I have seen still maintain the forward view that defines a value associated with a current timestep $t$, for the same reason as explained above - it is at timestep $t$ where this value is of most interest as a prediction.
  
  In practice during training you often wait until $t+n$ before you know the correct value of $g\_t$\* to apply as a training value - which is then used to update value estimates for $\hat{v}(s\_t)$ or $\hat{q}(s\_t, a\_t)$. It is possible to make partial updates before that ending time step using techniques such as [eligibility traces](https://towardsdatascience.com/eligibility-traces-in-reinforcement-learning-a6b458c019d6).
  
  ---
  
  \* Using notation of uppercase $G$ for random variable, and lowercase $g$ for a measured value.
  
  
  ---
  
  