Answers.Body: <p><strong>Short answer:</strong> I the <em>return-to-go</em> is just a <strong>synonym</strong> of the <em>reward-to-go</em>, which instead is a well known term used to define the basic form of <em>policy gradient</em>, e.g. REINFORCE-like.</p>
  <h3>Reward-to-go</h3>
  <p>As you correctly wrote, the reward-to-go is just a sum of rewards collected in a trajectory <span class="math-container">$\tau$</span> from a certain timestep <span class="math-container">$t$</span> until termination <span class="math-container">$T$</span>. If we define <span class="math-container">$R(\tau)=\sum_{t=1}^T r_t^{\tau}$</span> as the sum of immediate rewards <span class="math-container">$r_t$</span> got in trajectory <span class="math-container">$\tau$</span>, from that we can define the rewards-to-go just by considering a subset of the trajectory, so discarding the past, i.e starting from a time <span class="math-container">$t$</span>:
  <span class="math-container">$$R_t(\tau)=\sum_{t'=t}r_{t'}^\tau$$</span></p>
  <p>In particular, the quantity <span class="math-container">$R_t(\tau)$</span> figures in the (basic) policy gradient formulation:
  <span class="math-container">$$\nabla_\theta J(\theta)\approx \sum_i\Big(\sum_{t=1}^T \nabla_\theta\log \pi_\theta\big(a_t^{(i)}\mid s_t^{(i)}\big) R_t\big(\tau^{(i)}\big)  \Big)$$</span></p>
  <p>Basically, the rewards-to-go score actions within a trajectory <span class="math-container">$\tau^{(i)}$</span>: the <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#don-t-let-the-past-distract-you" rel="nofollow noreferrer">spinning-up rl article</a> you mentioned refers to the fact that the original (although naive) PG formulation uses <span class="math-container">$R(\tau)$</span>, i.e. sum of rewards, to score the policy's actions but it's useless to consider the immediate rewards <span class="math-container">$r_k$</span> gained in the past (so when <span class="math-container">$k&lt;t$</span>) to score actions happened at time <span class="math-container">$t$</span> and onward until the end of the trajectory <span class="math-container">$T$</span>. In this regard, both <span class="math-container">$R(\tau)$</span> and <span class="math-container">$R_t(\tau)$</span> are empirical estimates of the value of an action <span class="math-container">$a_t$</span> from a given state <span class="math-container">$s_t$</span> - think of it as a single-sample estimate of the action-value function, <span class="math-container">$Q(s_t,a_t)\approx R_t$</span> - but <span class="math-container">$R(\tau)$</span> has a higher variance due the additional sums.</p>
  <p>This is reasonable, since in more recent PG formulations we subtract a <em>baseline function</em> <span class="math-container">$b(s)$</span> to the rewards-to-go to reduce the variance of the gradient estimate: <span class="math-container">$\nabla_\theta\log\pi_\theta(a_t\mid s_t)(R_t - b(s_t))$</span>. If <span class="math-container">$b(s)=V(s)$</span> is chosen to be the value function, then <span class="math-container">$R_t - b(s_t)$</span> is an estimate of the advantage function <span class="math-container">$A(s_t, a_t) = Q(s_t,a_t) - V(s_t)$</span> since <span class="math-container">$R_t$</span> is the action-value gathered in the current trajectory (but not <em>expected</em>, as in the Q-function.) Formulations that use <span class="math-container">$A(s,a)$</span> yield the gradient with lowest variance: A2C uses this.</p>
  <blockquote>
  <p>In which situations do we need to care about it?</p>
  </blockquote>
  <p>We talk about rewards-to-go mainly when dealing with the PG estimation, and defining new baseline functions to improve the gradient estimate: otherwise the term return is largely preferred. Note: it is demonstrated that subtracting a baseline does not alter the gradient formulation, and if correctly picked it does not either add bias to the estimate.</p>
  <blockquote>
  <p>the reward-to-go is analyzed in the paper which states that the <em>expected</em> reward-to-go is the value function</p>
  </blockquote>
  <p>The rewards-to-go are more appropriate as action-value estimates rather than state-value estimates (i.e. the return <span class="math-container">$G_t$</span>) since we start from a state-action pair. This is a technicality that <em>does not hold</em> in practice, since the value function is also estimated with empirical returns (i.e. the return of a single trajectory <span class="math-container">$\tau$</span>), i.e. <span class="math-container">$G_t(\tau) = r_t^\tau + r_{t+1}^\tau +\cdots + r_T^\tau$</span> that correspond to the rewards-to-go <span class="math-container">$R_t(\tau)$</span>: think about learning a value function in deep RL, we just fit the neural-net to empirical returns than MSE + GD does the magic. Now, if we average the collected rewards or returns over all possible trajectories we obtain the <em>expected return</em> that is the value function: <span class="math-container">$V(s_t) = \mathbb{E}_\tau[G_t(\tau)] = \mathbb{E}_\tau[R_t(\tau)] = \mathbb{E}_\tau[r_t^\tau + r_{t+1}^\tau + \cdots + r_{T}^\tau\mid s_t]$</span>.</p>
  <blockquote>
  <p>How is the return-to-go related to the reward-to-go?</p>
  </blockquote>
  <p>They are same thing. I'd say to avoid &quot;return-to-go&quot; and use return, instead (or at most reward-to-go when talking about PG.)</p>
  <hr />
  <p>As a side note: none of both terms are mentioned in the famous book &quot;An introduction to reinforcement learning&quot; by Sutton and Barto, so I guess it's something introduced in modern deep RL.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-27T20:41:36.390Z
Answers.LastActivityDate: 2023-04-27T20:41:36.390Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>In reinforcement learning, the <em>return</em> is defined as some function of the rewards. For example, you can have the <em>discounted return</em>, where you multiply the rewards received at later time steps by increasingly smaller numbers, so that the rewards closer to the current time step have a higher weight. You can also have <span class="math-container">$n$</span>-step returns or <span class="math-container">$\lambda$</span>-returns.</p>
  <p>Recently, I have come across the concept of <strong>return-to-go</strong> in a few research papers, such as <a href="https://arxiv.org/pdf/1511.05952.pdf" rel="noreferrer">Prioritized Experience Replay</a> (appendix <strong>A. Prioritization Variants</strong>, p. 12) or <a href="https://arxiv.org/pdf/1911.01546.pdf" rel="noreferrer">Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy</a> (section <strong>Theoretical Analysis</strong>, p. 3).</p>
  <p>What exactly is the <em>return-to-go</em>? How is it mathematically defined? In which situations do we need to care about it? The name suggests that this is the return starting from a certain time step <span class="math-container">$t$</span>, but wouldn't this be the same thing as the <em>return</em> (which is defined starting from a certain time step <span class="math-container">$t$</span> and often denoted as <span class="math-container">$G_t$</span> for that same reason)?</p>
  <p>There is also the concept of <strong>reward-to-go</strong>. For example, the reward-to-go is analyzed in the paper <a href="https://www.jmlr.org/papers/volume17/14-335/14-335.pdf" rel="noreferrer">Learning the Variance of the Reward-To-Go</a>, which states that the <em>expected reward-to-go</em> is the value function, which seems to be consistent with <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#don-t-let-the-past-distract-you" rel="noreferrer">this explanation of the reward-to-go</a>, where the reward-to-go is defined as</p>
  <p><span class="math-container">$$\hat{R}_{t} \doteq \sum_{t^{\prime}=t}^{T} R\left(s_{t^{\prime}}, a_{t^{\prime}}, s_{t^{\prime}+1}\right)$$</span></p>
  <p>We also had a few questions that involve the reward-to-go: for example, <a href="https://ai.stackexchange.com/q/21784/2444">this</a> or <a href="https://ai.stackexchange.com/q/9614/2444">this</a>. How is the return-to-go related to the reward-to-go? Are they the same thing? For example, in <a href="https://arxiv.org/pdf/2005.02305.pdf" rel="noreferrer">this paper</a>, the return-to-go seems to be used as a synonym for reward-to-go (as used in <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#don-t-let-the-past-distract-you" rel="noreferrer">this article</a>), i.e. they call <span class="math-container">$R(t)$</span> the &quot;return to-go&quot; (e.g. on page 2), which should be the return starting from time step <span class="math-container">$t$</span>, which should actually be the reward-to-go.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-10T16:00:28.093Z
Comments.Text: "return-to-go" is probably not standard terminology, but I think that some people will also have this question (if they come across the same papers), that's why I asked it (i.e. to serve as a future reference).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-27T17:18:14.260Z
Comments.Text: so have you figured it out?
Comments.UserDisplayName: Sam
Tags: <reinforcement-learning><comparison><definitions><reward-to-go>
Title: What is the return-to-go in reinforcement learning?
UserDisplayName: nbro
ThreadText: # What is the return-to-go in reinforcement learning?
  
  (Asked by: username_0 on 2020-10-10)
  
  In reinforcement learning, the *return* is defined as some function of the rewards. For example, you can have the *discounted return*, where you multiply the rewards received at later time steps by increasingly smaller numbers, so that the rewards closer to the current time step have a higher weight. You can also have $n$-step returns or $\lambda$-returns.
  
  Recently, I have come across the concept of **return-to-go** in a few research papers, such as [Prioritized Experience Replay](https://arxiv.org/pdf/1511.05952.pdf) (appendix **A. Prioritization Variants**, p. 12) or [Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy](https://arxiv.org/pdf/1911.01546.pdf) (section **Theoretical Analysis**, p. 3).
  
  What exactly is the *return-to-go*? How is it mathematically defined? In which situations do we need to care about it? The name suggests that this is the return starting from a certain time step $t$, but wouldn't this be the same thing as the *return* (which is defined starting from a certain time step $t$ and often denoted as $G\_t$ for that same reason)?
  
  There is also the concept of **reward-to-go**. For example, the reward-to-go is analyzed in the paper [Learning the Variance of the Reward-To-Go](https://www.jmlr.org/papers/volume17/14-335/14-335.pdf), which states that the *expected reward-to-go* is the value function, which seems to be consistent with [this explanation of the reward-to-go](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#don-t-let-the-past-distract-you), where the reward-to-go is defined as
  
  $$\hat{R}\_{t} \doteq \sum\_{t^{\prime}=t}^{T} R\left(s\_{t^{\prime}}, a\_{t^{\prime}}, s\_{t^{\prime}+1}\right)$$
  
  We also had a few questions that involve the reward-to-go: for example, [this](https://ai.stackexchange.com/q/21784/2444) or [this](https://ai.stackexchange.com/q/9614/2444). How is the return-to-go related to the reward-to-go? Are they the same thing? For example, in [this paper](https://arxiv.org/pdf/2005.02305.pdf), the return-to-go seems to be used as a synonym for reward-to-go (as used in [this article](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#don-t-let-the-past-distract-you)), i.e. they call $R(t)$ the "return to-go" (e.g. on page 2), which should be the return starting from time step $t$, which should actually be the reward-to-go.
  
  **Tags:** reinforcement-learning, comparison, definitions, reward-to-go
  
  > **Comments:**
  > **username_0**: "return-to-go" is probably not standard terminology, but I think that some people will also have this question (if they come across the same papers), that's why I asked it (i.e. to serve as a future reference).  
  > **username_1**: so have you figured it out?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-04-27. Score: 1
  
  **Short answer:** I the *return-to-go* is just a **synonym** of the *reward-to-go*, which instead is a well known term used to define the basic form of *policy gradient*, e.g. REINFORCE-like.
  
  ### Reward-to-go
  
  As you correctly wrote, the reward-to-go is just a sum of rewards collected in a trajectory $\tau$ from a certain timestep $t$ until termination $T$. If we define $R(\tau)=\sum\_{t=1}^T r\_t^{\tau}$ as the sum of immediate rewards $r\_t$ got in trajectory $\tau$, from that we can define the rewards-to-go just by considering a subset of the trajectory, so discarding the past, i.e starting from a time $t$:
  $$R\_t(\tau)=\sum\_{t'=t}r\_{t'}^\tau$$
  
  In particular, the quantity $R\_t(\tau)$ figures in the (basic) policy gradient formulation:
  $$\nabla\_\theta J(\theta)\approx \sum\_i\Big(\sum\_{t=1}^T \nabla\_\theta\log \pi\_\theta\big(a\_t^{(i)}\mid s\_t^{(i)}\big) R\_t\big(\tau^{(i)}\big) \Big)$$
  
  Basically, the rewards-to-go score actions within a trajectory $\tau^{(i)}$: the [spinning-up rl article](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#don-t-let-the-past-distract-you) you mentioned refers to the fact that the original (although naive) PG formulation uses $R(\tau)$, i.e. sum of rewards, to score the policy's actions but it's useless to consider the immediate rewards $r\_k$ gained in the past (so when $k<t$) to score actions happened at time $t$ and onward until the end of the trajectory $T$. In this regard, both $R(\tau)$ and $R\_t(\tau)$ are empirical estimates of the value of an action $a\_t$ from a given state $s\_t$ - think of it as a single-sample estimate of the action-value function, $Q(s\_t,a\_t)\approx R\_t$ - but $R(\tau)$ has a higher variance due the additional sums.
  
  This is reasonable, since in more recent PG formulations we subtract a *baseline function* $b(s)$ to the rewards-to-go to reduce the variance of the gradient estimate: $\nabla\_\theta\log\pi\_\theta(a\_t\mid s\_t)(R\_t - b(s\_t))$. If $b(s)=V(s)$ is chosen to be the value function, then $R\_t - b(s\_t)$ is an estimate of the advantage function $A(s\_t, a\_t) = Q(s\_t,a\_t) - V(s\_t)$ since $R\_t$ is the action-value gathered in the current trajectory (but not *expected*, as in the Q-function.) Formulations that use $A(s,a)$ yield the gradient with lowest variance: A2C uses this.
  
  > In which situations do we need to care about it?
  
  We talk about rewards-to-go mainly when dealing with the PG estimation, and defining new baseline functions to improve the gradient estimate: otherwise the term return is largely preferred. Note: it is demonstrated that subtracting a baseline does not alter the gradient formulation, and if correctly picked it does not either add bias to the estimate.
  
  > the reward-to-go is analyzed in the paper which states that the *expected* reward-to-go is the value function
  
  The rewards-to-go are more appropriate as action-value estimates rather than state-value estimates (i.e. the return $G\_t$) since we start from a state-action pair. This is a technicality that *does not hold* in practice, since the value function is also estimated with empirical returns (i.e. the return of a single trajectory $\tau$), i.e. $G\_t(\tau) = r\_t^\tau + r\_{t+1}^\tau +\cdots + r\_T^\tau$ that correspond to the rewards-to-go $R\_t(\tau)$: think about learning a value function in deep RL, we just fit the neural-net to empirical returns than MSE + GD does the magic. Now, if we average the collected rewards or returns over all possible trajectories we obtain the *expected return* that is the value function: $V(s\_t) = \mathbb{E}\_\tau[G\_t(\tau)] = \mathbb{E}\_\tau[R\_t(\tau)] = \mathbb{E}\_\tau[r\_t^\tau + r\_{t+1}^\tau + \cdots + r\_{T}^\tau\mid s\_t]$.
  
  > How is the return-to-go related to the reward-to-go?
  
  They are same thing. I'd say to avoid "return-to-go" and use return, instead (or at most reward-to-go when talking about PG.)
  
  ---
  
  As a side note: none of both terms are mentioned in the famous book "An introduction to reinforcement learning" by Sutton and Barto, so I guess it's something introduced in modern deep RL.
  
  
  ---
  
  