Answers.Body: <p>Formally, the <strong>return</strong> (also known as the <em>cumulative future discounted reward</em>) can be defined as</p>
  <p><span class="math-container">$$
  G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1},
  $$</span></p>
  <p>where <span class="math-container">$0 \leq \gamma \leq 1$</span> is the discount factor and <span class="math-container">$R_{i}$</span> is the reward at time step <span class="math-container">$i$</span>. Here <span class="math-container">$G_t$</span> and <span class="math-container">$R_i$</span> are considered random variables (and r.v.s are usually denoted with capital letters, so I am using the notation used in the book <a href="http://incompleteideas.net/book/bookdraft2017nov5.pdf" rel="nofollow noreferrer">Reinforcement Learning: An Introduction</a>, <strong>2nd edition</strong>).</p>
  <p>The <strong>expected return</strong> is defined as</p>
  <p><span class="math-container">\begin{align}
  v^\pi(s) 
  &amp;= \mathbb{E}\left[G_t \mid S_t = s \right] \\
  &amp;= \mathbb{E}\left[\sum_{k=0}^\infty \gamma^k R_{t+k+1} \bigm\vert S_t = s \right]
  \end{align}</span></p>
  <p>In other words, the value of a state <span class="math-container">$s$</span> (associated with a policy <span class="math-container">$\pi$</span>) is equal to the expectation of the return <span class="math-container">$G_t$</span> given that <span class="math-container">$S_t = s$</span>, so <span class="math-container">$v^\pi(s)$</span> is defined as a <a href="https://en.wikipedia.org/wiki/Conditional_expectation" rel="nofollow noreferrer">conditional expectation</a>. Note also that the expected value is usually defined with respect to a random variable, which is the case. Note also that <span class="math-container">$S_t$</span> is a random variable, while <span class="math-container">$s$</span> is a realization of this random variable.</p>
  <p>A policy is <strong>not</strong> a probability distribution of choosing the next state. A stochastic policy is a family of a conditional probability distribution <em>over actions given states</em>. There are also deterministic policies. Have a look at this question <a href="https://ai.stackexchange.com/q/12274/2444">What is the difference between a stochastic and a deterministic policy?</a> for more details about the definition of stochastic and deterministic policies.</p>
  <blockquote>
  <p>If an agent is following a policy, the agent wants to find the value at <span class="math-container">$S_{t}$</span> by calculating a sort of &quot;weighted average&quot; of all the returns from <span class="math-container">$S_{t}.$</span> This is called the expected return.</p>
  </blockquote>
  <p>In the case of Monte Carlo Prediction, the value of a state associated with a specific policy, that is, the expected value of the return given a state is approximated with a finite (weighted) average. See e.g. <a href="https://ai.stackexchange.com/a/10818/2444">What is the difference between First-Visit Monte-Carlo and Every-Visit Monte-Carlo Policy Evaluation?</a>. Furthermore, note that the <a href="https://en.wikipedia.org/wiki/Expected_value" rel="nofollow noreferrer">expectation</a> of a <em>discrete</em> random variable is defined as a weighted average.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-30T15:39:17.697Z
Answers.LastActivityDate: 2021-01-22T20:56:33.813Z
Answers.LastEditDate: 2021-01-22T20:56:33.813Z
Answers.UserDisplayName: nbro
Answers.Body: <p>You're correct, the return is the discounted future reward from the one iteration while the expected return is averaged over a bunch of iterations.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-12T17:04:15.520Z
Answers.LastActivityDate: 2019-09-12T17:04:15.520Z
Answers.UserDisplayName: lzl
Body: <p>At a time step <span class="math-container">$t$</span>, for a state <span class="math-container">$S_{t}$</span>, the return is defined as the discounted cumulative reward from that time step <span class="math-container">$t$</span>.</p>
  
  <p>If an agent is following a policy (which in itself is a probability distribution of choosing a next state <span class="math-container">$S_{t+1}$</span> from <span class="math-container">$S_{t}$</span>), the agent wants to find the value at <span class="math-container">$S_{t}$</span> by calculating sort of "weighted average" of all the returns from <span class="math-container">$S_{t}.$</span> This is called the expected return.</p>
  
  <p>Is my understanding correct?</p>
  
Tags: <reinforcement-learning><comparison><q-learning><return><expectation>
Title: What is the difference between return and expected return?
UserDisplayName: digi philos
ThreadText: # What is the difference between return and expected return?
  
  (Asked by: username_0 on 2019-06-30)
  
  At a time step $t$, for a state $S\_{t}$, the return is defined as the discounted cumulative reward from that time step $t$.
  
  If an agent is following a policy (which in itself is a probability distribution of choosing a next state $S\_{t+1}$ from $S\_{t}$), the agent wants to find the value at $S\_{t}$ by calculating sort of "weighted average" of all the returns from $S\_{t}.$ This is called the expected return.
  
  Is my understanding correct?
  
  **Tags:** reinforcement-learning, comparison, q-learning, return, expectation
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-06-30. Score: 1
  
  Formally, the **return** (also known as the *cumulative future discounted reward*) can be defined as
  
  $$
  G\_t = \sum\_{k=0}^\infty \gamma^k R\_{t+k+1},
  $$
  
  where $0 \leq \gamma \leq 1$ is the discount factor and $R\_{i}$ is the reward at time step $i$. Here $G\_t$ and $R\_i$ are considered random variables (and r.v.s are usually denoted with capital letters, so I am using the notation used in the book [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/bookdraft2017nov5.pdf), **2nd edition**).
  
  The **expected return** is defined as
  
  \begin{align}
  v^\pi(s)
  &= \mathbb{E}\left[G\_t \mid S\_t = s \right] \\
  &= \mathbb{E}\left[\sum\_{k=0}^\infty \gamma^k R\_{t+k+1} \bigm\vert S\_t = s \right]
  \end{align}
  
  In other words, the value of a state $s$ (associated with a policy $\pi$) is equal to the expectation of the return $G\_t$ given that $S\_t = s$, so $v^\pi(s)$ is defined as a [conditional expectation](https://en.wikipedia.org/wiki/Conditional_expectation). Note also that the expected value is usually defined with respect to a random variable, which is the case. Note also that $S\_t$ is a random variable, while $s$ is a realization of this random variable.
  
  A policy is **not** a probability distribution of choosing the next state. A stochastic policy is a family of a conditional probability distribution *over actions given states*. There are also deterministic policies. Have a look at this question [What is the difference between a stochastic and a deterministic policy?](https://ai.stackexchange.com/q/12274/2444) for more details about the definition of stochastic and deterministic policies.
  
  > If an agent is following a policy, the agent wants to find the value at $S\_{t}$ by calculating a sort of "weighted average" of all the returns from $S\_{t}.$ This is called the expected return.
  
  In the case of Monte Carlo Prediction, the value of a state associated with a specific policy, that is, the expected value of the return given a state is approximated with a finite (weighted) average. See e.g. [What is the difference between First-Visit Monte-Carlo and Every-Visit Monte-Carlo Policy Evaluation?](https://ai.stackexchange.com/a/10818/2444). Furthermore, note that the [expectation](https://en.wikipedia.org/wiki/Expected_value) of a *discrete* random variable is defined as a weighted average.
  
  
  ---
  
  ### Answer by username_2 on 2019-09-12. Score: 1
  
  You're correct, the return is the discounted future reward from the one iteration while the expected return is averaged over a bunch of iterations.
  
  
  ---
  
  