Answers.Body: <p>In addition to <a href="https://ai.stackexchange.com/a/19975/2444">this answer</a>, I would like to note that, if the future trajectories were fixed (i.e. the environment and the policies were deterministic, and the agent always starts from the same state), the expectation of the sum (of the fixed rewards) would simply correspond to the actual sum, because the sum is a constant (i.e. the expectation of a constant is the constant itself), so the expectation operator also applies to the deterministic cases. Therefore, the expectation is a general way of expressing the value of a state in all possible cases (both when trajectories are fixed or not).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-03T20:03:12.720Z
Answers.LastActivityDate: 2020-04-03T20:11:30.470Z
Answers.LastEditDate: 2020-04-03T20:11:30.470Z
Answers.UserDisplayName: nbro
Answers.Body: <p>There needs to be an <span class="math-container">$E_{\pi}$</span> over the infinite discounted return term because of two reasons-</p>
  
  <ol>
  <li>The policy could be stochastic in nature. That is, for any given state <span class="math-container">$s_t$</span> at time <span class="math-container">$t$</span>, the policy <span class="math-container">$\pi(s_t)$</span> does not provide a deterministic action <span class="math-container">$a$</span>, but rather, it provides us with a distribution over the possible next states, that is the action at time <span class="math-container">$t$</span>, <span class="math-container">$a_t$</span> is distributed as-
  <span class="math-container">$$a_t \sim \pi(s_t)$$</span></li>
  <li>Even if the policy <span class="math-container">$\pi$</span> being followed by an agent is deterministic, there still needs to be an expectation over the behavior of the underlying stochastic MDP environment. That is, any action <span class="math-container">$a_t$</span>, in general, only provides us with a distribution over the possible next states of the agent. That is, 
  <span class="math-container">$$P(s_{t + 1} = s') = P_{\pi}(s' | s_t) = \sum_{a \in A} T(s,a,s') \times P_{\pi}(a_t = a)$$</span> 
  Here <span class="math-container">$T(s, a, s')$</span> is the transition function for the MDP and the above equation captures the stochasticity arising from both 1 and 2.</li>
  </ol>
  
  <p>As you see the expectation does not have to do with averaging over a collection of trajectories. However, that idea is often used in <a href="https://en.wikipedia.org/wiki/Monte_Carlo_method" rel="nofollow noreferrer">Monte-Carlo</a> estimation of value functions. </p>
  
  <p>EDIT:
  As pointed out in the comments, it is not correct to say that the expectation is not over a collection of trajectories.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-04T00:50:07.440Z
Answers.Text: While your explanation is correct, your conclusion is not. The expectation does indeed correspond to an averaging over a collection of trajectories since you need to consider all the possible combinations of states and actions that lead to the return $G_t$ and weight them by how probable the combination is under the policy $\pi$ and the transition function $T$.
Answers.UserDisplayName: Diego Gomez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-04T04:34:41.263Z
Answers.Text: @DiegoGomez Have added an edit, thanks for pointing out.
Answers.UserDisplayName: ijuneja
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-06T16:29:41.687Z
Answers.Text: Thanks. So in the equation https://imgur.com/4g6fjIU the flashed symbols are the ones that denote the probabilities of the expectation formula?
Answers.UserDisplayName: Jack
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-06T16:30:53.803Z
Answers.Text: If so, why we only have $E_{\pi}$, shouldn't it be $E_{\pi,p}$? Because I suppose the $p$ of environment transitions is not dependent on policy $\pi$, is it?
Answers.UserDisplayName: Jack
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-06T16:34:58.560Z
Answers.Text: @Jack in the linked image, summation symbols have been highlighted. The inner summation is over the possible next-state, reward tuple and the outer one is over a stochastic policy, so yes all of them together are capturing the expectation if that is what you meant.
Answers.UserDisplayName: ijuneja
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-06T16:37:17.520Z
Answers.Text: @Jack yes I agree, perhaps to be rigorous, it should be made explicit that the expectation is over both $\pi$ and the transition distribution $p$.
Answers.UserDisplayName: ijuneja
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-03T19:19:47.907Z
Answers.LastActivityDate: 2020-04-04T04:34:14.487Z
Answers.LastEditDate: 2020-04-04T04:34:14.487Z
Answers.UserDisplayName: ijuneja
Body: <p>In chapter 3.5 of Sutton's book, the value function is defined as:</p>
  
  <p><a href="https://i.sstatic.net/eNqIS.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/eNqIS.png" alt="enter image description here"></a></p>
  
  <p>Can someone give me some clarification about why there is the expectation sign behind the entire equation? Considering that the agent is following a fixed policy <span class="math-container">$\pi$</span>, why there should be an expectation when the trajectory of the future possible states is fixed (or maybe I am getting it wrong and it's not). In total, if the expectation here has the meaning of averaging over a series of trajectories, what are those trajectories and what are the weights of them when we want to compute the expected value over them according to <a href="https://en.wikipedia.org/wiki/Expected_value" rel="nofollow noreferrer">this Wikipedia definition of the expected values</a>?</p>
  
Tags: <reinforcement-learning><value-functions><expectation><bellman-equations>
Title: Why is there an expectation sign in the Bellman equation?
UserDisplayName: Jack
ThreadText: # Why is there an expectation sign in the Bellman equation?
  
  (Asked by: username_0 on 2020-04-03)
  
  In chapter 3.5 of Sutton's book, the value function is defined as:
  
  [![enter image description here](https://i.sstatic.net/eNqIS.png)](https://i.sstatic.net/eNqIS.png)
  
  Can someone give me some clarification about why there is the expectation sign behind the entire equation? Considering that the agent is following a fixed policy $\pi$, why there should be an expectation when the trajectory of the future possible states is fixed (or maybe I am getting it wrong and it's not). In total, if the expectation here has the meaning of averaging over a series of trajectories, what are those trajectories and what are the weights of them when we want to compute the expected value over them according to [this Wikipedia definition of the expected values](https://en.wikipedia.org/wiki/Expected_value)?
  
  **Tags:** reinforcement-learning, value-functions, expectation, bellman-equations
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-04-03. Score: 4 | Accepted answer
  
  There needs to be an $E\_{\pi}$ over the infinite discounted return term because of two reasons-
  
  1. The policy could be stochastic in nature. That is, for any given state $s\_t$ at time $t$, the policy $\pi(s\_t)$ does not provide a deterministic action $a$, but rather, it provides us with a distribution over the possible next states, that is the action at time $t$, $a\_t$ is distributed as-
     $$a\_t \sim \pi(s\_t)$$
  2. Even if the policy $\pi$ being followed by an agent is deterministic, there still needs to be an expectation over the behavior of the underlying stochastic MDP environment. That is, any action $a\_t$, in general, only provides us with a distribution over the possible next states of the agent. That is,
     $$P(s\_{t + 1} = s') = P\_{\pi}(s' | s\_t) = \sum\_{a \in A} T(s,a,s') \times P\_{\pi}(a\_t = a)$$
     Here $T(s, a, s')$ is the transition function for the MDP and the above equation captures the stochasticity arising from both 1 and 2.
  
  As you see the expectation does not have to do with averaging over a collection of trajectories. However, that idea is often used in [Monte-Carlo](https://en.wikipedia.org/wiki/Monte_Carlo_method) estimation of value functions.
  
  EDIT:
  As pointed out in the comments, it is not correct to say that the expectation is not over a collection of trajectories.
  
  > **Comments:**
  > **username_3**: While your explanation is correct, your conclusion is not. The expectation does indeed correspond to an averaging over a collection of trajectories since you need to consider all the possible combinations of states and actions that lead to the return $G_t$ and weight them by how probable the combination is under the policy $\pi$ and the transition function $T$.  
  > **username_2**: @username_3 Have added an edit, thanks for pointing out.  
  > **username_0**: Thanks. So in the equation https://imgur.com/4g6fjIU the flashed symbols are the ones that denote the probabilities of the expectation formula?  
  > **username_0**: If so, why we only have $E_{\pi}$, shouldn't it be $E_{\pi,p}$? Because I suppose the $p$ of environment transitions is not dependent on policy $\pi$, is it?  
  > **username_2**: @username_0 in the linked image, summation symbols have been highlighted. The inner summation is over the possible next-state, reward tuple and the outer one is over a stochastic policy, so yes all of them together are capturing the expectation if that is what you meant.  
  > **username_2**: @username_0 yes I agree, perhaps to be rigorous, it should be made explicit that the expectation is over both $\pi$ and the transition distribution $p$.  
  
  ---
  
  ### Answer by username_1 on 2020-04-03. Score: 2
  
  In addition to [this answer](https://ai.stackexchange.com/a/19975/2444), I would like to note that, if the future trajectories were fixed (i.e. the environment and the policies were deterministic, and the agent always starts from the same state), the expectation of the sum (of the fixed rewards) would simply correspond to the actual sum, because the sum is a constant (i.e. the expectation of a constant is the constant itself), so the expectation operator also applies to the deterministic cases. Therefore, the expectation is a general way of expressing the value of a state in all possible cases (both when trajectories are fixed or not).
  
  
  ---
  
  