Answers.Body: <p>David Ireland gives a fantastic answer, and I will provide an intuitive and gentle (but less rigorous) answer for those who are unfamiliar with the relevant statistical concepts. </p>
  
  <p><strong>Next reward <span class="math-container">$R_{t+1}$</span>:</strong> The next reward <span class="math-container">$R_{t+1}$</span> is solely dependent on the current state <span class="math-container">$S_t$</span> and action <span class="math-container">$A_t$</span>. It is only dependent on the policy because the policy details the probability distribution of actions given a state. Since we assume that the current state and action are given when calculating the expectation <span class="math-container">$\left(S_t = s, A_t = a\right)$</span>, then the policy does not give us any new information, and therefore the next reward is independent of the policy.</p>
  
  <p><strong>Return <span class="math-container">$G_{t+1}$</span></strong>: By definition, <span class="math-container">$v_{\pi}(s') = \mathbb{E}_{\pi}[G_{t+1}|S_{t+1} = s']$</span>. The value function is unaffected by sampling actions from the policy in the outer expectation <span class="math-container">$\left(\mathbb{E}_{\pi}[v_{\pi}(s')] = \mathbb{E}[v_{\pi}(s')]\right)$</span> since the value function is an expectation under the policy, and hence samples actions from the policy already.</p>
  
  <p><strong>Dropping <span class="math-container">$\pi$</span> from <span class="math-container">$\mathbb{E}_{\pi}$</span></strong>: The expectation under the current policy samples next states and rewards from the environment and also samples actions from our policy <span class="math-container">$\pi$</span>. Because the next reward is independent of the policy given the current state and action, and because the value function is unaffected by sampling actions from the policy in the outer expectation, we can simply drop the policy from the outer expectation (the outer expectation will still sample next states and rewards from the environment). </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-08T03:27:02.593Z
Answers.LastActivityDate: 2020-06-08T03:27:02.593Z
Answers.UserDisplayName: DeepQZero
Answers.Body: <p>Let's first write the state-value function as</p>
  <p><span class="math-container">$$q_{\pi}(s,a) = \mathbb{E}_{p, \pi}[R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a]\;,$$</span></p>
  <p>where <span class="math-container">$R_{t+1}$</span> is the random variable that represents the reward gained at time <span class="math-container">$t+1$</span>, i.e. after we have taken action <span class="math-container">$A_t = a$</span> in state <span class="math-container">$S_t = s$</span>, while <span class="math-container">$G_{t+1}$</span> is the random variable that represents the return, the sum of future rewards. This allows us to show that the expectation is taken under the conditional joint distribution <span class="math-container">$p(s', r \mid s, a)$</span>, which is the environment dynamics, and future actions are taken from our policy distribution <span class="math-container">$\pi$</span>.</p>
  <p>As <span class="math-container">$R_{t+1}$</span> depends on <span class="math-container">$S_t = s, A_t = a$</span> and <span class="math-container">$p(s', r \mid s, a)$</span>,  the only random variable in the expectation that is dependent on our policy <span class="math-container">$\pi$</span> is <span class="math-container">$G_{t+1}$</span>, because this is the sum of future reward signals and so will depend on future state-action values. Thus, we can rewrite again as
  <span class="math-container">$$q_{\pi}(s,a) = \mathbb{E}_{p}[R_{t+1} + \gamma \mathbb{E}_{\pi}[ G_{t+1} |S_{t+1} = s'] | S_t = s, A_t = a]\;,$$</span>
  where the inner expectation (coupled with the fact its inside an expectation over the state and reward distributions) should look familiar to you as the state value function, i.e.
  <span class="math-container">$$\mathbb{E}_{\pi}[ G_{t+1} |S_{t+1} = s'] = v_{\pi}(s')\;.$$</span>
  This leads us to get what you have
  <span class="math-container">$$q_{\pi}(s,a) = \mathbb{E}_{p}[R_{t+1} + \gamma v_{\pi}(s') | S_t = s, A_t = a]\;,$$</span>
  where the only difference is that we have made clear what our expectation is taken with respect to.</p>
  <p>The expectation is taken with respect to the conditional joint distribution <span class="math-container">$p(s', r \mid s, a)$</span>, and we usually include the <span class="math-container">$\pi$</span> subscript to denote that they are also taking the expectation with respect to the policy, but here this does not affect the first term as we have conditioned on knowing <span class="math-container">$A_t = a$</span> and only applies to the future reward signals.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-31T15:46:38.943Z
Answers.Text: I now this post was closed a while ago, but I hope someone sees this.. How do you arrive at $E_\pi[G_{t+1} | S_{t+1} = s']$ in the second paragraph?
Answers.UserDisplayName: mNugget
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-01T13:30:23.433Z
Answers.Text: @mNugget because $R_{t+1}$ does not depend on the policy, since we have conditioned on $S_t, A_t$. This means that the only term which depends on the policy is the future returns $G_{t+1}$. We write the expectation explicitly conditioned on the $S_{t+1}$ so it resembles the value function, but it is valid because the outer expectation is also wrt to the dynamics functions $p$, so we are averaging over all possible next states.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-06T11:07:56.247Z
Answers.LastActivityDate: 2022-01-25T13:10:01.103Z
Answers.LastEditDate: 2022-01-25T13:10:01.103Z
Answers.UserDisplayName: David
Body: <p>I often see that the state-action value function is expressed as:</p>
  <p><span class="math-container">$$q_{\pi}(s,a)=\color{red}{\mathbb{E}_{\pi}}[R_{t+1}+\gamma G_{t+1} | S_t=s, A_t = a] = \color{blue}{\mathbb{E}}[R_{t+1}+\gamma v_{\pi}(s') |S_t = s, A_t =a]$$</span></p>
  <p>Why does expressing the future return in the time <span class="math-container">$t+1$</span> as a state value function <span class="math-container">$v_{\pi}$</span> make the expected value under policy change to expected value in general?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-24T12:58:44.287Z
Comments.Text: [Here](https://ai.stackexchange.com/q/20982/2444) is a related question.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><value-functions><bellman-equations><expectation>
Title: Why does the state-action value function, defined as an expected value of the reward and state value function, not need to follow a policy?
UserDisplayName: Daniel Wiczew
ThreadText: # Why does the state-action value function, defined as an expected value of the reward and state value function, not need to follow a policy?
  
  (Asked by: username_0 on 2020-06-06)
  
  I often see that the state-action value function is expressed as:
  
  $$q\_{\pi}(s,a)=\color{red}{\mathbb{E}\_{\pi}}[R\_{t+1}+\gamma G\_{t+1} | S\_t=s, A\_t = a] = \color{blue}{\mathbb{E}}[R\_{t+1}+\gamma v\_{\pi}(s') |S\_t = s, A\_t =a]$$
  
  Why does expressing the future return in the time $t+1$ as a state value function $v\_{\pi}$ make the expected value under policy change to expected value in general?
  
  **Tags:** reinforcement-learning, value-functions, bellman-equations, expectation
  
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/q/20982/2444) is a related question.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2020-06-06. Score: 10 | Accepted answer
  
  Let's first write the state-value function as
  
  $$q\_{\pi}(s,a) = \mathbb{E}\_{p, \pi}[R\_{t+1} + \gamma G\_{t+1} | S\_t = s, A\_t = a]\;,$$
  
  where $R\_{t+1}$ is the random variable that represents the reward gained at time $t+1$, i.e. after we have taken action $A\_t = a$ in state $S\_t = s$, while $G\_{t+1}$ is the random variable that represents the return, the sum of future rewards. This allows us to show that the expectation is taken under the conditional joint distribution $p(s', r \mid s, a)$, which is the environment dynamics, and future actions are taken from our policy distribution $\pi$.
  
  As $R\_{t+1}$ depends on $S\_t = s, A\_t = a$ and $p(s', r \mid s, a)$, the only random variable in the expectation that is dependent on our policy $\pi$ is $G\_{t+1}$, because this is the sum of future reward signals and so will depend on future state-action values. Thus, we can rewrite again as
  $$q\_{\pi}(s,a) = \mathbb{E}\_{p}[R\_{t+1} + \gamma \mathbb{E}\_{\pi}[ G\_{t+1} |S\_{t+1} = s'] | S\_t = s, A\_t = a]\;,$$
  where the inner expectation (coupled with the fact its inside an expectation over the state and reward distributions) should look familiar to you as the state value function, i.e.
  $$\mathbb{E}\_{\pi}[ G\_{t+1} |S\_{t+1} = s'] = v\_{\pi}(s')\;.$$
  This leads us to get what you have
  $$q\_{\pi}(s,a) = \mathbb{E}\_{p}[R\_{t+1} + \gamma v\_{\pi}(s') | S\_t = s, A\_t = a]\;,$$
  where the only difference is that we have made clear what our expectation is taken with respect to.
  
  The expectation is taken with respect to the conditional joint distribution $p(s', r \mid s, a)$, and we usually include the $\pi$ subscript to denote that they are also taking the expectation with respect to the policy, but here this does not affect the first term as we have conditioned on knowing $A\_t = a$ and only applies to the future reward signals.
  
  > **Comments:**
  > **username_4**: I now this post was closed a while ago, but I hope someone sees this.. How do you arrive at $E_\pi[G_{t+1} | S_{t+1} = s']$ in the second paragraph?  
  > **username_3**: @username_4 because $R_{t+1}$ does not depend on the policy, since we have conditioned on $S_t, A_t$. This means that the only term which depends on the policy is the future returns $G_{t+1}$. We write the expectation explicitly conditioned on the $S_{t+1}$ so it resembles the value function, but it is valid because the outer expectation is also wrt to the dynamics functions $p$, so we are averaging over all possible next states.  
  
  ---
  
  ### Answer by username_2 on 2020-06-08. Score: 4
  
  username_3 Ireland gives a fantastic answer, and I will provide an intuitive and gentle (but less rigorous) answer for those who are unfamiliar with the relevant statistical concepts.
  
  **Next reward $R\_{t+1}$:** The next reward $R\_{t+1}$ is solely dependent on the current state $S\_t$ and action $A\_t$. It is only dependent on the policy because the policy details the probability distribution of actions given a state. Since we assume that the current state and action are given when calculating the expectation $\left(S\_t = s, A\_t = a\right)$, then the policy does not give us any new information, and therefore the next reward is independent of the policy.
  
  **Return $G\_{t+1}$**: By definition, $v\_{\pi}(s') = \mathbb{E}\_{\pi}[G\_{t+1}|S\_{t+1} = s']$. The value function is unaffected by sampling actions from the policy in the outer expectation $\left(\mathbb{E}\_{\pi}[v\_{\pi}(s')] = \mathbb{E}[v\_{\pi}(s')]\right)$ since the value function is an expectation under the policy, and hence samples actions from the policy already.
  
  **Dropping $\pi$ from $\mathbb{E}\_{\pi}$**: The expectation under the current policy samples next states and rewards from the environment and also samples actions from our policy $\pi$. Because the next reward is independent of the policy given the current state and action, and because the value function is unaffected by sampling actions from the policy in the outer expectation, we can simply drop the policy from the outer expectation (the outer expectation will still sample next states and rewards from the environment).
  
  
  ---
  
  