Answers.Body: <blockquote>
    <p>Also, in general, in the conditional expectation, which distribution do we compute the expectation with respect to? From what I have seen, in <span class="math-container">$\mathbb{E}[X|Y]$</span>, we always calculate the expected value over distribution <span class="math-container">$X$</span>.</p>
  </blockquote>
  
  <p>No, for <span class="math-container">$\mathbb{E}[X|Y]$</span> we take expectation of <span class="math-container">$X$</span> with respect to the conditional distribution <span class="math-container">$X|Y$</span>, i.e. </p>
  
  <p><span class="math-container">$$\mathbb{E}[X|Y] = \int_\mathbb{R} x p(x|y)dx\;;$$</span></p>
  
  <p>where <span class="math-container">$p(x|y)$</span> is the density function of the conditional distribution. If your random variables are discrete then replace the integral with a summation. Also note that <span class="math-container">$\mathbb{E}[X|Y]$</span> is still a random variable in <span class="math-container">$Y$</span>. </p>
  
  <blockquote>
    <p>How does <span class="math-container">$\mathbb{E}$</span> suddenly change to <span class="math-container">$\mathbb{E}_{\pi '}$</span> and the <span class="math-container">$A_t = \pi '(s)$</span> term disappears? </p>
  </blockquote>
  
  <p>This is because in this instance <span class="math-container">$\pi '(s)$</span> is a  deterministic policy, i.e. in state <span class="math-container">$s$</span> the policy will take action <span class="math-container">$b$</span> with probability 1 and all other actions with probability 0. NB: this is the convention used in Sutton and Barto to denote a deterministic policy.</p>
  
  <p>Without loss of generality, assume that <span class="math-container">$\pi'(s) = b$</span>. The implication of this is that in the first expectation we have 
  <span class="math-container">$$\mathbb{E}[R_{t+1} + \gamma v(S_{t+1}) | S_t = s, A_t = \pi'(s) = b] = \sum_{s',r}p(s',r|s,a=b)(r + \gamma v(s'))\;,$$</span>
  and in the second expectation we have 
  <span class="math-container">$$\mathbb{E}_{\pi'}[R_{t+1} + \gamma v(S_{t+1}) | S_t = s] = \sum_a\pi'(a|s)\sum_{s',r}p(s',r|s,a)(r + \gamma v(s'))\;;$$</span>
  However, we know that <span class="math-container">$\pi'(a|s) = 0 \; \forall a \neq b$</span>, so this sum over <span class="math-container">$a$</span> would equal 0 for all <span class="math-container">$a$</span> except when <span class="math-container">$a=b$</span>, in which case we know that <span class="math-container">$\pi'(b|s) = 1$</span>, and so the expectation becomes </p>
  
  <p><span class="math-container">$$\mathbb{E}_{\pi'}[R_{t+1} + \gamma v(S_{t+1}) | S_t = s] = \sum_{s',r}p(s',r|s,a=b)(r + \gamma v(s'))\;;$$</span></p>
  
  <p>and so we have equality of the two expectations. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-07T12:41:56.837Z
Answers.Text: This conversation has been [moved to chat](https://chat.stackexchange.com/rooms/109022/discussion-on-answer-by-david-ireland-how-does-mathbbe-suddenly-change-to).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-07T11:40:44.903Z
Answers.LastActivityDate: 2020-06-07T13:33:12.080Z
Answers.LastEditDate: 2020-06-07T13:33:12.080Z
Answers.UserDisplayName: David
Body: <p>In <a href="http://incompleteideas.net/book/bookdraft2017nov5.pdf#page=81" rel="nofollow noreferrer">Sutton-Barto's book</a> on page 63 (81 of the pdf):
  <span class="math-container">$$\mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_t=s,A_t=\pi'(s)] = \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_{t} = s]$$</span></p>
  
  <p>How does <span class="math-container">$\mathbb{E}$</span> suddenly change to <span class="math-container">$\mathbb{E}_{\pi'}$</span> and the <span class="math-container">$A_t = \pi'(s)$</span> term disappears?</p>
  
  <p>Also, in general, in the conditional expectation, which distribution do we compute the expectation with respect to? From what I have seen, in <span class="math-container">$\mathbb{E}[X \mid Y]$</span>, we always calculate the expected value over distribution <span class="math-container">$X$</span>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-08T11:10:47.273Z
Comments.Text: Can you additionally provide a reference to the equation number, so that it's found more easily?
Comments.UserDisplayName: tmaric
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-08T20:43:56.217Z
Comments.Text: Its in a proof given below equation 4.8
Comments.UserDisplayName: ZERO NULLS
Tags: <reinforcement-learning><probability><probability-distribution><expectation><statistics>
Title: How does $\mathbb{E}$ suddenly change to $\mathbb{E}_{\pi'}$ in this equation?
UserDisplayName: ZERO NULLS
ThreadText: # How does $\mathbb{E}$ suddenly change to $\mathbb{E}_{\pi'}$ in this equation?
  
  (Asked by: username_0 on 2020-06-07)
  
  In [Sutton-Barto's book](http://incompleteideas.net/book/bookdraft2017nov5.pdf#page=81) on page 63 (81 of the pdf):
  $$\mathbb{E}[R\_{t+1} + \gamma v\_\pi(S\_{t+1}) \mid S\_t=s,A\_t=\pi'(s)] = \mathbb{E}\_{\pi'}[R\_{t+1} + \gamma v\_\pi(S\_{t+1}) \mid S\_{t} = s]$$
  
  How does $\mathbb{E}$ suddenly change to $\mathbb{E}\_{\pi'}$ and the $A\_t = \pi'(s)$ term disappears?
  
  Also, in general, in the conditional expectation, which distribution do we compute the expectation with respect to? From what I have seen, in $\mathbb{E}[X \mid Y]$, we always calculate the expected value over distribution $X$.
  
  **Tags:** reinforcement-learning, probability, probability-distribution, expectation, statistics
  
  > **Comments:**
  > **username_1**: Can you additionally provide a reference to the equation number, so that it's found more easily?  
  > **username_0**: Its in a proof given below equation 4.8  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-06-07. Score: 3 | Accepted answer
  
  > Also, in general, in the conditional expectation, which distribution do we compute the expectation with respect to? From what I have seen, in $\mathbb{E}[X|Y]$, we always calculate the expected value over distribution $X$.
  
  No, for $\mathbb{E}[X|Y]$ we take expectation of $X$ with respect to the conditional distribution $X|Y$, i.e.
  
  $$\mathbb{E}[X|Y] = \int\_\mathbb{R} x p(x|y)dx\;;$$
  
  where $p(x|y)$ is the density function of the conditional distribution. If your random variables are discrete then replace the integral with a summation. Also note that $\mathbb{E}[X|Y]$ is still a random variable in $Y$.
  
  > How does $\mathbb{E}$ suddenly change to $\mathbb{E}\_{\pi '}$ and the $A\_t = \pi '(s)$ term disappears?
  
  This is because in this instance $\pi '(s)$ is a deterministic policy, i.e. in state $s$ the policy will take action $b$ with probability 1 and all other actions with probability 0. NB: this is the convention used in Sutton and Barto to denote a deterministic policy.
  
  Without loss of generality, assume that $\pi'(s) = b$. The implication of this is that in the first expectation we have
  $$\mathbb{E}[R\_{t+1} + \gamma v(S\_{t+1}) | S\_t = s, A\_t = \pi'(s) = b] = \sum\_{s',r}p(s',r|s,a=b)(r + \gamma v(s'))\;,$$
  and in the second expectation we have
  $$\mathbb{E}\_{\pi'}[R\_{t+1} + \gamma v(S\_{t+1}) | S\_t = s] = \sum\_a\pi'(a|s)\sum\_{s',r}p(s',r|s,a)(r + \gamma v(s'))\;;$$
  However, we know that $\pi'(a|s) = 0 \; \forall a \neq b$, so this sum over $a$ would equal 0 for all $a$ except when $a=b$, in which case we know that $\pi'(b|s) = 1$, and so the expectation becomes
  
  $$\mathbb{E}\_{\pi'}[R\_{t+1} + \gamma v(S\_{t+1}) | S\_t = s] = \sum\_{s',r}p(s',r|s,a=b)(r + \gamma v(s'))\;;$$
  
  and so we have equality of the two expectations.
  
  > **Comments:**
  > **username_3**: This conversation has been [moved to chat](https://chat.stackexchange.com/rooms/109022/discussion-on-answer-by-username_2-ireland-how-does-mathbbe-suddenly-change-to).  
  
  ---
  
  