Answers.Body: <p>First note that <span class="math-container">$\mathbb{E}[R_{t+1} |S_t=s] = \sum_{s',r}rm(s',r|s)$</span> where <span class="math-container">$m(\cdot)$</span> is the mass function for the joint distribution of <span class="math-container">$S_{t+1},R_{t+1}$</span>. </p>
  
  <p>If you are currently in state <span class="math-container">$S_t$</span> and we condition on taking action <span class="math-container">$a$</span> then the expected reward at time <span class="math-container">$t+1$</span> is given as follows:</p>
  
  <p><span class="math-container">\begin{align} \mathbb{E}[R_{t+1} | S_t = s, A_t=a] &amp; = \sum_{s',r}rp(s'r|s,a)\;.
  \end{align}</span></p>
  
  <p>However, action <span class="math-container">$A_t$</span> is taken according to some stochastic policy <span class="math-container">$\pi$</span> so we need to marginalise this out of our expectation by using the <a href="https://en.wikipedia.org/wiki/Law_of_total_expectation" rel="nofollow noreferrer">tower law</a> - i.e. we take </p>
  
  <p><span class="math-container">$$\mathbb{E}_{A_t \sim \pi}[\mathbb{E}[R_{t+1} | S_t = s, A_t=a]|S_t = s] = \sum_a \pi(a|s)\sum_{s',r}rp(s'r|s,a) = \mathbb{E}[R_{t+1} | S_t = s]\;.$$</span></p>
  
  <p>To see why this holds we can re-write using some arbitrary mass functions <span class="math-container">$f(\cdot),h(\cdot),g(\cdot),m(\cdot)$</span> as</p>
  
  <p><span class="math-container">$$\pi(a|s)p(s'r|s,a) = \frac{f(a,s)}{g(s)} \times \frac{h(s',r,a,s)}{f(a,s)} = m(s',r,a|s)\;,$$</span>
  so when we would end up with (after re-arranging the summations)</p>
  
  <p><span class="math-container">$$\sum_{s',r}r \sum_{a}m(s',r,a|s) = \sum_{s',r}r m(s',r|s) = \mathbb{E}[R_{t+1}|S_t = s]\;;$$</span>
  as required. </p>
  
  <p>NB: what you have done is mostly correct except be careful when going from (2) to (3). They are exactly the same equation except for the time stamps, which means you would have to change the time stamps in your <span class="math-container">$r(s,a)$</span>. Note that when you are at time step <span class="math-container">$t$</span> you take your action <span class="math-container">$A_t$</span> from your current <span class="math-container">$S_t$</span> to transition into state <span class="math-container">$S_{t+1}$</span> and then receive reward <span class="math-container">$R_{t+1}$</span> (and the next state). </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-07T10:30:24.417Z
Answers.Text: How is the equation preceding the mention of "tower law" equal to the equation following "tower law"? Can you share some other references that help understanding this transition?
Answers.UserDisplayName: Sid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-07T15:01:24.987Z
Answers.Text: They are not equal. The latter is the expected value of the former over the conditional action distribution $\pi$. It is a standard use of the tower property which I’ve linked, no other resources should be necessary.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-21T17:31:05.063Z
Answers.Text: @David Why at the first summand you used the mass m(.) and in the next you used p(.) for the joint?
Answers.UserDisplayName: piero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-21T17:41:55.570Z
Answers.Text: @David One more thing, when you took the expectation $\mathbb{E}_{A_t \sim \pi} [\mathbb{E}[R_{t+1}| S_t=s,A_t=a] | S_t=s]$, $A_t$ is fixed to $a$, in the same way of the last equation, but them you make it vary with the outer sum on $a$, this seems a little confusing, unless $\sum_{s',r}rp(s'r|s,a)$ is not meant to be a quantity, but actually a function of $a$ and $s$.
Answers.UserDisplayName: piero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-05T14:17:03.647Z
Answers.LastActivityDate: 2020-06-07T10:29:39.763Z
Answers.LastEditDate: 2020-06-07T10:29:39.763Z
Answers.UserDisplayName: David
Body: <p>I'm trying to solve exercise 3.11 from the book <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">Sutton and Barto's book</a> (2nd edition)</p>
  <blockquote>
  <p>Exercise 3.11 If the current state is <span class="math-container">$S_t$</span> , and actions are selected according to a stochastic policy <span class="math-container">$\pi$</span>, then what is the expectation of <span class="math-container">$R_{t+1}$</span> in terms of <span class="math-container">$\pi$</span> and the four-argument function <span class="math-container">$p$</span> (3.2)?</p>
  </blockquote>
  <p>Here's my attempt.</p>
  <p>For each state <span class="math-container">$s$</span>, the expected immediate reward when taking action <span class="math-container">$a$</span> is given in terms of <span class="math-container">$p$</span> by eq 3.5 in the book:</p>
  <p><span class="math-container">$$r(s,a) = \sum_{r \in R} r \, \sum_{s'\in S} p(s',r \mid s,a) = E[R_t \mid S_{t-1} = s, A_{t-1} = a] \tag{1}\label{1}$$</span></p>
  <p>The policy <span class="math-container">$\pi(a \mid s)$</span>, on the other hand, gives the probability of taking action <span class="math-container">$a$</span> given the state <span class="math-container">$s$</span>.</p>
  <p>Is it possible to express the expectation of the immediate reward over all actions <span class="math-container">$A$</span> from the state <span class="math-container">$s$</span> using (1) as</p>
  <p><span class="math-container">$$E[R_t \mid S_{t-1} = s, A] = \sum_{a \in A} \pi(a \mid s) r(a,s)  \tag{2}\label{2}$$</span></p>
  <p>?</p>
  <p>If this is valid, is this also valid in the next time step</p>
  <p><span class="math-container">$$E[R_{t+1} \mid S_{t} = s, A] = \sum_{a \in A} \pi(a \mid s) r(a, s) \tag{3}\label{3}$$</span></p>
  <p>?</p>
  <p>If (2) and (3) are OK, then</p>
  <p><span class="math-container">$$E[R_{t+1} \mid S_{t} = s, A] = \sum_{a \in A} \pi(a \mid s) \sum_{r \in R} r \, \sum_{s'\in S} p(s',r \mid s,a)$$</span></p>
  
Tags: <reinforcement-learning><rewards><sutton-barto><expectation><transition-model>
Title: If the current state is $S_t$ and the actions are chosen according to $\pi$, what is the expectation of $R_{t+1}$ in terms of $\pi$ and $p$?
UserDisplayName: tmaric
ThreadText: # If the current state is $S_t$ and the actions are chosen according to $\pi$, what is the expectation of $R_{t+1}$ in terms of $\pi$ and $p$?
  
  (Asked by: username_0 on 2020-06-05)
  
  I'm trying to solve exercise 3.11 from the book [Sutton and Barto's book](http://incompleteideas.net/book/RLbook2020.pdf) (2nd edition)
  
  > Exercise 3.11 If the current state is $S\_t$ , and actions are selected according to a stochastic policy $\pi$, then what is the expectation of $R\_{t+1}$ in terms of $\pi$ and the four-argument function $p$ (3.2)?
  
  Here's my attempt.
  
  For each state $s$, the expected immediate reward when taking action $a$ is given in terms of $p$ by eq 3.5 in the book:
  
  $$r(s,a) = \sum\_{r \in R} r \, \sum\_{s'\in S} p(s',r \mid s,a) = E[R\_t \mid S\_{t-1} = s, A\_{t-1} = a] \tag{1}\label{1}$$
  
  The policy $\pi(a \mid s)$, on the other hand, gives the probability of taking action $a$ given the state $s$.
  
  Is it possible to express the expectation of the immediate reward over all actions $A$ from the state $s$ using (1) as
  
  $$E[R\_t \mid S\_{t-1} = s, A] = \sum\_{a \in A} \pi(a \mid s) r(a,s) \tag{2}\label{2}$$
  
  ?
  
  If this is valid, is this also valid in the next time step
  
  $$E[R\_{t+1} \mid S\_{t} = s, A] = \sum\_{a \in A} \pi(a \mid s) r(a, s) \tag{3}\label{3}$$
  
  ?
  
  If (2) and (3) are OK, then
  
  $$E[R\_{t+1} \mid S\_{t} = s, A] = \sum\_{a \in A} \pi(a \mid s) \sum\_{r \in R} r \, \sum\_{s'\in S} p(s',r \mid s,a)$$
  
  **Tags:** reinforcement-learning, rewards, sutton-barto, expectation, transition-model
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-05. Score: 5 | Accepted answer
  
  First note that $\mathbb{E}[R\_{t+1} |S\_t=s] = \sum\_{s',r}rm(s',r|s)$ where $m(\cdot)$ is the mass function for the joint distribution of $S\_{t+1},R\_{t+1}$.
  
  If you are currently in state $S\_t$ and we condition on taking action $a$ then the expected reward at time $t+1$ is given as follows:
  
  \begin{align} \mathbb{E}[R\_{t+1} | S\_t = s, A\_t=a] & = \sum\_{s',r}rp(s'r|s,a)\;.
  \end{align}
  
  However, action $A\_t$ is taken according to some stochastic policy $\pi$ so we need to marginalise this out of our expectation by using the [tower law](https://en.wikipedia.org/wiki/Law_of_total_expectation) - i.e. we take
  
  $$\mathbb{E}\_{A\_t \sim \pi}[\mathbb{E}[R\_{t+1} | S\_t = s, A\_t=a]|S\_t = s] = \sum\_a \pi(a|s)\sum\_{s',r}rp(s'r|s,a) = \mathbb{E}[R\_{t+1} | S\_t = s]\;.$$
  
  To see why this holds we can re-write using some arbitrary mass functions $f(\cdot),h(\cdot),g(\cdot),m(\cdot)$ as
  
  $$\pi(a|s)p(s'r|s,a) = \frac{f(a,s)}{g(s)} \times \frac{h(s',r,a,s)}{f(a,s)} = m(s',r,a|s)\;,$$
  so when we would end up with (after re-arranging the summations)
  
  $$\sum\_{s',r}r \sum\_{a}m(s',r,a|s) = \sum\_{s',r}r m(s',r|s) = \mathbb{E}[R\_{t+1}|S\_t = s]\;;$$
  as required.
  
  NB: what you have done is mostly correct except be careful when going from (2) to (3). They are exactly the same equation except for the time stamps, which means you would have to change the time stamps in your $r(s,a)$. Note that when you are at time step $t$ you take your action $A\_t$ from your current $S\_t$ to transition into state $S\_{t+1}$ and then receive reward $R\_{t+1}$ (and the next state).
  
  > **Comments:**
  > **username_2**: How is the equation preceding the mention of "tower law" equal to the equation following "tower law"? Can you share some other references that help understanding this transition?  
  > **username_1**: They are not equal. The latter is the expected value of the former over the conditional action distribution $\pi$. It is a standard use of the tower property which I’ve linked, no other resources should be necessary.  
  > **username_3**: @David Why at the first summand you used the mass m(.) and in the next you used p(.) for the joint?  
  > **username_3**: @David One more thing, when you took the expectation $\mathbb{E}_{A_t \sim \pi} [\mathbb{E}[R_{t+1}| S_t=s,A_t=a] | S_t=s]$, $A_t$ is fixed to $a$, in the same way of the last equation, but them you make it vary with the outer sum on $a$, this seems a little confusing, unless $\sum_{s',r}rp(s'r|s,a)$ is not meant to be a quantity, but actually a function of $a$ and $s$.  
  
  ---
  
  