Answers.Body: <p>To expand <span class="math-container">$\mathbb{E}_\pi[\gamma G_{t+1}|S_t=s]$</span>, you can take the same expectation over next state and reward as for <span class="math-container">$R_{t+1}$</span> (in fact this is normally shown without separating the two terms as you have done, but as it is the expansion of this part where you want help, we can do it separately).</p>
  <p>The key thing is to move forward one time step - choosing the action using the policy, and the reward and next state using the state transition function - and express the expectation as a sum of the probabilities for the next step (also noticing that this changes the condition from <span class="math-container">$S_t=s$</span> to <span class="math-container">$S_{t+1}=s'$</span>):</p>
  <p><span class="math-container">$$\mathbb{E}_\pi[\gamma G_{t+1}|S_t=s] = \gamma \sum_{a}\pi(a|s) \sum_{r,s'} p(r,s'|s,a) \mathbb{E}_\pi[\gamma G_{t+1}|S_{t+1}=s']$$</span></p>
  <p>Then, we can notice that <span class="math-container">$\mathbb{E}_\pi[\gamma G_{t+1}|S_{t+1}=s']$</span> is <span class="math-container">$v_{\pi}(s')$</span> and get</p>
  <p><span class="math-container">$$\mathbb{E}_\pi[\gamma G_{t+1}|S_t=s] = \gamma \sum_{a}\pi(a|s)\sum_{r,s'} p(r,s'|s,a) v_{\pi}(s')$$</span></p>
  <p>Now we can recombine this with the other expression for expected immediate reward that you have already resolved, because the summation is the same in both parts, and get result 3.14 from the book.</p>
  <p>It is more common to resolve both parts of the expectation the same way though, and not split them up to later recombine. The tricky part is perhaps realising that you cannot resolve the expected return fully when looking ahead one time step, only express it as a sum of expected returns from all possible next states.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-08T07:06:02.220Z
Answers.LastActivityDate: 2023-05-08T07:11:15.743Z
Answers.LastEditDate: 2023-05-08T07:11:15.743Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am reading the Sutton &amp; Barto (2018) RL textbook.</p>
  <p>On page 59, it derives the recursive property of the state-value function as below.
  Could you explain the steps of third and fourth equality?
  Here is what I have.
  <span class="math-container">\begin{aligned}
  E_\pi[R_{t+1}+\gamma G_{t+1}|S_t=s] &amp;= E_\pi[R_{t+1}|S_t=s]+E_\pi[\gamma G_{t+1}|S_t=s] \\
  &amp;= \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a) \times r + ???
  \end{aligned}</span></p>
  <p><a href="https://i.sstatic.net/VybNs.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VybNs.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-10T17:29:41.547Z
Comments.Text: This post asks the same question: https://stats.stackexchange.com/questions/243384/deriving-bellmans-equation-in-reinforcement-learning
Comments.UserDisplayName: Emad
Tags: <reinforcement-learning><markov-decision-process><value-functions><sutton-barto><bellman-equations>
Title: Could you explain these 2 steps of the derivation of the Bellman equation as a recursive equation in Sutton & Barto?
UserDisplayName: tesio
ThreadText: # Could you explain these 2 steps of the derivation of the Bellman equation as a recursive equation in Sutton & Barto?
  
  (Asked by: username_0 on 2023-05-08)
  
  I am reading the Sutton & Barto (2018) RL textbook.
  
  On page 59, it derives the recursive property of the state-value function as below.
  Could you explain the steps of third and fourth equality?
  Here is what I have.
  \begin{aligned}
  E\_\pi[R\_{t+1}+\gamma G\_{t+1}|S\_t=s] &= E\_\pi[R\_{t+1}|S\_t=s]+E\_\pi[\gamma G\_{t+1}|S\_t=s] \\
  &= \sum\_a \pi(a|s) \sum\_{s',r} p(s',r|s,a) \times r + ???
  \end{aligned}
  
  [![enter image description here](https://i.sstatic.net/VybNs.png)](https://i.sstatic.net/VybNs.png)
  
  **Tags:** reinforcement-learning, markov-decision-process, value-functions, sutton-barto, bellman-equations
  
  > **Comments:**
  > **username_1**: This post asks the same question: https://stats.stackexchange.com/questions/243384/deriving-bellmans-equation-in-reinforcement-learning  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-05-08. Score: 1 | Accepted answer
  
  To expand $\mathbb{E}\_\pi[\gamma G\_{t+1}|S\_t=s]$, you can take the same expectation over next state and reward as for $R\_{t+1}$ (in fact this is normally shown without separating the two terms as you have done, but as it is the expansion of this part where you want help, we can do it separately).
  
  The key thing is to move forward one time step - choosing the action using the policy, and the reward and next state using the state transition function - and express the expectation as a sum of the probabilities for the next step (also noticing that this changes the condition from $S\_t=s$ to $S\_{t+1}=s'$):
  
  $$\mathbb{E}\_\pi[\gamma G\_{t+1}|S\_t=s] = \gamma \sum\_{a}\pi(a|s) \sum\_{r,s'} p(r,s'|s,a) \mathbb{E}\_\pi[\gamma G\_{t+1}|S\_{t+1}=s']$$
  
  Then, we can notice that $\mathbb{E}\_\pi[\gamma G\_{t+1}|S\_{t+1}=s']$ is $v\_{\pi}(s')$ and get
  
  $$\mathbb{E}\_\pi[\gamma G\_{t+1}|S\_t=s] = \gamma \sum\_{a}\pi(a|s)\sum\_{r,s'} p(r,s'|s,a) v\_{\pi}(s')$$
  
  Now we can recombine this with the other expression for expected immediate reward that you have already resolved, because the summation is the same in both parts, and get result 3.14 from the book.
  
  It is more common to resolve both parts of the expectation the same way though, and not split them up to later recombine. The tricky part is perhaps realising that you cannot resolve the expected return fully when looking ahead one time step, only express it as a sum of expected returns from all possible next states.
  
  
  ---
  
  