Answers.Body: <p>for first, let's see what <span class="math-container">$S_{t+1}$</span> means: that means from next time-step.
  please remember what was V(s) define: what you expect to gain from this state(<span class="math-container">$S_t$</span>) until end of trajectory. so, you need to know for each action what is expected reward and that is the probability of next state occurrence if you choose action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span> (<span class="math-container">$p(s',a|s,a)$</span>) multiply in (reward and next state value function). in fact it is a bootstrapping formula:</p>
  <p>python code:</p>
  <pre><code>values = list()
  for s in P:
      values.append(list())
      for a in P[s]:
          values[-1].append(sum([prob * (reward + gamma * V[next_state])
                                    for prob, next_state, reward, _ in P[s][a]]))
  </code></pre>
  <p>mind that <code>P</code> is <span class="math-container">$MDP$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-09T15:25:22.677Z
Answers.LastActivityDate: 2022-09-09T15:48:32.513Z
Answers.LastEditDate: 2022-09-09T15:48:32.513Z
Answers.UserDisplayName: MoRe
Body: <p>I'm having trouble going to the 2nd to last line of (3.14),</p>
  <p><a href="http://incompleteideas.net/book/RLbook2020.pdf#page=81" rel="nofollow noreferrer">http://incompleteideas.net/book/RLbook2020.pdf#page=81</a></p>
  
  <p><span class="math-container">$$
  \require{enclose}
  \begin{aligned}
  v_{\pi}(s) &amp; \doteq \mathbb{E}_{\pi}\left[G_{t} \mid S_{t}=s\right] \\
  &amp;=\mathbb{E}_{\pi}\left[R_{t+1}+\gamma G_{t+1} \mid S_{t}=s\right] \\
  &amp;=\sum_{a} \pi(a \mid s) \sum_{s^{\prime}} \sum_{r} p\left(s^{\prime}, r \mid s, a\right)\left[r+\gamma \mathbb{E}_{\pi}\left[G_{t+1} \mid \enclose{circle}[mathcolor="red"]{S_{t+1}}=s^{\prime}\right]\right] \\
  &amp;=\sum_{a} \pi(a \mid s) \sum_{s^{\prime}, r} p\left(s^{\prime}, r \mid s, a\right)\left[r+\gamma v_{\pi}\left(s^{\prime}\right)\right], \quad \text { for all } s \in \mathcal{S},
  \end{aligned}
  $$</span></p>
  <p>I don't understand where the red circled term comes from. Namely, where the <span class="math-container">$S_{t+1}$</span> comes from, since I was expecting an <span class="math-container">$S_t$</span> from the previous line.</p>
  <p>Can you please explain? Thank you.</p>
  
Tags: <reinforcement-learning><machine-learning>
Title: How do I derive Sutton and Barto Equation 3.14?
UserDisplayName: Mondo Duke
ThreadText: # How do I derive Sutton and Barto Equation 3.14?
  
  (Asked by: username_0 on 2022-09-09)
  
  I'm having trouble going to the 2nd to last line of (3.14),
  
  <http://incompleteideas.net/book/RLbook2020.pdf#page=81>
  
  $$
  \require{enclose}
  \begin{aligned}
  v\_{\pi}(s) & \doteq \mathbb{E}\_{\pi}\left[G\_{t} \mid S\_{t}=s\right] \\
  &=\mathbb{E}\_{\pi}\left[R\_{t+1}+\gamma G\_{t+1} \mid S\_{t}=s\right] \\
  &=\sum\_{a} \pi(a \mid s) \sum\_{s^{\prime}} \sum\_{r} p\left(s^{\prime}, r \mid s, a\right)\left[r+\gamma \mathbb{E}\_{\pi}\left[G\_{t+1} \mid \enclose{circle}[mathcolor="red"]{S\_{t+1}}=s^{\prime}\right]\right] \\
  &=\sum\_{a} \pi(a \mid s) \sum\_{s^{\prime}, r} p\left(s^{\prime}, r \mid s, a\right)\left[r+\gamma v\_{\pi}\left(s^{\prime}\right)\right], \quad \text { for all } s \in \mathcal{S},
  \end{aligned}
  $$
  
  I don't understand where the red circled term comes from. Namely, where the $S\_{t+1}$ comes from, since I was expecting an $S\_t$ from the previous line.
  
  Can you please explain? Thank you.
  
  **Tags:** reinforcement-learning, machine-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-09-09. Score: 1
  
  for first, let's see what $S\_{t+1}$ means: that means from next time-step.
  please remember what was V(s) define: what you expect to gain from this state($S\_t$) until end of trajectory. so, you need to know for each action what is expected reward and that is the probability of next state occurrence if you choose action $a$ in state $s$ ($p(s',a|s,a)$) multiply in (reward and next state value function). in fact it is a bootstrapping formula:
  
  python code:
  
  ```
  values = list()
  for s in P:
      values.append(list())
      for a in P[s]:
          values[-1].append(sum([prob * (reward + gamma * V[next_state])
                                    for prob, next_state, reward, _ in P[s][a]]))
  
  ```
  
  mind that `P` is $MDP$
  
  
  ---
  
  