Answers.Body: <p>This equality:</p>
  <blockquote>
  <p><span class="math-container">$\mathop{\mathbb{E}}[R_{t+2}+\gamma v_\pi(S_{t+2})|S_{t+1},A_{t+1}=\pi'(S_{t+1})]=R_{t+2}+\gamma v_\pi(S_{t+2})$</span></p>
  </blockquote>
  <p>Does <strong>not</strong> hold in general, and in fact is not well-formed, because <span class="math-container">$R_{t+2}$</span> and <span class="math-container">$S_{t+2}$</span> are random variables following a distribution, whilst the expectation of the expression is a single real number.</p>
  <p>However, the conversion works in the context of the outer expection, because the conditional part <span class="math-container">$S_{t+1},A_{t+1}=\pi'(S_{t+1})$</span> matches the outer expectation being <span class="math-container">$\mathop{\mathbb{E}_{\pi'}}$</span> i.e. the expectation over action choices made using <span class="math-container">$\pi'$</span>. So the inner expectation, and its condition can simply be dropped because the condition on choosing the action using the new improved policy is the same as the outer expectation, just using different notation.</p>
  <p>It is important that here the inner expectation is simply added to other terms. If it was multiplied by a variable term that it was dependent upon, the expression could not be unpacked from the inner expectation without affecting the result.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-15T23:54:50.643Z
Answers.Text: If so, I think the random variables $S_t$ and $A_t$ are the same as $S_{t+1}$ and $A_{t+1}$. So, the conditions of inner and other expectations are the same, so they can be dropped from inner expectation.
Answers.UserDisplayName: tesio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-16T07:39:17.780Z
Answers.Text: @teslo No those are different variables, the time step index difference is important. The reason it's the same condition is due to *how* those variables have been chosen. If it's not clear how they relate to the improved policy, and not the original one, I could add more about that to the answer
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-20T11:00:55.060Z
Answers.Text: May I ask more details?
Answers.UserDisplayName: tesio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-20T11:02:28.247Z
Answers.Text: @tesio: Sure, but could you please say where you are stuck, or what you don't understand, so I can add the right details that help you?
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-20T11:04:51.147Z
Answers.Text: Oh, I can't understand how they relate to the improved policy, and not the original one
Answers.UserDisplayName: tesio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-15T15:45:32.610Z
Answers.LastActivityDate: 2023-06-15T15:58:49.330Z
Answers.LastEditDate: 2023-06-15T15:58:49.330Z
Answers.UserDisplayName: Neil Slater
Body: <p><a href="https://i.sstatic.net/gorIN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gorIN.png" alt="enter image description here" /></a></p>
  <p>I don't understand one step of the proof.
  The equality means <span class="math-container">$\mathop{\mathbb{E}}[R_{t+2}+\gamma v_\pi(S_{t+2})|S_{t+1},A_{t+1}=\pi'(S_{t+1})]=R_{t+2}+\gamma v_\pi(S_{t+2})$</span></p>
  <p>Could you explain the detailed steps of why the equality holds?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-15T07:10:58.767Z
Comments.Text: You seem to be missing a factor of $\gamma$ on the right hand side, or should remove it from the left hand side, of the equality you are questioning
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-15T12:12:38.133Z
Comments.Text: Sorry for the confusion. I removed the gamma on the left side
Comments.UserDisplayName: tesio
Tags: <reinforcement-learning><policy-improvement-theorem>
Title: Could you explain steps of proofs of policy improvement theorem in sutton & barto book?
UserDisplayName: tesio
ThreadText: # Could you explain steps of proofs of policy improvement theorem in sutton & barto book?
  
  (Asked by: username_0 on 2023-06-15)
  
  [![enter image description here](https://i.sstatic.net/gorIN.png)](https://i.sstatic.net/gorIN.png)
  
  I don't understand one step of the proof.
  The equality means $\mathop{\mathbb{E}}[R\_{t+2}+\gamma v\_\pi(S\_{t+2})|S\_{t+1},A\_{t+1}=\pi'(S\_{t+1})]=R\_{t+2}+\gamma v\_\pi(S\_{t+2})$
  
  Could you explain the detailed steps of why the equality holds?
  
  **Tags:** reinforcement-learning, policy-improvement-theorem
  
  > **Comments:**
  > **username_1**: You seem to be missing a factor of $\gamma$ on the right hand side, or should remove it from the left hand side, of the equality you are questioning  
  > **username_0**: Sorry for the confusion. I removed the gamma on the left side  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-06-15. Score: 1
  
  This equality:
  
  > $\mathop{\mathbb{E}}[R\_{t+2}+\gamma v\_\pi(S\_{t+2})|S\_{t+1},A\_{t+1}=\pi'(S\_{t+1})]=R\_{t+2}+\gamma v\_\pi(S\_{t+2})$
  
  Does **not** hold in general, and in fact is not well-formed, because $R\_{t+2}$ and $S\_{t+2}$ are random variables following a distribution, whilst the expectation of the expression is a single real number.
  
  However, the conversion works in the context of the outer expection, because the conditional part $S\_{t+1},A\_{t+1}=\pi'(S\_{t+1})$ matches the outer expectation being $\mathop{\mathbb{E}\_{\pi'}}$ i.e. the expectation over action choices made using $\pi'$. So the inner expectation, and its condition can simply be dropped because the condition on choosing the action using the new improved policy is the same as the outer expectation, just using different notation.
  
  It is important that here the inner expectation is simply added to other terms. If it was multiplied by a variable term that it was dependent upon, the expression could not be unpacked from the inner expectation without affecting the result.
  
  > **Comments:**
  > **username_0**: If so, I think the random variables $S_t$ and $A_t$ are the same as $S_{t+1}$ and $A_{t+1}$. So, the conditions of inner and other expectations are the same, so they can be dropped from inner expectation.  
  > **username_1**: @teslo No those are different variables, the time step index difference is important. The reason it's the same condition is due to *how* those variables have been chosen. If it's not clear how they relate to the improved policy, and not the original one, I could add more about that to the answer  
  > **username_0**: May I ask more details?  
  > **username_1**: @username_0: Sure, but could you please say where you are stuck, or what you don't understand, so I can add the right details that help you?  
  > **username_0**: Oh, I can't understand how they relate to the improved policy, and not the original one  
  
  ---
  
  