Answers.Body: <p>Your highlighted equation 2 is incorporating the probabilities under policy <span class="math-container">$\pi'$</span> which as stated in your referenced page is the <span class="math-container">$\epsilon$</span>-greedy policy with respect to <span class="math-container">$q_{\pi}$</span>, the coefficients (<span class="math-container">$\frac{\epsilon}{|A(s)|},(1-\epsilon)$</span>) in each summation term are the probabilities.</p>
  <p>Here to demonstrate policy improvement theorem you only really need action values of <span class="math-container">$\pi$</span> as seen in your referenced derivation, the goal is to compute and compare state values for any generic state <span class="math-container">$s$</span> under both policies <span class="math-container">$\pi$</span> and <span class="math-container">$\pi'$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-09T08:33:33.370Z
Answers.Text: Thank you very much! So the second equation is just a transformation of the first equation to the $\epsilon$-greedy version under the same policy $\pi'$ , which has to be equal or greater than the $\epsilon$-soft equation 3 under policy $\pi$.
Answers.UserDisplayName: bake_thi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-09T13:20:40.037Z
Answers.Text: @bake_thi that sounds right.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-07T06:09:29.410Z
Answers.LastActivityDate: 2024-04-07T06:09:29.410Z
Answers.UserDisplayName: cinch
Body: <p>I am currently studying the equations 5.2 in <a href="http://incompleteideas.net/book/RLbook2020.pdf#page=122" rel="nofollow noreferrer">Reinforcement Learning An Introduction By Sutton and Barto on page 101</a>.<br />
  I want to comprehent the proof by a simple example:</p>
  <p>Having only one State with two Actions, each having an Action Value and a Probability of Selection under Policy <span class="math-container">$\pi$</span> and <span class="math-container">$\pi'$</span></p>
  <p>I calculated the action-values for each step with an <span class="math-container">$\epsilon = 0.1$</span>. The equations hold true for part 1 and 3. But Equation 2 is not the same as Equation 1 as suggested.</p>
  <p>Why is <span class="math-container">$\Sigma q_\pi(s,a)$</span> in Equation 2 only summing all Action Values without incomporating the Probabilities under policy <span class="math-container">$\pi$</span>?</p>
  <p>Do I need different Action Values for <span class="math-container">$\pi'$</span> and <span class="math-container">$\pi$</span>?</p>
  <p><a href="https://i.sstatic.net/WFNpT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WFNpT.png" alt="enter image description here" /></a></p>
  
Tags: <reinforcement-learning><proofs><monte-carlo-methods><epsilon-greedy-policy><policy-improvement-theorem>
Title: Understanding the policy improvement theorem for Monte Carlo Control without Exploring Starts
UserDisplayName: bake_thi
ThreadText: # Understanding the policy improvement theorem for Monte Carlo Control without Exploring Starts
  
  (Asked by: username_0 on 2024-04-06)
  
  I am currently studying the equations 5.2 in [Reinforcement Learning An Introduction By Sutton and Barto on page 101](http://incompleteideas.net/book/RLbook2020.pdf#page=122).  
  I want to comprehent the proof by a simple example:
  
  Having only one State with two Actions, each having an Action Value and a Probability of Selection under Policy $\pi$ and $\pi'$
  
  I calculated the action-values for each step with an $\epsilon = 0.1$. The equations hold true for part 1 and 3. But Equation 2 is not the same as Equation 1 as suggested.
  
  Why is $\Sigma q\_\pi(s,a)$ in Equation 2 only summing all Action Values without incomporating the Probabilities under policy $\pi$?
  
  Do I need different Action Values for $\pi'$ and $\pi$?
  
  [![enter image description here](https://i.sstatic.net/WFNpT.png)](https://i.sstatic.net/WFNpT.png)
  
  **Tags:** reinforcement-learning, proofs, monte-carlo-methods, epsilon-greedy-policy, policy-improvement-theorem
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-04-07. Score: 1 | Accepted answer
  
  Your highlighted equation 2 is incorporating the probabilities under policy $\pi'$ which as stated in your referenced page is the $\epsilon$-greedy policy with respect to $q\_{\pi}$, the coefficients ($\frac{\epsilon}{|A(s)|},(1-\epsilon)$) in each summation term are the probabilities.
  
  Here to demonstrate policy improvement theorem you only really need action values of $\pi$ as seen in your referenced derivation, the goal is to compute and compare state values for any generic state $s$ under both policies $\pi$ and $\pi'$.
  
  > **Comments:**
  > **username_0**: Thank you very much! So the second equation is just a transformation of the first equation to the $\epsilon$-greedy version under the same policy $\pi'$ , which has to be equal or greater than the $\epsilon$-soft equation 3 under policy $\pi$.  
  > **username_1**: @bake_thi that sounds right.  
  
  ---
  
  