<p>Your highlighted equation 2 is incorporating the probabilities under policy <span class="math-container">$\pi'$</span> which as stated in your referenced page is the <span class="math-container">$\epsilon$</span>-greedy policy with respect to <span class="math-container">$q_{\pi}$</span>, the coefficients (<span class="math-container">$\frac{\epsilon}{|A(s)|},(1-\epsilon)$</span>) in each summation term are the probabilities.</p>
<p>Here to demonstrate policy improvement theorem you only really need action values of <span class="math-container">$\pi$</span> as seen in your referenced derivation, the goal is to compute and compare state values for any generic state <span class="math-container">$s$</span> under both policies <span class="math-container">$\pi$</span> and <span class="math-container">$\pi'$</span>.</p>

CC BY-SA 4.0
2024-04-09T08:33:33.370Z
Thank you very much! So the second equation is just a transformation of the first equation to the $\epsilon$-greedy version under the same policy $\pi'$ , which has to be equal or greater than the $\epsilon$-soft equation 3 under policy $\pi$.
bake_thi
CC BY-SA 4.0
2024-04-09T13:20:40.037Z
@bake_thi that sounds right.
cinch
CC BY-SA 4.0
2024-04-07T06:09:29.410Z
2024-04-07T06:09:29.410Z
cinch
<p>I am currently studying the equations 5.2 in <a href="http://incompleteideas.net/book/RLbook2020.pdf#page=122" rel="nofollow noreferrer">Reinforcement Learning An Introduction By Sutton and Barto on page 101</a>.<br />
I want to comprehent the proof by a simple example:</p>
<p>Having only one State with two Actions, each having an Action Value and a Probability of Selection under Policy <span class="math-container">$\pi$</span> and <span class="math-container">$\pi'$</span></p>
<p>I calculated the action-values for each step with an <span class="math-container">$\epsilon = 0.1$</span>. The equations hold true for part 1 and 3. But Equation 2 is not the same as Equation 1 as suggested.</p>
<p>Why is <span class="math-container">$\Sigma q_\pi(s,a)$</span> in Equation 2 only summing all Action Values without incomporating the Probabilities under policy <span class="math-container">$\pi$</span>?</p>
<p>Do I need different Action Values for <span class="math-container">$\pi'$</span> and <span class="math-container">$\pi$</span>?</p>
<p><a href="https://i.sstatic.net/WFNpT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WFNpT.png" alt="enter image description here" /></a></p>

<reinforcement-learning><proofs><monte-carlo-methods><epsilon-greedy-policy><policy-improvement-theorem>
Understanding the policy improvement theorem for Monte Carlo Control without Exploring Starts
bake_thi
# Understanding the policy improvement theorem for Monte Carlo Control without Exploring Starts

(Asked by: username_0 on 2024-04-06)

I am currently studying the equations 5.2 in [Reinforcement Learning An Introduction By Sutton and Barto on page 101](http://incompleteideas.net/book/RLbook2020.pdf#page=122).  
I want to comprehent the proof by a simple example:

Having only one State with two Actions, each having an Action Value and a Probability of Selection under Policy $\pi$ and $\pi'$

I calculated the action-values for each step with an $\epsilon = 0.1$. The equations hold true for part 1 and 3. But Equation 2 is not the same as Equation 1 as suggested.

Why is $\Sigma q\_\pi(s,a)$ in Equation 2 only summing all Action Values without incomporating the Probabilities under policy $\pi$?

Do I need different Action Values for $\pi'$ and $\pi$?

[![enter image description here](https://i.sstatic.net/WFNpT.png)](https://i.sstatic.net/WFNpT.png)

**Tags:** reinforcement-learning, proofs, monte-carlo-methods, epsilon-greedy-policy, policy-improvement-theorem


---

## 1 answers

### Answer by username_1 on 2024-04-07. Score: 1 | Accepted answer

Your highlighted equation 2 is incorporating the probabilities under policy $\pi'$ which as stated in your referenced page is the $\epsilon$-greedy policy with respect to $q\_{\pi}$, the coefficients ($\frac{\epsilon}{|A(s)|},(1-\epsilon)$) in each summation term are the probabilities.

Here to demonstrate policy improvement theorem you only really need action values of $\pi$ as seen in your referenced derivation, the goal is to compute and compare state values for any generic state $s$ under both policies $\pi$ and $\pi'$.

> **Comments:**
> **username_0**: Thank you very much! So the second equation is just a transformation of the first equation to the $\epsilon$-greedy version under the same policy $\pi'$ , which has to be equal or greater than the $\epsilon$-soft equation 3 under policy $\pi$.  
> **username_1**: @username_0 that sounds right.  

---

