<p>I think this equation answer your question:
<span class="math-container">$$ q_{\pi^{i}}(s,\pi^{i+1}(s)) = \mathbf{E}[q_{\pi^{i}}(s,\pi^{i+1}(s))] = \sum_{a \in A}\pi^{i+1}(a|s)q_{\pi^{i}}(s,a)$$</span></p>

<p>value of the Q while taking action from policy <span class="math-container">$\pi^{i+1}$</span> and thereafter following the policy <span class="math-container">$\pi^{i}$</span> is equal to the expected q value while taking action from policy <span class="math-container">$\pi^{i+1}$</span> and thereafter following the policy <span class="math-container">$\pi^{i}$</span>. And for the second part of your question the answer is:</p>

<p><span class="math-container">$$ V_{\pi^{i}}(s) = q_{\pi^{i}}(s,\pi^{i}(s))$$</span></p>

<p>state Value function following the policy <span class="math-container">$\pi^{i}$</span> is the same as action-value function while taking action from policy <span class="math-container">$\pi^{i}$</span> and thereafter following the policy <span class="math-container">$\pi^{i}$</span>. </p>

CC BY-SA 4.0
2020-04-26T15:00:44.207Z
2020-04-26T15:36:03.217Z
2020-04-26T15:36:03.217Z
Swakshar Deb
<p>I was going through university slides and this particular slide is trying to prove that in a Monte Carlo Policy Iteration algorithm using an epsilon-greedy policy, the state Values (V-Values) are monotonically improving.</p>

<p><a href="https://i.sstatic.net/MIV5O.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MIV5O.png" alt="enter image description here"></a></p>

<p>My question is about the first line of computation.</p>

<p><a href="https://i.sstatic.net/ctrJD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ctrJD.png" alt="enter image description here"></a></p>

<p>Isn't this actually the formula for the expected value of Q? It is calculating a probability of occurrence following the policy times actual Q values, then doing the summation.</p>

<p>If that is the case, could you help me understand the relationship between the expected value of Q and the expected value of V ?</p>

<p>Also, if above is true, in a real world scenario, depending on how many episodes we sample and on stochasticity, does it mean that the V values of the new policy could be worse than the V values of the old policy ?</p>

<reinforcement-learning><proofs><monte-carlo-methods><policy-iteration><policy-improvement>
Monte Carlo epsilon-greedy Policy Iteration: monotonic improvement for all cases or for the expected value?
devidduma
# Monte Carlo epsilon-greedy Policy Iteration: monotonic improvement for all cases or for the expected value?

(Asked by: username_0 on 2020-04-25)

I was going through university slides and this particular slide is trying to prove that in a Monte Carlo Policy Iteration algorithm using an epsilon-greedy policy, the state Values (V-Values) are monotonically improving.

[![enter image description here](https://i.sstatic.net/MIV5O.png)](https://i.sstatic.net/MIV5O.png)

My question is about the first line of computation.

[![enter image description here](https://i.sstatic.net/ctrJD.png)](https://i.sstatic.net/ctrJD.png)

Isn't this actually the formula for the expected value of Q? It is calculating a probability of occurrence following the policy times actual Q values, then doing the summation.

If that is the case, could you help me understand the relationship between the expected value of Q and the expected value of V ?

Also, if above is true, in a real world scenario, depending on how many episodes we sample and on stochasticity, does it mean that the V values of the new policy could be worse than the V values of the old policy ?

**Tags:** reinforcement-learning, proofs, monte-carlo-methods, policy-iteration, policy-improvement


---

## 1 answers

### Answer by username_1 on 2020-04-26. Score: 1 | Accepted answer

I think this equation answer your question:
$$ q\_{\pi^{i}}(s,\pi^{i+1}(s)) = \mathbf{E}[q\_{\pi^{i}}(s,\pi^{i+1}(s))] = \sum\_{a \in A}\pi^{i+1}(a|s)q\_{\pi^{i}}(s,a)$$

value of the Q while taking action from policy $\pi^{i+1}$ and thereafter following the policy $\pi^{i}$ is equal to the expected q value while taking action from policy $\pi^{i+1}$ and thereafter following the policy $\pi^{i}$. And for the second part of your question the answer is:

$$ V\_{\pi^{i}}(s) = q\_{\pi^{i}}(s,\pi^{i}(s))$$

state Value function following the policy $\pi^{i}$ is the same as action-value function while taking action from policy $\pi^{i}$ and thereafter following the policy $\pi^{i}$.


---

