Answers.Body: <p>I think this equation answer your question:
  <span class="math-container">$$ q_{\pi^{i}}(s,\pi^{i+1}(s)) = \mathbf{E}[q_{\pi^{i}}(s,\pi^{i+1}(s))] = \sum_{a \in A}\pi^{i+1}(a|s)q_{\pi^{i}}(s,a)$$</span></p>
  
  <p>value of the Q while taking action from policy <span class="math-container">$\pi^{i+1}$</span> and thereafter following the policy <span class="math-container">$\pi^{i}$</span> is equal to the expected q value while taking action from policy <span class="math-container">$\pi^{i+1}$</span> and thereafter following the policy <span class="math-container">$\pi^{i}$</span>. And for the second part of your question the answer is:</p>
  
  <p><span class="math-container">$$ V_{\pi^{i}}(s) = q_{\pi^{i}}(s,\pi^{i}(s))$$</span></p>
  
  <p>state Value function following the policy <span class="math-container">$\pi^{i}$</span> is the same as action-value function while taking action from policy <span class="math-container">$\pi^{i}$</span> and thereafter following the policy <span class="math-container">$\pi^{i}$</span>. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-26T15:00:44.207Z
Answers.LastActivityDate: 2020-04-26T15:36:03.217Z
Answers.LastEditDate: 2020-04-26T15:36:03.217Z
Answers.UserDisplayName: Swakshar Deb
Body: <p>I was going through university slides and this particular slide is trying to prove that in a Monte Carlo Policy Iteration algorithm using an epsilon-greedy policy, the state Values (V-Values) are monotonically improving.</p>
  
  <p><a href="https://i.sstatic.net/MIV5O.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MIV5O.png" alt="enter image description here"></a></p>
  
  <p>My question is about the first line of computation.</p>
  
  <p><a href="https://i.sstatic.net/ctrJD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ctrJD.png" alt="enter image description here"></a></p>
  
  <p>Isn't this actually the formula for the expected value of Q? It is calculating a probability of occurrence following the policy times actual Q values, then doing the summation.</p>
  
  <p>If that is the case, could you help me understand the relationship between the expected value of Q and the expected value of V ?</p>
  
  <p>Also, if above is true, in a real world scenario, depending on how many episodes we sample and on stochasticity, does it mean that the V values of the new policy could be worse than the V values of the old policy ?</p>
  
Tags: <reinforcement-learning><proofs><monte-carlo-methods><policy-iteration><policy-improvement>
Title: Monte Carlo epsilon-greedy Policy Iteration: monotonic improvement for all cases or for the expected value?
UserDisplayName: devidduma
ThreadText: # Monte Carlo epsilon-greedy Policy Iteration: monotonic improvement for all cases or for the expected value?
  
  (Asked by: username_0 on 2020-04-25)
  
  I was going through university slides and this particular slide is trying to prove that in a Monte Carlo Policy Iteration algorithm using an epsilon-greedy policy, the state Values (V-Values) are monotonically improving.
  
  [![enter image description here](https://i.sstatic.net/MIV5O.png)](https://i.sstatic.net/MIV5O.png)
  
  My question is about the first line of computation.
  
  [![enter image description here](https://i.sstatic.net/ctrJD.png)](https://i.sstatic.net/ctrJD.png)
  
  Isn't this actually the formula for the expected value of Q? It is calculating a probability of occurrence following the policy times actual Q values, then doing the summation.
  
  If that is the case, could you help me understand the relationship between the expected value of Q and the expected value of V ?
  
  Also, if above is true, in a real world scenario, depending on how many episodes we sample and on stochasticity, does it mean that the V values of the new policy could be worse than the V values of the old policy ?
  
  **Tags:** reinforcement-learning, proofs, monte-carlo-methods, policy-iteration, policy-improvement
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-26. Score: 1 | Accepted answer
  
  I think this equation answer your question:
  $$ q\_{\pi^{i}}(s,\pi^{i+1}(s)) = \mathbf{E}[q\_{\pi^{i}}(s,\pi^{i+1}(s))] = \sum\_{a \in A}\pi^{i+1}(a|s)q\_{\pi^{i}}(s,a)$$
  
  value of the Q while taking action from policy $\pi^{i+1}$ and thereafter following the policy $\pi^{i}$ is equal to the expected q value while taking action from policy $\pi^{i+1}$ and thereafter following the policy $\pi^{i}$. And for the second part of your question the answer is:
  
  $$ V\_{\pi^{i}}(s) = q\_{\pi^{i}}(s,\pi^{i}(s))$$
  
  state Value function following the policy $\pi^{i}$ is the same as action-value function while taking action from policy $\pi^{i}$ and thereafter following the policy $\pi^{i}$.
  
  
  ---
  
  