Answers.Body: <p>This lingering question is closely related to your recent question <a href="https://ai.stackexchange.com/questions/45092/how-to-derive-value-iteration-from-policy-iteration">how to derive value iteration from policy iteration</a>. Policy evaluation (PE) required in policy iteration method needs exact or approximated state or action value evaluation during each iteration, since it's based on the idea of policy improvement theorem (PIT) and Bellman equation, so it's the original and pure (not <em>generalized</em>) policy iteration. And you're right it could be said as a form of GPI.</p>
  <p>On the other extreme end, value iteration is based on the idea of solving Bellman Optimal equation, on each policy update (PU) step we move the value function toward the exact <span class="math-container">$q_{\pi_k}$</span>, but we do not expect to actually get close except over many steps. Therefore it's a kind of <em>generalized</em> policy iteration.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-22T01:57:26.240Z
Answers.Text: I was asking difference between (i) and (ii).
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-22T02:07:23.907Z
Answers.Text: My first section above is for your case (i), and second section is for the extreme one-sweep case (ii) which is rightfully called GPI as I've explained during so called 'evaluation' step we do not expect to actually get close to true state/action values except over many steps, therefore it's a kind of generalized policy iteration.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-22T00:34:24.473Z
Answers.LastActivityDate: 2024-03-22T00:34:24.473Z
Answers.UserDisplayName: cinch
Body: <p>Here, <a href="https://lrscy.github.io/2020/07/09/Coursera-Reinforcement-Learning-Course2-Week2-Notes" rel="nofollow noreferrer">https://lrscy.github.io/2020/07/09/Coursera-Reinforcement-Learning-Course2-Week2-Notes</a></p>
  <p>(See the &quot;Monte Carlo Control&quot; and then &quot;Solutions of Two Assumptions&quot;
  sections)</p>
  <p>two approaches in solving &quot;Infinite number of episodes&quot; in Monte Carlo Control with exploring starts are given as follows:</p>
  <p>(i) One is to hold firm to the idea of approximating <span class="math-container">$q_{\pi_k}$</span>  in each policy evaluation. However, it is also likely to require far too many episodes to be useful in practice on any but the smallest problems.</p>
  <p>(ii) Another one is similar to the idea of GPI. On each evaluation step we move the value function toward <span class="math-container">$q_{\pi_k}$</span> , but we do not expect to actually get close except over many steps. One extreme form of the idea is to alternatively apply policy improvement and policy evaluation.</p>
  <p>I did not understand the difference between (i) and (ii). For me, (i) is also a form of GPI.</p>
  
Tags: <reinforcement-learning>
Title: What is the difference between two approaches in solving "Infinite number of episodes" in Monte Carlo Control with exploring starts?
UserDisplayName: DSPinfinity
ThreadText: # What is the difference between two approaches in solving "Infinite number of episodes" in Monte Carlo Control with exploring starts?
  
  (Asked by: username_0 on 2024-03-21)
  
  Here, <https://lrscy.github.io/2020/07/09/Coursera-Reinforcement-Learning-Course2-Week2-Notes>
  
  (See the "Monte Carlo Control" and then "Solutions of Two Assumptions"
  sections)
  
  two approaches in solving "Infinite number of episodes" in Monte Carlo Control with exploring starts are given as follows:
  
  (i) One is to hold firm to the idea of approximating $q\_{\pi\_k}$ in each policy evaluation. However, it is also likely to require far too many episodes to be useful in practice on any but the smallest problems.
  
  (ii) Another one is similar to the idea of GPI. On each evaluation step we move the value function toward $q\_{\pi\_k}$ , but we do not expect to actually get close except over many steps. One extreme form of the idea is to alternatively apply policy improvement and policy evaluation.
  
  I did not understand the difference between (i) and (ii). For me, (i) is also a form of GPI.
  
  **Tags:** reinforcement-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-03-22. Score: 2 | Accepted answer
  
  This lingering question is closely related to your recent question [how to derive value iteration from policy iteration](https://ai.stackexchange.com/questions/45092/how-to-derive-value-iteration-from-policy-iteration). Policy evaluation (PE) required in policy iteration method needs exact or approximated state or action value evaluation during each iteration, since it's based on the idea of policy improvement theorem (PIT) and Bellman equation, so it's the original and pure (not *generalized*) policy iteration. And you're right it could be said as a form of GPI.
  
  On the other extreme end, value iteration is based on the idea of solving Bellman Optimal equation, on each policy update (PU) step we move the value function toward the exact $q\_{\pi\_k}$, but we do not expect to actually get close except over many steps. Therefore it's a kind of *generalized* policy iteration.
  
  > **Comments:**
  > **username_0**: I was asking difference between (i) and (ii).  
  > **username_1**: My first section above is for your case (i), and second section is for the extreme one-sweep case (ii) which is rightfully called GPI as I've explained during so called 'evaluation' step we do not expect to actually get close to true state/action values except over many steps, therefore it's a kind of generalized policy iteration.  
  
  ---
  
  