Answers.Body: <p>In the standard policy iteration algorithm presented in Sutton and Barto's book, you alternate between a policy evaluation (PE) step and a policy improvement (PI) step (i.e. PE, PI, PE, PI, PE, PI, PE, ...). However, in general, you don't have to follow this alternation <em>strictly</em> in order to converge (in the limit) to the optimal policy. For example, value iteration (VI) is an example of a <em>truncated</em> policy iteration that still converges to the optimal policy. </p>
  
  <p>The term <a href="http://incompleteideas.net/book/first/ebook/node46.html" rel="nofollow noreferrer"><em>generalized policy iteration</em> (GPI)</a> refers to all algorithms based on policy iteration, such as value iteration, that alternate <em>in some order</em> PI and PE, and that are guaranteed to converge to the optimal policy, provided PE and PI are executed enough times.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-25T17:05:26.253Z
Answers.LastActivityDate: 2020-04-25T17:10:55.217Z
Answers.LastEditDate: 2020-04-25T17:10:55.217Z
Answers.UserDisplayName: nbro
Body: <p>I am reading Sutton and Barto's material now. I know value iteration, which is an iterative algorithm taking the maximum value of adjacent states, and policy iteration. But what is generalized policy iteration? </p>
  
Tags: <reinforcement-learning><definitions><value-iteration><policy-iteration>
Title: What is generalized policy iteration?
UserDisplayName: dato nefaridze
ThreadText: # What is generalized policy iteration?
  
  (Asked by: username_0 on 2020-04-25)
  
  I am reading Sutton and Barto's material now. I know value iteration, which is an iterative algorithm taking the maximum value of adjacent states, and policy iteration. But what is generalized policy iteration?
  
  **Tags:** reinforcement-learning, definitions, value-iteration, policy-iteration
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-25. Score: 3
  
  In the standard policy iteration algorithm presented in Sutton and Barto's book, you alternate between a policy evaluation (PE) step and a policy improvement (PI) step (i.e. PE, PI, PE, PI, PE, PI, PE, ...). However, in general, you don't have to follow this alternation *strictly* in order to converge (in the limit) to the optimal policy. For example, value iteration (VI) is an example of a *truncated* policy iteration that still converges to the optimal policy.
  
  The term [*generalized policy iteration* (GPI)](http://incompleteideas.net/book/first/ebook/node46.html) refers to all algorithms based on policy iteration, such as value iteration, that alternate *in some order* PI and PE, and that are guaranteed to converge to the optimal policy, provided PE and PI are executed enough times.
  
  
  ---
  
  