Answers.Body: <p><strong>Value Iteration Vs Policy Iteration</strong></p>
  <p>Below is the list of differences &amp; similarities between value iteration and policy iteration</p>
  <p><strong>Differences</strong></p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th></th>
  <th>Value Iteration</th>
  <th>Policy Iteration</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td><strong>Architecture</strong></td>
  <td><a href="https://i.sstatic.net/0RbG4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0RbG4.png" alt="Value iteration network" /></a><a href="https://papers.nips.cc/paper/2016/file/c21002f464c5fc5bee3b98ced83963b8-Paper.pdf" rel="nofollow noreferrer">VI Architecture reference</a></td>
  <td><a href="https://i.sstatic.net/fxEm6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fxEm6.png" alt="Policy iteration architecture" /></a> <a href="https://users.cs.duke.edu/%7Eparr/jmlr03.pdf" rel="nofollow noreferrer">PI Architecture Reference</a></td>
  </tr>
  <tr>
  <td><strong>Execution starts with a</strong></td>
  <td>random value function</td>
  <td>random policy</td>
  </tr>
  <tr>
  <td><strong>Algorithm</strong></td>
  <td>simpler</td>
  <td>complex</td>
  </tr>
  <tr>
  <td><strong>Computation costs</strong></td>
  <td>more expensive</td>
  <td>cheaper</td>
  </tr>
  <tr>
  <td><strong>Execution time</strong></td>
  <td>Slower</td>
  <td>Faster</td>
  </tr>
  <tr>
  <td><strong>No of Iterations to converge</strong></td>
  <td>significantly more</td>
  <td>takes fewer iteration to converge</td>
  </tr>
  <tr>
  <td><strong>Guaranteed to converge</strong></td>
  <td>Yes</td>
  <td>Yes</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p><strong>Similarities</strong></p>
  <ul>
  <li>both are dynamic programming algorithms</li>
  <li>both employ variations of Bellman updates</li>
  <li>Both exploit one-step look-ahead</li>
  <li>Both algorithms are guaranteed to converge to an optimal policy in the end</li>
  </ul>
  <p>Policy iteration is reported to conclude faster than value iteration</p>
  <p><strong>USAGE PREFERENCE</strong></p>
  <p>As mentioned earlier in the difference, the main advantage for using Policy iteration over value iteration is its ability to conclude faster with fewer iterations thereby reducing its computation costs and execution time.</p>
  <p><strong>REFERENCES</strong></p>
  <ol>
  <li>Research papers
  <ul>
  <li><a href="https://www.ics.uci.edu/%7Edechter/publications/r42a-mdp_report.pdf" rel="nofollow noreferrer">Value iteration and policy iteration
  algorithms
  for Markov decision problem</a></li>
  <li><a href="https://arxiv.org/ftp/arxiv/papers/1301/1301.6718.pdf" rel="nofollow noreferrer">On the Complexity of Policy Iteration </a></li>
  </ul>
  </li>
  <li>Book
  <ul>
  <li>Artificial Intelligence: A Modern Approach, by Peter Norvig and Stuart J. Russell Chapter 17 Making Complex decisions</li>
  </ul>
  </li>
  <li>Architecture
  <ul>
  <li><a href="https://papers.nips.cc/paper/2016/file/c21002f464c5fc5bee3b98ced83963b8-Paper.pdf" rel="nofollow noreferrer">VI Architecture reference</a></li>
  <li><a href="https://users.cs.duke.edu/%7Eparr/jmlr03.pdf" rel="nofollow noreferrer">PI Architecture Reference</a></li>
  </ul>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-12T06:45:00.063Z
Answers.Text: The presentation/readability of this answer is good, but this does not answer the question "When to use one over the other?". Moreover, you do not explain 1. why value iteration is slower and more expensive than policy iteration and 2. who claimed that (i.e. cite something reliable). In any case, I don't understand why you say that policy iteration is "complex". Both algorithms are simple enough.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-26T18:52:21.960Z
Answers.Text: I added the notice because you're not really citing anything that supports the claim e.g. that policy iteration "takes fewer iteration to converge". Provide inline references/citations to research papers or books that support those claims (e.g. you can use this mathjax style `[\[1\][1]]` and it will be shown as `[1]`. Moreover, your last edit actually made things worse. For example, the diagram for value iteration is wrong. You're confused. Please, do not provide wrong information to the site. If you're not reasonably familiar with the concepts, don't provide an answer just by guessing.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-29T14:09:54.457Z
Answers.Text: @nbro: The claim that policy iteration is "complex" is made, though not explained, in [this tutorial](https://www.baeldung.com/cs/ml-value-iteration-vs-policy-iteration), on which this answer appears to be very closely based.
Answers.UserDisplayName: Scortchi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-29T14:16:08.083Z
Answers.Text: @Scortchi-ReinstateMonica Yes, thanks. It looks like this answer is based on that tutorial. The initial table provided by the OP is really analogous to the one given in that tutorial. This answer is now more misleading than originally. In the last edit, the OP provided the diagram of "Value Iteration Network", which is not "value iteration". We may need to delete this answer to avoid spreading misinformation, although some info in this answer may be correct. More importantly, this does not even answer the question "when to use one over the other?". Not sure why it was accepted.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-12T00:38:04.370Z
Answers.LastActivityDate: 2021-12-26T18:41:35.750Z
Answers.LastEditDate: 2021-12-26T18:41:35.750Z
Answers.UserDisplayName: Archana David
Body: <p>Both value iteration and policy iteration are General Policy Iteration (GPI) algorithms.  However, they differ in the mechanics of their updates.  Policy Iteration seeks to first find a completed value function for a policy, then derive the Q function from this and improve the policy greedily from this Q.  Meanwhile, Value Iteration uses a truncated V function to then obtain Q updates, only returning the policy once V has converged.</p>
  <p>What are the inherent advantages of using one over the other in a practical setting?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-11T22:20:04.237Z
Comments.Text: I think there's a typo here "V function to then obtain Q updates". What do you mean by "Q updates"? Maybe you mean "to obtain policy updates".
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><comparison><q-learning><value-functions><policy-iteration>
Title: When to use Value Iteration vs. Policy Iteration
UserDisplayName: SeeDerekEngineer
ThreadText: # When to use Value Iteration vs. Policy Iteration
  
  (Asked by: username_0 on 2021-10-11)
  
  Both value iteration and policy iteration are General Policy Iteration (GPI) algorithms. However, they differ in the mechanics of their updates. Policy Iteration seeks to first find a completed value function for a policy, then derive the Q function from this and improve the policy greedily from this Q. Meanwhile, Value Iteration uses a truncated V function to then obtain Q updates, only returning the policy once V has converged.
  
  What are the inherent advantages of using one over the other in a practical setting?
  
  **Tags:** reinforcement-learning, comparison, q-learning, value-functions, policy-iteration
  > **Comments:**
  > **username_1**: I think there's a typo here "V function to then obtain Q updates". What do you mean by "Q updates"? Maybe you mean "to obtain policy updates".  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-10-12. Score: 1 | Accepted answer
  
  **Value Iteration Vs Policy Iteration**
  
  Below is the list of differences & similarities between value iteration and policy iteration
  
  **Differences**
  
  |  | Value Iteration | Policy Iteration |
  | --- | --- | --- |
  | **Architecture** | [Value iteration network](https://i.sstatic.net/0RbG4.png)[VI Architecture reference](https://papers.nips.cc/paper/2016/file/c21002f464c5fc5bee3b98ced83963b8-Paper.pdf) | [Policy iteration architecture](https://i.sstatic.net/fxEm6.png) [PI Architecture Reference](https://users.cs.duke.edu/%7Eparr/jmlr03.pdf) |
  | **Execution starts with a** | random value function | random policy |
  | **Algorithm** | simpler | complex |
  | **Computation costs** | more expensive | cheaper |
  | **Execution time** | Slower | Faster |
  | **No of Iterations to converge** | significantly more | takes fewer iteration to converge |
  | **Guaranteed to converge** | Yes | Yes |
  
  **Similarities**
  
  * both are dynamic programming algorithms
  * both employ variations of Bellman updates
  * Both exploit one-step look-ahead
  * Both algorithms are guaranteed to converge to an optimal policy in the end
  
  Policy iteration is reported to conclude faster than value iteration
  
  **USAGE PREFERENCE**
  
  As mentioned earlier in the difference, the main advantage for using Policy iteration over value iteration is its ability to conclude faster with fewer iterations thereby reducing its computation costs and execution time.
  
  **REFERENCES**
  
  1. Research papers
     * [Value iteration and policy iteration
       algorithms
       for Markov decision problem](https://www.ics.uci.edu/%7Edechter/publications/r42a-mdp_report.pdf)
     * [On the Complexity of Policy Iteration](https://arxiv.org/ftp/arxiv/papers/1301/1301.6718.pdf)
  2. Book
     * Artificial Intelligence: A Modern Approach, by Peter Norvig and Stuart J. Russell Chapter 17 Making Complex decisions
  3. Architecture
     * [VI Architecture reference](https://papers.nips.cc/paper/2016/file/c21002f464c5fc5bee3b98ced83963b8-Paper.pdf)
     * [PI Architecture Reference](https://users.cs.duke.edu/%7Eparr/jmlr03.pdf)
  
  > **Comments:**
  > **username_1**: The presentation/readability of this answer is good, but this does not answer the question "When to use one over the other?". Moreover, you do not explain 1. why value iteration is slower and more expensive than policy iteration and 2. who claimed that (i.e. cite something reliable). In any case, I don't understand why you say that policy iteration is "complex". Both algorithms are simple enough.  
  > **username_1**: I added the notice because you're not really citing anything that supports the claim e.g. that policy iteration "takes fewer iteration to converge". Provide inline references/citations to research papers or books that support those claims (e.g. you can use this mathjax style `[\[1\][1]]` and it will be shown as `[1]`. Moreover, your last edit actually made things worse. For example, the diagram for value iteration is wrong. You're confused. Please, do not provide wrong information to the site. If you're not reasonably familiar with the concepts, don't provide an answer just by guessing.  
  > **username_3**: @nbro: The claim that policy iteration is "complex" is made, though not explained, in [this tutorial](https://www.baeldung.com/cs/ml-value-iteration-vs-policy-iteration), on which this answer appears to be very closely based.  
  > **username_1**: @Scortchi-ReinstateMonica Yes, thanks. It looks like this answer is based on that tutorial. The initial table provided by the OP is really analogous to the one given in that tutorial. This answer is now more misleading than originally. In the last edit, the OP provided the diagram of "Value Iteration Network", which is not "value iteration". We may need to delete this answer to avoid spreading misinformation, although some info in this answer may be correct. More importantly, this does not even answer the question "when to use one over the other?". Not sure why it was accepted.  
  
  ---
  
  