Answers.Body: <p>Both value iteration (VI) and policy iteration (PI) algorithms are guaranteed to converge to the optimal policy, so it is expected that you get similar policies from both algorithms (if they have converged). </p>
  
  <p>However, they do this differently. VI can be seen as truncated version of PI.  </p>
  
  <p>Let me first illustrate the pseudocode of both algorithms (taken from <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">Barto and Sutton's book</a>), which I suggest you get familiar with (but you are probably already familiar with them if you implemented both algorithms).</p>
  
  <p><a href="https://i.sstatic.net/kKZx7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/kKZx7.png" alt="enter image description here"></a></p>
  
  <p>As you can see, policy iteration updates the policy multiple times, because it alternates a step of policy evaluation and a step of policy improvement, where a better policy is derived from the current best estimate of the value function.</p>
  
  <p><a href="https://i.sstatic.net/CAAu5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CAAu5.png" alt="enter image description here"></a></p>
  
  <p>On the other hand, value iteration updates the policy only once (at the end). </p>
  
  <p>In both cases, the policies are derived from the value functions in the same way. So, if you obtain similar policies, you may think that they are necessarily derived from similar <em>final</em> value functions. However, in general, this may not the case, and this is actually the motivation for the existence of value iteration, i.e. you may derive an optimal policy from an non-optimal value function.</p>
  
  <p>Barto and Sutton's book provide an example. See figure 4.1 on page 77 (<a href="http://incompleteideas.net/book/RLbook2020.pdf#page=99" rel="nofollow noreferrer">p. 99 of the pdf</a>). For completeness, here's a screenshot of the figure.</p>
  
  <p><a href="https://i.sstatic.net/DwyQL.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/DwyQL.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-21T22:46:50.760Z
Answers.Text: That cleared it up a little! Thank you! Thanks for making the question more clear too! I also have an implementation related question, in the policy iteration, `policy evaluation` subroutine how is V(s) exactly updated? I get that in `value iteration` we are taking the value of the action that gives the maximum value, but what are we doing in the `policy evaluation` case?
Answers.UserDisplayName: r4bb1t
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-21T23:16:00.893Z
Answers.Text: @PyWalker27 There are actually two possible implementations of PE: one where you update V in place and the other where you maintain two arrays. Sutton and Barto's book explain this. Have a look at chapter 4.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-21T23:18:25.557Z
Answers.Text: The algorithm above, it appears is doing in an inplace update, right?
Answers.UserDisplayName: r4bb1t
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-22T12:13:41.657Z
Answers.Text: @PyWalker27 Yes, it appears to be in-place (but I suggest you read the section about the topic of chapter 4 to confirm that this is the case). If it wasn't in-place, we should be using a different `V(s)` on the left-hand side of $\leftarrow$.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-21T21:27:19.697Z
Answers.LastActivityDate: 2020-04-21T21:35:52.143Z
Answers.LastEditDate: 2020-04-21T21:35:52.143Z
Answers.UserDisplayName: nbro
Body: <p>I am trying to implement value and policy iteration algorithms. My value function from policy iteration looks vastly different from the values from value iteration, but the policy obtained from both is very similar. How is this possible? And what could be the possible reasons for this?</p>
  
Tags: <reinforcement-learning><comparison><policies><value-iteration><policy-iteration>
Title: Why do value iteration and policy iteration obtain similar policies even though they have different value functions?
UserDisplayName: r4bb1t
ThreadText: # Why do value iteration and policy iteration obtain similar policies even though they have different value functions?
  
  (Asked by: username_0 on 2020-04-21)
  
  I am trying to implement value and policy iteration algorithms. My value function from policy iteration looks vastly different from the values from value iteration, but the policy obtained from both is very similar. How is this possible? And what could be the possible reasons for this?
  
  **Tags:** reinforcement-learning, comparison, policies, value-iteration, policy-iteration
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-21. Score: 2 | Accepted answer
  
  Both value iteration (VI) and policy iteration (PI) algorithms are guaranteed to converge to the optimal policy, so it is expected that you get similar policies from both algorithms (if they have converged).
  
  However, they do this differently. VI can be seen as truncated version of PI.
  
  Let me first illustrate the pseudocode of both algorithms (taken from [Barto and Sutton's book](http://incompleteideas.net/book/RLbook2020.pdf)), which I suggest you get familiar with (but you are probably already familiar with them if you implemented both algorithms).
  
  [![enter image description here](https://i.sstatic.net/kKZx7.png)](https://i.sstatic.net/kKZx7.png)
  
  As you can see, policy iteration updates the policy multiple times, because it alternates a step of policy evaluation and a step of policy improvement, where a better policy is derived from the current best estimate of the value function.
  
  [![enter image description here](https://i.sstatic.net/CAAu5.png)](https://i.sstatic.net/CAAu5.png)
  
  On the other hand, value iteration updates the policy only once (at the end).
  
  In both cases, the policies are derived from the value functions in the same way. So, if you obtain similar policies, you may think that they are necessarily derived from similar *final* value functions. However, in general, this may not the case, and this is actually the motivation for the existence of value iteration, i.e. you may derive an optimal policy from an non-optimal value function.
  
  Barto and Sutton's book provide an example. See figure 4.1 on page 77 ([p. 99 of the pdf](http://incompleteideas.net/book/RLbook2020.pdf#page=99)). For completeness, here's a screenshot of the figure.
  
  [![enter image description here](https://i.sstatic.net/DwyQL.png)](https://i.sstatic.net/DwyQL.png)
  
  > **Comments:**
  > **username_0**: That cleared it up a little! Thank you! Thanks for making the question more clear too! I also have an implementation related question, in the policy iteration, `policy evaluation` subroutine how is V(s) exactly updated? I get that in `value iteration` we are taking the value of the action that gives the maximum value, but what are we doing in the `policy evaluation` case?  
  > **username_1**: @PyWalker27 There are actually two possible implementations of PE: one where you update V in place and the other where you maintain two arrays. Sutton and Barto's book explain this. Have a look at chapter 4.  
  > **username_0**: The algorithm above, it appears is doing in an inplace update, right?  
  > **username_1**: @PyWalker27 Yes, it appears to be in-place (but I suggest you read the section about the topic of chapter 4 to confirm that this is the case). If it wasn't in-place, we should be using a different `V(s)` on the left-hand side of $\leftarrow$.  
  
  ---
  
  