<p>The Policy Iteration algorithm (given in the question) is model-based.</p>
<p>However, note that there exist methods that fall into the <em>Generalized Policy Iteration</em> category, such as SARSA, which are model-free.</p>
<blockquote>
<p>From what I understand, policy iteration is a model-free algorithm</p>
</blockquote>
<p>Maybe this was referring to generalized policy iteration methods.</p>
<hr />
<p>(Answer based on comments from @Neil Slater.)</p>

CC BY-SA 4.0
2021-10-21T12:59:51.080Z
Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/130724/discussion-on-answer-by-daswesen-how-can-the-policy-iteration-algorithm-be-model).
nbro
CC BY-SA 4.0
2021-10-20T09:42:40.547Z
2021-10-21T17:19:11.823Z
2021-10-21T17:19:11.823Z
dasWesen
<p>Everything you say in your post is correct, apart from the wrong assumption that policy iteration is model-free. PI is a model-based algorithm because of the reasons you're mentioning. </p>

<p>See <a href="https://ai.stackexchange.com/a/8820/2444">my answer</a> to the question <a href="https://ai.stackexchange.com/q/4456/2444">What&#39;s the difference between model-free and model-based reinforcement learning?</a>.</p>

CC BY-SA 4.0
2020-03-11T16:25:37.447Z
2020-03-11T16:25:37.447Z
nbro
<p>I'm actually trying to understand the policy iteration in the context of RL. I read an article presenting it and, at some point, a pseudo-code of the algorithm is given : <a href="https://i.sstatic.net/YcKxP.png" rel="noreferrer"><img src="https://i.sstatic.net/YcKxP.png" alt="enter image description here"></a></p>

<p>What I can't understand is this line : </p>

<p><a href="https://i.sstatic.net/IRJAR.png" rel="noreferrer"><img src="https://i.sstatic.net/IRJAR.png" alt="enter image description here"></a></p>

<p>From what I understand, policy iteration is a model-free algorithm, which means that it doesn't need to know the environment's dynamics. But, in this line, we need <span class="math-container">$p(s',r \mid s, \pi(s))$</span> (which in my understanding is the transition function of the MDP that gave us the probability of landing in the state <span class="math-container">$s'$</span> knowing previous <span class="math-container">$s$</span> state and the action taken) to compute <span class="math-container">$V(s)$</span>. So I don't understand how we can compute <span class="math-container">$V(s)$</span> with the quantity <span class="math-container">$p(s',r \mid s, \pi(s))$</span> since it is a parameter of the environment. </p>

<reinforcement-learning><comparison><model-based-methods><model-free-methods><policy-iteration>
How can the policy iteration algorithm be model-free if it uses the transition probabilities?
Samuel Beaussant
# How can the policy iteration algorithm be model-free if it uses the transition probabilities?

(Asked by: username_0 on 2020-03-11)

I'm actually trying to understand the policy iteration in the context of RL. I read an article presenting it and, at some point, a pseudo-code of the algorithm is given : [![enter image description here](https://i.sstatic.net/YcKxP.png)](https://i.sstatic.net/YcKxP.png)

What I can't understand is this line :

[![enter image description here](https://i.sstatic.net/IRJAR.png)](https://i.sstatic.net/IRJAR.png)

From what I understand, policy iteration is a model-free algorithm, which means that it doesn't need to know the environment's dynamics. But, in this line, we need $p(s',r \mid s, \pi(s))$ (which in my understanding is the transition function of the MDP that gave us the probability of landing in the state $s'$ knowing previous $s$ state and the action taken) to compute $V(s)$. So I don't understand how we can compute $V(s)$ with the quantity $p(s',r \mid s, \pi(s))$ since it is a parameter of the environment.

**Tags:** reinforcement-learning, comparison, model-based-methods, model-free-methods, policy-iteration


---

## 2 answers

### Answer by username_2 on 2020-03-11. Score: 4 | Accepted answer

Everything you say in your post is correct, apart from the wrong assumption that policy iteration is model-free. PI is a model-based algorithm because of the reasons you're mentioning.

See [my answer](https://ai.stackexchange.com/a/8820/2444) to the question [What's the difference between model-free and model-based reinforcement learning?](https://ai.stackexchange.com/q/4456/2444).


---

### Answer by username_1 on 2021-10-20. Score: 0

The Policy Iteration algorithm (given in the question) is model-based.

However, note that there exist methods that fall into the *Generalized Policy Iteration* category, such as SARSA, which are model-free.

> From what I understand, policy iteration is a model-free algorithm

Maybe this was referring to generalized policy iteration methods.

---

(Answer based on comments from @Neil Slater.)

> **Comments:**
> **username_2**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/130724/discussion-on-answer-by-daswesen-how-can-the-policy-iteration-algorithm-be-model).  

---

