Answers.Body: <p>The Policy Iteration algorithm (given in the question) is model-based.</p>
  <p>However, note that there exist methods that fall into the <em>Generalized Policy Iteration</em> category, such as SARSA, which are model-free.</p>
  <blockquote>
  <p>From what I understand, policy iteration is a model-free algorithm</p>
  </blockquote>
  <p>Maybe this was referring to generalized policy iteration methods.</p>
  <hr />
  <p>(Answer based on comments from @Neil Slater.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-21T12:59:51.080Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/130724/discussion-on-answer-by-daswesen-how-can-the-policy-iteration-algorithm-be-model).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-20T09:42:40.547Z
Answers.LastActivityDate: 2021-10-21T17:19:11.823Z
Answers.LastEditDate: 2021-10-21T17:19:11.823Z
Answers.UserDisplayName: dasWesen
Answers.Body: <p>Everything you say in your post is correct, apart from the wrong assumption that policy iteration is model-free. PI is a model-based algorithm because of the reasons you're mentioning. </p>
  
  <p>See <a href="https://ai.stackexchange.com/a/8820/2444">my answer</a> to the question <a href="https://ai.stackexchange.com/q/4456/2444">What&#39;s the difference between model-free and model-based reinforcement learning?</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-11T16:25:37.447Z
Answers.LastActivityDate: 2020-03-11T16:25:37.447Z
Answers.UserDisplayName: nbro
Body: <p>I'm actually trying to understand the policy iteration in the context of RL. I read an article presenting it and, at some point, a pseudo-code of the algorithm is given : <a href="https://i.sstatic.net/YcKxP.png" rel="noreferrer"><img src="https://i.sstatic.net/YcKxP.png" alt="enter image description here"></a></p>
  
  <p>What I can't understand is this line : </p>
  
  <p><a href="https://i.sstatic.net/IRJAR.png" rel="noreferrer"><img src="https://i.sstatic.net/IRJAR.png" alt="enter image description here"></a></p>
  
  <p>From what I understand, policy iteration is a model-free algorithm, which means that it doesn't need to know the environment's dynamics. But, in this line, we need <span class="math-container">$p(s',r \mid s, \pi(s))$</span> (which in my understanding is the transition function of the MDP that gave us the probability of landing in the state <span class="math-container">$s'$</span> knowing previous <span class="math-container">$s$</span> state and the action taken) to compute <span class="math-container">$V(s)$</span>. So I don't understand how we can compute <span class="math-container">$V(s)$</span> with the quantity <span class="math-container">$p(s',r \mid s, \pi(s))$</span> since it is a parameter of the environment. </p>
  
Tags: <reinforcement-learning><comparison><model-based-methods><model-free-methods><policy-iteration>
Title: How can the policy iteration algorithm be model-free if it uses the transition probabilities?
UserDisplayName: Samuel Beaussant
ThreadText: # How can the policy iteration algorithm be model-free if it uses the transition probabilities?
  
  (Asked by: username_0 on 2020-03-11)
  
  I'm actually trying to understand the policy iteration in the context of RL. I read an article presenting it and, at some point, a pseudo-code of the algorithm is given : [![enter image description here](https://i.sstatic.net/YcKxP.png)](https://i.sstatic.net/YcKxP.png)
  
  What I can't understand is this line :
  
  [![enter image description here](https://i.sstatic.net/IRJAR.png)](https://i.sstatic.net/IRJAR.png)
  
  From what I understand, policy iteration is a model-free algorithm, which means that it doesn't need to know the environment's dynamics. But, in this line, we need $p(s',r \mid s, \pi(s))$ (which in my understanding is the transition function of the MDP that gave us the probability of landing in the state $s'$ knowing previous $s$ state and the action taken) to compute $V(s)$. So I don't understand how we can compute $V(s)$ with the quantity $p(s',r \mid s, \pi(s))$ since it is a parameter of the environment.
  
  **Tags:** reinforcement-learning, comparison, model-based-methods, model-free-methods, policy-iteration
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-03-11. Score: 4 | Accepted answer
  
  Everything you say in your post is correct, apart from the wrong assumption that policy iteration is model-free. PI is a model-based algorithm because of the reasons you're mentioning.
  
  See [my answer](https://ai.stackexchange.com/a/8820/2444) to the question [What's the difference between model-free and model-based reinforcement learning?](https://ai.stackexchange.com/q/4456/2444).
  
  
  ---
  
  ### Answer by username_1 on 2021-10-20. Score: 0
  
  The Policy Iteration algorithm (given in the question) is model-based.
  
  However, note that there exist methods that fall into the *Generalized Policy Iteration* category, such as SARSA, which are model-free.
  
  > From what I understand, policy iteration is a model-free algorithm
  
  Maybe this was referring to generalized policy iteration methods.
  
  ---
  
  (Answer based on comments from @Neil Slater.)
  
  > **Comments:**
  > **username_2**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/130724/discussion-on-answer-by-daswesen-how-can-the-policy-iteration-algorithm-be-model).  
  
  ---
  
  