Answers.Body: <p>Policy and value iteration both require you to, for each possible transition and each corresponding possible reward at each state, compute a statistic of <span class="math-container">$r + \gamma V(s')$</span>. In order for this to be tractable, you need for there to be at most finitely many states, actions, possible rewards, and possible transitions at each state. You also need to know the transition model. This is the case in gridworld.</p>
  <p>Gridworld is not the only example of an MDP that can be solved with policy or value iteration, but all other examples must have finite (and small enough) state and action spaces. For example, take any MDP with a known model and bounded state and action spaces of fairly low dimension. Then you can approximate the state and action spaces with a finite number of bins, each corresponding to its own &quot; discretized state/action&quot;. With smooth enough dynamics and enough bins, you'll be able to solve the MDP with policy/value iteration on the discretized spaces.</p>
  <p>In many interesting RL problems though,</p>
  <ol>
  <li>You don't know the transition model, and/or</li>
  <li>The state space, action space, and/or reward space are too large</li>
  </ol>
  <p>In these cases you wouldn't be able to compute the value function exactly, so you can't really do policy/value iteration. However, in most value based RL algorithms, the policy evaluation / policy improvement steps are approximated using sample transitions and function approximators.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-16T09:07:20.550Z
Answers.Text: Sorry, just one comment. Suppose that I am in a scenario in which I want to use value iteration, but I don't have a final state to reach. I would like to have a reward equal to zero if the agent remains in the initial state and a reward of -1 if it moves. I want to find the best state. Can I solve this with value iteration?  Thanks.
Answers.UserDisplayName: dcr
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-16T10:50:38.280Z
Answers.Text: You should be able to solve that with value iteration if the state and action spaces are finite or can be discretized.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-15T15:18:07.793Z
Answers.LastActivityDate: 2021-06-15T21:26:27.950Z
Answers.LastEditDate: 2021-06-15T21:26:27.950Z
Answers.UserDisplayName: harwiltz
Body: <p>I am trying to self learn reinforcement learning. At the moment I am focusing on <strong>policy and value iteration</strong>, and I am finding several problems and doubts.</p>
  <p>One of the main doubts is given by the fact that I can't find many diversified examples on how to implement these on python, instead I find always only the classical grid world example.</p>
  <p>So, my doubt is: <strong>Are policy and value iteration used only in grid world like scenarios, or can be used also in other contexts?</strong></p>
  
Tags: <reinforcement-learning><value-iteration><policy-iteration><dynamic-programming>
Title: Are policy and value iteration used only in grid world like scenarios?
UserDisplayName: dcr
ThreadText: # Are policy and value iteration used only in grid world like scenarios?
  
  (Asked by: username_0 on 2021-06-15)
  
  I am trying to self learn reinforcement learning. At the moment I am focusing on **policy and value iteration**, and I am finding several problems and doubts.
  
  One of the main doubts is given by the fact that I can't find many diversified examples on how to implement these on python, instead I find always only the classical grid world example.
  
  So, my doubt is: **Are policy and value iteration used only in grid world like scenarios, or can be used also in other contexts?**
  
  **Tags:** reinforcement-learning, value-iteration, policy-iteration, dynamic-programming
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-06-15. Score: 2 | Accepted answer
  
  Policy and value iteration both require you to, for each possible transition and each corresponding possible reward at each state, compute a statistic of $r + \gamma V(s')$. In order for this to be tractable, you need for there to be at most finitely many states, actions, possible rewards, and possible transitions at each state. You also need to know the transition model. This is the case in gridworld.
  
  Gridworld is not the only example of an MDP that can be solved with policy or value iteration, but all other examples must have finite (and small enough) state and action spaces. For example, take any MDP with a known model and bounded state and action spaces of fairly low dimension. Then you can approximate the state and action spaces with a finite number of bins, each corresponding to its own " discretized state/action". With smooth enough dynamics and enough bins, you'll be able to solve the MDP with policy/value iteration on the discretized spaces.
  
  In many interesting RL problems though,
  
  1. You don't know the transition model, and/or
  2. The state space, action space, and/or reward space are too large
  
  In these cases you wouldn't be able to compute the value function exactly, so you can't really do policy/value iteration. However, in most value based RL algorithms, the policy evaluation / policy improvement steps are approximated using sample transitions and function approximators.
  
  > **Comments:**
  > **username_0**: Sorry, just one comment. Suppose that I am in a scenario in which I want to use value iteration, but I don't have a final state to reach. I would like to have a reward equal to zero if the agent remains in the initial state and a reward of -1 if it moves. I want to find the best state. Can I solve this with value iteration?  Thanks.  
  > **username_1**: You should be able to solve that with value iteration if the state and action spaces are finite or can be discretized.  
  
  ---
  
  