Answers.Body: <h3>TL;DR</h3>
  <p>I don't think your understanding of the notion of plan and policy is correct. These notions are independent of the nature of the environment but you seem to think that you can only find plans in deterministic environments and you need a policy for stochastic ones.</p>
  <h3>Long answer</h3>
  <p>Your post is very confusing because apparently you're very confused and mixing a lot of concepts. I'll provide clarifications for everything that I think is confusing.</p>
  <ol>
  <li><p>The first equation is just a special case of the second, where <span class="math-container">$P(a, s, s') = 1$</span> for some <span class="math-container">$s'$</span> and <span class="math-container">$P(a, s, s') = 0$</span> for all others. So, in reality, we can just use the second (more general) equation and forget about the first. However, it's true: we can use the first equation if you know the environment is deterministic.</p>
  </li>
  <li><p>A plan is a <strong>sequence of actions</strong>. A policy <span class="math-container">$\pi$</span> is a <a href="https://ai.stackexchange.com/q/12274/2444">function that maps states to actions (or probability distributions over actions)</a>. So, given a sequence of states <span class="math-container">$s_1, \dots, s_N$</span>, a deterministic policy will give you a sequence of actions <span class="math-container">$\pi(s_1), \dots, \pi(s_N) = a_1, \dots, a_N$</span>, which can be thought of as a plan. If <span class="math-container">$\pi$</span> is stochastic, you need to sample actions, so the same policy can give you multiple different plans. For more info about the notion of a plan in RL, see <a href="https://ai.stackexchange.com/a/10616/2444">this</a> answer.</p>
  </li>
  <li><p>If you have a plan or not does not depend on whether your environment is stochastic or not. You can find plans in stochastic and deterministic environments. See <a href="https://ai.stackexchange.com/a/26900/2444">this</a> answer for more info about the notion of planning and how it's related to search.</p>
  </li>
  <li><p><a href="https://ai.stackexchange.com/q/12274/2444">There are stochastic and deterministic policies</a>. There are stochastic and deterministic environments. These are difference concepts. For example, there are stochastic environments where the optimal policy is actually deterministic and there are stochastic environment where it's not.</p>
  </li>
  <li><p>There's no difference between your first and second diagrams, with respect to the definition of a plan or policy. They both look like plans, or sequences of actions generated by a policy.</p>
  </li>
  <li><p>To plan in an MDP, you usually need <span class="math-container">$P$</span> (the transition function), even if it's deterministic. You can use a dynamic programming algorithm like <em>policy iteration</em> to find a plan. However, you could also use a reinforcement learning algorithm like Q-learning to find a policy, which will give you a plan, as defined above.</p>
  </li>
  <li><p>So, how come that Q-learning is usually not considered a planning algorithm? I think this is more a convention. We could consider all algorithms that find sequence of actions planning algorithms. However, if an algorithm needs to explore the environment by taking random actions, we don't consider it a planning algorithm. So, planning involves also some notion of <em>thinking ahead</em> before executing given a model, but Q-learning does not really have a model and does not really think ahead.</p>
  </li>
  <li><p>You're wrong that to find a policy you need to know <span class="math-container">$P$</span> (the transition function). Q-learning finds policies but does not know or use <span class="math-container">$P$</span>.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-31T11:03:04.397Z
Answers.LastActivityDate: 2022-12-31T11:09:17.113Z
Answers.LastEditDate: 2022-12-31T11:09:17.113Z
Answers.UserDisplayName: nbro
Body: <p>I am confused regarding the difference between policy and plan in reinforcement learning. According to my understanding, when we calculate the value of state using Bellman equation in deterministic environment :</p>
  <p><a href="https://i.sstatic.net/Q2IOp.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Q2IOp.png" alt="enter image description here" /></a>
  The plan in this case will be the strict state action pair, that is gathered using finding the max. value for each action in every state and it will be something like the below image in a maze game as an example :
  <a href="https://i.sstatic.net/w0FWV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/w0FWV.png" alt="enter image description here" /></a></p>
  <p>However, in a stochastic environment the Bellman equation will be:</p>
  <p><a href="https://i.sstatic.net/xLUp0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xLUp0.png" alt="enter image description here" /></a></p>
  <p>And then we will have something like this:</p>
  <p><a href="https://i.sstatic.net/1oWP7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1oWP7.png" alt="enter image description here" /></a></p>
  <p>In this case, to develop the policy we will need to know the state-action pair for every state like the  above image plus mentioning the probability distribution for all the actions at every state and we need to keep in mind that the actions in the above image will not always happen due to the stochastic nature of the environment.</p>
  <p>Is my understanding correct regarding the difference between policy and plan?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-06T21:02:52.630Z
Comments.Text: Hi @AAA and welcome to AI Stack Exchange! Just out of curiosity, do you have an online reference that uses the word 'plan' in the context of reinforcement learning? I've seen it a few times before (e.g. in the Sutton and Barto reference that you mentioned), but I want to be sure that I'm using your intended definition before working on an answer.
Comments.UserDisplayName: DeepQZero
Tags: <reinforcement-learning><comparison><terminology><policies><planning>
Title: Is my understanding correct regarding the difference between policy and plan?
UserDisplayName: AAA
ThreadText: # Is my understanding correct regarding the difference between policy and plan?
  
  (Asked by: username_0 on 2022-12-06)
  
  I am confused regarding the difference between policy and plan in reinforcement learning. According to my understanding, when we calculate the value of state using Bellman equation in deterministic environment :
  
  [![enter image description here](https://i.sstatic.net/Q2IOp.png)](https://i.sstatic.net/Q2IOp.png)
  The plan in this case will be the strict state action pair, that is gathered using finding the max. value for each action in every state and it will be something like the below image in a maze game as an example :
  [![enter image description here](https://i.sstatic.net/w0FWV.png)](https://i.sstatic.net/w0FWV.png)
  
  However, in a stochastic environment the Bellman equation will be:
  
  [![enter image description here](https://i.sstatic.net/xLUp0.png)](https://i.sstatic.net/xLUp0.png)
  
  And then we will have something like this:
  
  [![enter image description here](https://i.sstatic.net/1oWP7.png)](https://i.sstatic.net/1oWP7.png)
  
  In this case, to develop the policy we will need to know the state-action pair for every state like the above image plus mentioning the probability distribution for all the actions at every state and we need to keep in mind that the actions in the above image will not always happen due to the stochastic nature of the environment.
  
  Is my understanding correct regarding the difference between policy and plan?
  
  **Tags:** reinforcement-learning, comparison, terminology, policies, planning
  
  > **Comments:**
  > **username_1**: Hi @username_0 and welcome to AI Stack Exchange! Just out of curiosity, do you have an online reference that uses the word 'plan' in the context of reinforcement learning? I've seen it a few times before (e.g. in the Sutton and Barto reference that you mentioned), but I want to be sure that I'm using your intended definition before working on an answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-12-31. Score: 0
  
  ### TL;DR
  
  I don't think your understanding of the notion of plan and policy is correct. These notions are independent of the nature of the environment but you seem to think that you can only find plans in deterministic environments and you need a policy for stochastic ones.
  
  ### Long answer
  
  Your post is very confusing because apparently you're very confused and mixing a lot of concepts. I'll provide clarifications for everything that I think is confusing.
  
  1. The first equation is just a special case of the second, where $P(a, s, s') = 1$ for some $s'$ and $P(a, s, s') = 0$ for all others. So, in reality, we can just use the second (more general) equation and forget about the first. However, it's true: we can use the first equation if you know the environment is deterministic.
  2. A plan is a **sequence of actions**. A policy $\pi$ is a [function that maps states to actions (or probability distributions over actions)](https://ai.stackexchange.com/q/12274/2444). So, given a sequence of states $s\_1, \dots, s\_N$, a deterministic policy will give you a sequence of actions $\pi(s\_1), \dots, \pi(s\_N) = a\_1, \dots, a\_N$, which can be thought of as a plan. If $\pi$ is stochastic, you need to sample actions, so the same policy can give you multiple different plans. For more info about the notion of a plan in RL, see [this](https://ai.stackexchange.com/a/10616/2444) answer.
  3. If you have a plan or not does not depend on whether your environment is stochastic or not. You can find plans in stochastic and deterministic environments. See [this](https://ai.stackexchange.com/a/26900/2444) answer for more info about the notion of planning and how it's related to search.
  4. [There are stochastic and deterministic policies](https://ai.stackexchange.com/q/12274/2444). There are stochastic and deterministic environments. These are difference concepts. For example, there are stochastic environments where the optimal policy is actually deterministic and there are stochastic environment where it's not.
  5. There's no difference between your first and second diagrams, with respect to the definition of a plan or policy. They both look like plans, or sequences of actions generated by a policy.
  6. To plan in an MDP, you usually need $P$ (the transition function), even if it's deterministic. You can use a dynamic programming algorithm like *policy iteration* to find a plan. However, you could also use a reinforcement learning algorithm like Q-learning to find a policy, which will give you a plan, as defined above.
  7. So, how come that Q-learning is usually not considered a planning algorithm? I think this is more a convention. We could consider all algorithms that find sequence of actions planning algorithms. However, if an algorithm needs to explore the environment by taking random actions, we don't consider it a planning algorithm. So, planning involves also some notion of *thinking ahead* before executing given a model, but Q-learning does not really have a model and does not really think ahead.
  8. You're wrong that to find a policy you need to know $P$ (the transition function). Q-learning finds policies but does not know or use $P$.
  
  
  ---
  
  