Answers.Body: <h3>Deterministic Policy  :</h3>
  <p>Its means that for every state you have clear defined action you will take</p>
  <p><strong>For Example:</strong> We 100% know we will take action <strong>A</strong> from state <strong>X</strong>.</p>
  <h3>Stochastic Policy :</h3>
  <p>Its mean that for every state you do not have clear defined action to take but you have probability distribution for actions to take from that state.</p>
  <p><strong>For example</strong> there are 10% chance of taking action <strong>A</strong> from state <strong>S</strong>, There are 20% chance of taking <strong>B</strong> from State <strong>S</strong> and there are 70% chance of taking action <strong>C</strong> from state <strong>S</strong>, Its mean we don't have clear defined action to take but we have some probability of taking actions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T05:46:56.737Z
Answers.LastActivityDate: 2020-05-06T06:00:59.213Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Laeeq Khan Niazi
Answers.Body: <p>Apart from the answers above,</p>
  
  <hr>
  
  <p><em>Stochastic Policy function</em>: <span class="math-container">$\pi (s_1s_2 \dots s_n, a_1 a_2 \dots a_n): \mathcal S \times \mathcal A \rightarrow [0,1]$</span> is the probability distribution function, that, tells the probability that action sequence <span class="math-container">$a_1a_2 \dots a_n$</span> may be chosen in state sequence <span class="math-container">$s_1 s_2 \dots s_n$</span><a href="https://www.freecodecamp.org/news/an-introduction-to-policy-gradients-with-cartpole-and-doom-495b5ef2207f/" rel="nofollow noreferrer">[2]</a><a href="https://www.computing.dcu.ie/~humphrys/PhD/ch2.html" rel="nofollow noreferrer">[3]</a>. </p>
  
  <p>In <em>Markov Decision Process (MDP)</em>, it's only <span class="math-container">$\pi (s, a)$</span> following the assumptions<a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-825-techniques-in-artificial-intelligence-sma-5504-fall-2002/lecture-notes/Lecture20FinalPart1.pdf" rel="nofollow noreferrer">[1]</a>:
  <span class="math-container">$$ \mathbb P(\omega_{t+1}| \omega_t, a_t) = \mathbb P(\omega_{t+1}| \omega_t,a_t, \dots \omega_o,a_o)$$</span>
  Where <span class="math-container">$\omega \in \Omega$</span> which is the set of Observations.  <span class="math-container">$\mathcal A, \mathcal S$</span> denote the set of actions and states respectively. Since, the next observation is dependent only on present states and not the past, the policy function only needs the present state and action as parameter.</p>
  
  <p>The next action is chosen as[2]: <span class="math-container">$$ a^* = \arg \max_a \pi(s_{t+1}, a) \quad\forall a \in \mathcal A $$</span></p>
  
  <p><em>Deterministic Policy function</em> <a href="https://www.computing.dcu.ie/~humphrys/PhD/ch2.html" rel="nofollow noreferrer">[3]</a>: is a special case of Stochastic Policy function where for particular <span class="math-container">$a_o \in \mathcal A$</span>, <span class="math-container">$\pi(s, a_n) = \delta^o_n$</span> for all <span class="math-container">$a_n \in \mathcal A$</span>. Here, we are totally certain to choose particular action <span class="math-container">$a_o$</span> in some arbitrary state <span class="math-container">$s$</span> and no other. Here <span class="math-container">$\delta$</span> is <a href="https://en.wikipedia.org/wiki/Kronecker_delta" rel="nofollow noreferrer">Kronecker delta</a>. Since, the probability distribution here is discrete, it's often written in the form of <span class="math-container">$\pi(s): \mathcal S \rightarrow \mathcal A$</span>, where the function takes arbitrary state <span class="math-container">$s$</span> and maps it to an action <span class="math-container">$a$</span> which is 100% probable.</p>
  
  <h2>IMPORTANT</h2>
  
  <p>The <strong>Stochastic Policy function</strong> is not meant to be confused with the <strong>Transition Function</strong><a href="https://www.freecodecamp.org/news/an-introduction-to-policy-gradients-with-cartpole-and-doom-495b5ef2207f/" rel="nofollow noreferrer">[2]</a> (which is also a Probability Distribution Function), <span class="math-container">$T(s_t, a_t, s_{t+1}): \mathcal S \times \mathcal A \times \mathcal S \rightarrow [0, 1]$</span> which tells the probability that - at state <span class="math-container">$s_t$</span>, the action <span class="math-container">$a_t$</span> will lead us to next state <span class="math-container">$s_{t+1}$</span>.</p>
  
  <hr>
  
  <h2>References:</h2>
  
  <ol>
  <li><p><a href="https://ocw.mit.edu" rel="nofollow noreferrer">https://ocw.mit.edu</a>. <a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-825-techniques-in-artificial-intelligence-sma-5504-fall-2002/lecture-notes/Lecture20FinalPart1.pdf" rel="nofollow noreferrer"><em>6.825 Techniques in Artificial Intelligence</em></a>. <em><a href="https://ocw.mit.edu" rel="nofollow noreferrer">https://ocw.mit.edu</a></em>. Page Number - 6. Web. 6 May 2020</p></li>
  <li><p>Simonini, Thomas. <a href="https://www.freecodecamp.org" rel="nofollow noreferrer">https://www.freecodecamp.org</a> .<a href="https://www.freecodecamp.org/news/an-introduction-to-policy-gradients-with-cartpole-and-doom-495b5ef2207f/" rel="nofollow noreferrer"><em>An introduction to Policy Gradients with Cartpole and Doom</em></a>. 9 May 2018. Web. 6 May 2020.</p></li>
  <li><p><a href="https://www.computing.dcu.ie/" rel="nofollow noreferrer">https://www.computing.dcu.ie/</a>. Reinforcement Learning. <a href="https://www.computing.dcu.ie/~humphrys/PhD/ch2.html" rel="nofollow noreferrer"><em>2.1.1 Special case - Deterministic worlds</em></a>. Web. 6 May 2020 </p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T08:11:02.477Z
Answers.LastActivityDate: 2020-05-06T08:34:57.927Z
Answers.LastEditDate: 2020-05-06T08:34:57.927Z
Answers.UserDisplayName: Abhas Kumar Sinha
Answers.Body: <p>A <strong>deterministic policy</strong> is a function of the form <span class="math-container">$\pi_{\mathbb{d}}: S \rightarrow A$</span>, that is, a function from the set of states of the environment, <span class="math-container">$S$</span>, to the set of actions, <span class="math-container">$A$</span>. The subscript <span class="math-container">$_{\mathbb{d}}$</span> only indicates that this is a <span class="math-container">${\mathbb{d}}$</span>eterministic policy.</p>
  <p>For example, in a grid world, the set of states of the environment, <span class="math-container">$S$</span>, is composed of each cell of the grid, and the set of actions, <span class="math-container">$A$</span>, is composed of the actions &quot;left&quot;, &quot;right&quot;, &quot;up&quot; and &quot;down&quot;. Given a state <span class="math-container">$s \in S$</span>, <span class="math-container">$\pi(s)$</span> is, with probability <span class="math-container">$1$</span>, always the same action (e.g. &quot;up&quot;), unless the policy changes.</p>
  <p>A <strong>stochastic policy</strong> can be represented as a <em>family</em> of <a href="https://brilliant.org/wiki/conditional-probability-distribution/" rel="noreferrer">conditional probability distributions</a>, <span class="math-container">$\pi_{\mathbb{s}}(A \mid S)$</span>, from the set of states, <span class="math-container">$S$</span>, to the set of actions, <span class="math-container">$A$</span>. A probability distribution is a function that assigns a probability for each <a href="https://en.wikipedia.org/wiki/Event_(probability_theory)" rel="noreferrer">event</a> (in this case, the events are actions in certain states) and such that the sum of all the probabilities is <span class="math-container">$1$</span>.</p>
  <p>A stochastic policy is a family and not just one conditional probability distribution because, for a fixed state <span class="math-container">$s \in S$</span>, <span class="math-container">$\pi_{\mathbb{s}}(A \mid S = s)$</span> is a possibly distinct conditional probability distribution. In other words, <span class="math-container">$\pi_{\mathbb{s}}(A \mid S) = \{ \pi_{\mathbb{s}}(A \mid S = s_1), \dots, \pi_{\mathbb{s}}(A \mid S = s_{|S|})\}$</span>, where <span class="math-container">$\pi_{\mathbb{s}}(A \mid S = s)$</span> is a conditional probability distribution over actions given that the state is <span class="math-container">$s \in S$</span> and <span class="math-container">$|S|$</span> is the size of the set of states of the environment.</p>
  <p>Often, in the reinforcement learning context, a stochastic policy is misleadingly (at least in my opinion) denoted by <span class="math-container">$\pi_{\mathbb{s}}(a \mid s)$</span>, where <span class="math-container">$a \in A$</span> and <span class="math-container">$s \in S$</span> are respectively a specific action and state, so <span class="math-container">$\pi_{\mathbb{s}}(a \mid s)$</span> is just a number and not a conditional probability distribution. A single conditional probability distribution can be denoted by <span class="math-container">$\pi_{\mathbb{s}}(A \mid S = s)$</span>, for some fixed state <span class="math-container">$s \in S$</span>. However, <span class="math-container">$\pi_{\mathbb{s}}(a \mid s)$</span> can also denote a family of conditional probability distributions, that is, <span class="math-container">$\pi_{\mathbb{s}}(A \mid S) = \pi_{\mathbb{s}}(a \mid s)$</span>, if <span class="math-container">$a$</span> and <span class="math-container">$s$</span> are arbitrary. Alternatively, <span class="math-container">$a$</span> and <span class="math-container">$s$</span> in <span class="math-container">$\pi_{\mathbb{s}}(a \mid s)$</span> are just (dummy or input) variables of the function <span class="math-container">$\pi_{\mathbb{s}}(a \mid s)$</span> (i.e. p.m.f. or p.d.f.): this is probably the most sensible way of interpreting <span class="math-container">$\pi_{\mathbb{s}}(a \mid s)$</span> when you see this notation (see also <a href="https://ai.stackexchange.com/a/16842/2444">this answer</a>). In this case, you could also think of a stochastic policy as a function <span class="math-container">$\pi_{\mathbb{s}} : S \times A \rightarrow [0, 1]$</span>, but, in my view, although this may be the way you implement a stochastic policy in practice, this notation is misleading, as the action is not conceptually an input to the stochastic policy but rather an output (but in the end this is also just an interpretation).</p>
  <p>In the particular case of games of chance (e.g. poker), where there are sources of randomness, a deterministic policy might not always be appropriate. For example, in poker, not all information (e.g. the cards of the other players) is available. In those circumstances, the agent might decide to play differently depending on the round (time step). More concretely, the agent could decide to go &quot;all-in&quot; <span class="math-container">$\frac{2}{3}$</span> of the times whenever it has a hand with two aces and there are two uncovered aces on the table and decide to just &quot;raise&quot; <span class="math-container">$\frac{1}{3}$</span> of the other times.</p>
  <p>A deterministic policy can be interpreted as a stochastic policy that gives the probability of <span class="math-container">$1$</span> to one of the available actions (and <span class="math-container">$0$</span> to the remaining actions), for each state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-07T00:54:01.747Z
Answers.Text: Note: I need to update this answer. The set of states (in the grid world) is better described as the set of all possible combinations of the grid.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-12T18:50:32.373Z
Answers.LastActivityDate: 2021-06-29T12:41:58.950Z
Answers.LastEditDate: 2021-06-29T12:41:58.950Z
Answers.UserDisplayName: nbro
Body: <p>In reinforcement learning, there are the concepts of stochastic (or probabilistic) and deterministic policies. What is the difference between them?</p>
  
Tags: <reinforcement-learning><comparison><policies><deterministic-policy><stochastic-policy>
Title: What is the difference between a stochastic and a deterministic policy?
UserDisplayName: nbro
ThreadText: # What is the difference between a stochastic and a deterministic policy?
  
  (Asked by: username_0 on 2019-05-12)
  
  In reinforcement learning, there are the concepts of stochastic (or probabilistic) and deterministic policies. What is the difference between them?
  
  **Tags:** reinforcement-learning, comparison, policies, deterministic-policy, stochastic-policy
  
  ---
  
  ## 3 answers
  
  ### Answer by username_0 on 2019-05-12. Score: 15 | Accepted answer
  
  A **deterministic policy** is a function of the form $\pi\_{\mathbb{d}}: S \rightarrow A$, that is, a function from the set of states of the environment, $S$, to the set of actions, $A$. The subscript $\_{\mathbb{d}}$ only indicates that this is a ${\mathbb{d}}$eterministic policy.
  
  For example, in a grid world, the set of states of the environment, $S$, is composed of each cell of the grid, and the set of actions, $A$, is composed of the actions "left", "right", "up" and "down". Given a state $s \in S$, $\pi(s)$ is, with probability $1$, always the same action (e.g. "up"), unless the policy changes.
  
  A **stochastic policy** can be represented as a *family* of [conditional probability distributions](https://brilliant.org/wiki/conditional-probability-distribution/), $\pi\_{\mathbb{s}}(A \mid S)$, from the set of states, $S$, to the set of actions, $A$. A probability distribution is a function that assigns a probability for each [event](https://en.wikipedia.org/wiki/Event_(probability_theory)) (in this case, the events are actions in certain states) and such that the sum of all the probabilities is $1$.
  
  A stochastic policy is a family and not just one conditional probability distribution because, for a fixed state $s \in S$, $\pi\_{\mathbb{s}}(A \mid S = s)$ is a possibly distinct conditional probability distribution. In other words, $\pi\_{\mathbb{s}}(A \mid S) = \{ \pi\_{\mathbb{s}}(A \mid S = s\_1), \dots, \pi\_{\mathbb{s}}(A \mid S = s\_{|S|})\}$, where $\pi\_{\mathbb{s}}(A \mid S = s)$ is a conditional probability distribution over actions given that the state is $s \in S$ and $|S|$ is the size of the set of states of the environment.
  
  Often, in the reinforcement learning context, a stochastic policy is misleadingly (at least in my opinion) denoted by $\pi\_{\mathbb{s}}(a \mid s)$, where $a \in A$ and $s \in S$ are respectively a specific action and state, so $\pi\_{\mathbb{s}}(a \mid s)$ is just a number and not a conditional probability distribution. A single conditional probability distribution can be denoted by $\pi\_{\mathbb{s}}(A \mid S = s)$, for some fixed state $s \in S$. However, $\pi\_{\mathbb{s}}(a \mid s)$ can also denote a family of conditional probability distributions, that is, $\pi\_{\mathbb{s}}(A \mid S) = \pi\_{\mathbb{s}}(a \mid s)$, if $a$ and $s$ are arbitrary. Alternatively, $a$ and $s$ in $\pi\_{\mathbb{s}}(a \mid s)$ are just (dummy or input) variables of the function $\pi\_{\mathbb{s}}(a \mid s)$ (i.e. p.m.f. or p.d.f.): this is probably the most sensible way of interpreting $\pi\_{\mathbb{s}}(a \mid s)$ when you see this notation (see also [this answer](https://ai.stackexchange.com/a/16842/2444)). In this case, you could also think of a stochastic policy as a function $\pi\_{\mathbb{s}} : S \times A \rightarrow [0, 1]$, but, in my view, although this may be the way you implement a stochastic policy in practice, this notation is misleading, as the action is not conceptually an input to the stochastic policy but rather an output (but in the end this is also just an interpretation).
  
  In the particular case of games of chance (e.g. poker), where there are sources of randomness, a deterministic policy might not always be appropriate. For example, in poker, not all information (e.g. the cards of the other players) is available. In those circumstances, the agent might decide to play differently depending on the round (time step). More concretely, the agent could decide to go "all-in" $\frac{2}{3}$ of the times whenever it has a hand with two aces and there are two uncovered aces on the table and decide to just "raise" $\frac{1}{3}$ of the other times.
  
  A deterministic policy can be interpreted as a stochastic policy that gives the probability of $1$ to one of the available actions (and $0$ to the remaining actions), for each state.
  
  > **Comments:**
  > **username_0**: Note: I need to update this answer. The set of states (in the grid world) is better described as the set of all possible combinations of the grid.  
  
  ---
  
  ### Answer by username_1 on 2020-05-06. Score: 4
  
  ### Deterministic Policy :
  
  Its means that for every state you have clear defined action you will take
  
  **For Example:** We 100% know we will take action **A** from state **X**.
  
  ### Stochastic Policy :
  
  Its mean that for every state you do not have clear defined action to take but you have probability distribution for actions to take from that state.
  
  **For example** there are 10% chance of taking action **A** from state **S**, There are 20% chance of taking **B** from State **S** and there are 70% chance of taking action **C** from state **S**, Its mean we don't have clear defined action to take but we have some probability of taking actions.
  
  
  ---
  
  ### Answer by username_2 on 2020-05-06. Score: 1
  
  Apart from the answers above,
  
  ---
  
  *Stochastic Policy function*: $\pi (s\_1s\_2 \dots s\_n, a\_1 a\_2 \dots a\_n): \mathcal S \times \mathcal A \rightarrow [0,1]$ is the probability distribution function, that, tells the probability that action sequence $a\_1a\_2 \dots a\_n$ may be chosen in state sequence $s\_1 s\_2 \dots s\_n$[[2]](https://www.freecodecamp.org/news/an-introduction-to-policy-gradients-with-cartpole-and-doom-495b5ef2207f/)[[3]](https://www.computing.dcu.ie/~humphrys/PhD/ch2.html).
  
  In *Markov Decision Process (MDP)*, it's only $\pi (s, a)$ following the assumptions[[1]](https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-825-techniques-in-artificial-intelligence-sma-5504-fall-2002/lecture-notes/Lecture20FinalPart1.pdf):
  $$ \mathbb P(\omega\_{t+1}| \omega\_t, a\_t) = \mathbb P(\omega\_{t+1}| \omega\_t,a\_t, \dots \omega\_o,a\_o)$$
  Where $\omega \in \Omega$ which is the set of Observations. $\mathcal A, \mathcal S$ denote the set of actions and states respectively. Since, the next observation is dependent only on present states and not the past, the policy function only needs the present state and action as parameter.
  
  The next action is chosen as[2]: $$ a^\* = \arg \max\_a \pi(s\_{t+1}, a) \quad\forall a \in \mathcal A $$
  
  *Deterministic Policy function* [[3]](https://www.computing.dcu.ie/~humphrys/PhD/ch2.html): is a special case of Stochastic Policy function where for particular $a\_o \in \mathcal A$, $\pi(s, a\_n) = \delta^o\_n$ for all $a\_n \in \mathcal A$. Here, we are totally certain to choose particular action $a\_o$ in some arbitrary state $s$ and no other. Here $\delta$ is [Kronecker delta](https://en.wikipedia.org/wiki/Kronecker_delta). Since, the probability distribution here is discrete, it's often written in the form of $\pi(s): \mathcal S \rightarrow \mathcal A$, where the function takes arbitrary state $s$ and maps it to an action $a$ which is 100% probable.
  
  ## IMPORTANT
  
  The **Stochastic Policy function** is not meant to be confused with the **Transition Function**[[2]](https://www.freecodecamp.org/news/an-introduction-to-policy-gradients-with-cartpole-and-doom-495b5ef2207f/) (which is also a Probability Distribution Function), $T(s\_t, a\_t, s\_{t+1}): \mathcal S \times \mathcal A \times \mathcal S \rightarrow [0, 1]$ which tells the probability that - at state $s\_t$, the action $a\_t$ will lead us to next state $s\_{t+1}$.
  
  ---
  
  ## References:
  
  1. <https://ocw.mit.edu>. [*6.825 Techniques in Artificial Intelligence*](https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-825-techniques-in-artificial-intelligence-sma-5504-fall-2002/lecture-notes/Lecture20FinalPart1.pdf). *<https://ocw.mit.edu>*. Page Number - 6. Web. 6 May 2020
  2. Simonini, Thomas. <https://www.freecodecamp.org> .[*An introduction to Policy Gradients with Cartpole and Doom*](https://www.freecodecamp.org/news/an-introduction-to-policy-gradients-with-cartpole-and-doom-495b5ef2207f/). 9 May 2018. Web. 6 May 2020.
  3. <https://www.computing.dcu.ie/>. Reinforcement Learning. [*2.1.1 Special case - Deterministic worlds*](https://www.computing.dcu.ie/~humphrys/PhD/ch2.html). Web. 6 May 2020
  
  
  ---
  
  