Answers.Body: <p>You're right! <em>Behaving</em> according to a deterministic policy while still learning would be a terrible idea in most cases (with the exception of environments that "do the exploring for you"; see comments). But deterministic policies are learned off-policy. That is, the experience used to learn the deterministic policy is gathered by behaving according to a stochastic behavior policy. </p>
  
  <p>Under some reasonable assumptions--like that the environment is fully-observed and is stationary--an optimal deterministic policy always exists. The proof can be found in chapter 6 of "<a href="https://onlinelibrary.wiley.com/doi/book/10.1002/9780470316887" rel="nofollow noreferrer">Markov Decision Process -- Discrete Stochastic Dynamic Programming</a>" by Martin L. Puterman. The same cannot be said for stochastic polices. For this kind of environment (even if it's stochastic) an optimal policy is hardly ever stochastic.</p>
  
  <p>So, a motivation for wanting to learn a deterministic policy is often because we know that there is an optimal deterministic policy. </p>
  
  <p>It's possible your question was also tangentially about off-policy learning. "Why learn a deterministic policy directly (off-policy) when we can just use something like decaying <span class="math-container">$\epsilon$</span>-greedy?" Briefly, off-policy learning is very powerful and general. It's necessary in any algorithm that uses experience replay, for example. A discussion about the merits of off-policy learning is probably best left to another question, but reading Section 5.5 of Sutton and Barto's <a href="http://incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">RL book</a> should get you started.</p>
  
  <p>Finally, directly learning a deterministic policy could be more computationally efficient if using <a href="http://proceedings.mlr.press/v32/silver14.pdf" rel="nofollow noreferrer">deterministic policy gradients</a>. In the setting with continuous state and action spaces, the deterministic policy gradient exists and has a simpler expectation than the stochastic policy gradient.</p>
  
  <p>Stochastic policy gradient:</p>
  
  <p><span class="math-container">$$\begin{align*}
  \nabla_\theta J(\pi_\theta) &amp;= \int_\mathcal{S} \rho^\pi (s) \int_\mathcal{A} \nabla_\theta \pi_\theta (a|s) Q^\pi (s,a) \text{d}a\text{d}s\\
  &amp; = \mathbb{E}_{s\sim \rho^\pi, a\sim\pi_\theta}[\nabla_\theta \log \pi_\theta (a|s) Q^\pi (s,a)]
  \end{align*}$$</span></p>
  
  <p>Deterministic policy gradient:
  <span class="math-container">$$\begin{align*}
  \nabla_\theta J(\mu_\theta) &amp;= \int_\mathcal{S} \rho^\mu (s) \nabla_\theta \mu_\theta (s) \nabla_a Q^\mu (s,a)|_{a=\mu_\theta (s)}\text{d}s\\
  &amp; = \mathbb{E}_{s\sim \rho^\mu}[\nabla_\theta \mu_\theta (s) \nabla_a Q^\mu (s,a)|_{a=\mu_\theta (s)}]
  \end{align*}$$</span></p>
  
  <p>Notice that the expectation in the deterministic policy gradient isn't over the action space. Estimating this expectation would require many fewer samples in the setting of a continuous, high-dimensional action space.</p>
  
  <p>To recap:</p>
  
  <ul>
  <li>Optimal policies are often deterministic, not stochastic</li>
  <li>Learning deterministic policies directly (off-policy) is powerful and general</li>
  <li>It can also be more efficient if in a continuous, high-dimensional action space</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-04T19:17:13.557Z
Answers.Text: Maybe add second/third bullet points at the end: 2) A stochastic environment does not mean that there is an optimal stochastic policy 3) It is common for there to be no optimal stochastic policy
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-04T19:18:13.647Z
Answers.Text: I would also like to comment that actually if the environment has enough randomness, it is possible to learn optimal control, on-policy with a deterministic greedy policy. This is how TD-Gammon worked for instance. They key is that the environment adds the necessary state exploration
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-04T19:35:22.263Z
Answers.Text: @NeilSlater Good points. Edits should reflect those changes.
Answers.UserDisplayName: Philip Raeisghasem
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-04T19:04:38.950Z
Answers.LastActivityDate: 2019-04-04T19:32:49.473Z
Answers.LastEditDate: 2019-04-04T19:32:49.473Z
Answers.UserDisplayName: Philip Raeisghasem
Body: <p>What is the motivation behind using a deterministic policy? Given that the environment is uncertain, it seems stochastic policy makes more sense. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-04T19:14:14.507Z
Comments.Text: A related question: [Is the optimal policy always stochastic if the environment is also stochastic?](https://ai.stackexchange.com/q/10591/2444).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><deterministic-policy>
Title: What is the motivation behind using a deterministic policy?
UserDisplayName: ycenycute
ThreadText: # What is the motivation behind using a deterministic policy?
  
  (Asked by: username_0 on 2019-04-04)
  
  What is the motivation behind using a deterministic policy? Given that the environment is uncertain, it seems stochastic policy makes more sense.
  
  **Tags:** reinforcement-learning, deterministic-policy
  > **Comments:**
  > **username_1**: A related question: [Is the optimal policy always stochastic if the environment is also stochastic?](https://ai.stackexchange.com/q/10591/2444).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-04-04. Score: 3 | Accepted answer
  
  You're right! *Behaving* according to a deterministic policy while still learning would be a terrible idea in most cases (with the exception of environments that "do the exploring for you"; see comments). But deterministic policies are learned off-policy. That is, the experience used to learn the deterministic policy is gathered by behaving according to a stochastic behavior policy.
  
  Under some reasonable assumptions--like that the environment is fully-observed and is stationary--an optimal deterministic policy always exists. The proof can be found in chapter 6 of "[Markov Decision Process -- Discrete Stochastic Dynamic Programming](https://onlinelibrary.wiley.com/doi/book/10.1002/9780470316887)" by Martin L. Puterman. The same cannot be said for stochastic polices. For this kind of environment (even if it's stochastic) an optimal policy is hardly ever stochastic.
  
  So, a motivation for wanting to learn a deterministic policy is often because we know that there is an optimal deterministic policy.
  
  It's possible your question was also tangentially about off-policy learning. "Why learn a deterministic policy directly (off-policy) when we can just use something like decaying $\epsilon$-greedy?" Briefly, off-policy learning is very powerful and general. It's necessary in any algorithm that uses experience replay, for example. A discussion about the merits of off-policy learning is probably best left to another question, but reading Section 5.5 of Sutton and Barto's [RL book](http://incompleteideas.net/book/the-book-2nd.html) should get you started.
  
  Finally, directly learning a deterministic policy could be more computationally efficient if using [deterministic policy gradients](http://proceedings.mlr.press/v32/silver14.pdf). In the setting with continuous state and action spaces, the deterministic policy gradient exists and has a simpler expectation than the stochastic policy gradient.
  
  Stochastic policy gradient:
  
  $$\begin{align\*}
  \nabla\_\theta J(\pi\_\theta) &= \int\_\mathcal{S} \rho^\pi (s) \int\_\mathcal{A} \nabla\_\theta \pi\_\theta (a|s) Q^\pi (s,a) \text{d}a\text{d}s\\
  & = \mathbb{E}\_{s\sim \rho^\pi, a\sim\pi\_\theta}[\nabla\_\theta \log \pi\_\theta (a|s) Q^\pi (s,a)]
  \end{align\*}$$
  
  Deterministic policy gradient:
  $$\begin{align\*}
  \nabla\_\theta J(\mu\_\theta) &= \int\_\mathcal{S} \rho^\mu (s) \nabla\_\theta \mu\_\theta (s) \nabla\_a Q^\mu (s,a)|\_{a=\mu\_\theta (s)}\text{d}s\\
  & = \mathbb{E}\_{s\sim \rho^\mu}[\nabla\_\theta \mu\_\theta (s) \nabla\_a Q^\mu (s,a)|\_{a=\mu\_\theta (s)}]
  \end{align\*}$$
  
  Notice that the expectation in the deterministic policy gradient isn't over the action space. Estimating this expectation would require many fewer samples in the setting of a continuous, high-dimensional action space.
  
  To recap:
  
  * Optimal policies are often deterministic, not stochastic
  * Learning deterministic policies directly (off-policy) is powerful and general
  * It can also be more efficient if in a continuous, high-dimensional action space
  
  > **Comments:**
  > **username_3**: Maybe add second/third bullet points at the end: 2) A stochastic environment does not mean that there is an optimal stochastic policy 3) It is common for there to be no optimal stochastic policy  
  > **username_3**: I would also like to comment that actually if the environment has enough randomness, it is possible to learn optimal control, on-policy with a deterministic greedy policy. This is how TD-Gammon worked for instance. They key is that the environment adds the necessary state exploration  
  > **username_2**: @NeilSlater Good points. Edits should reflect those changes.  
  
  ---
  
  