Answers.Body: <p>The premise of this question is somewhat misleading. There is a deterministic optimal policy for a MDP, but this does not mean a stochastic optimal policy never exists. Talking about <em>the</em> optimal policy can be misleading, as there may be many different optimal policies.</p>
  <p>For example, certainly we could imagine an MDP where <span class="math-container">$Q^*(s,a_0) = Q^*(s,a_1)$</span> for two different actions <span class="math-container">$a_1$</span> and <span class="math-container">$a_2$</span> that both maximize the optimal action-value function <span class="math-container">$Q^*$</span> at some state <span class="math-container">$s$</span>. Then a stochastic policy choosing randomly between <span class="math-container">$a_1$</span> and <span class="math-container">$a_2$</span> at <span class="math-container">$s$</span> is optimal, but so is a deterministic policy that always picks <span class="math-container">$a_1$</span> at <span class="math-container">$s$</span>, and a deterministic policy that always picks <span class="math-container">$a_2$</span> at <span class="math-container">$s$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-19T11:40:39.527Z
Answers.Text: [Here](https://ai.stackexchange.com/a/24694/2444) you have an answer that goes in a similar direction of what you say in your answer.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-18T22:55:26.073Z
Answers.LastActivityDate: 2021-03-18T22:55:26.073Z
Answers.UserDisplayName: mikkola
Answers.Body: <p>Suppose you learned your action-value function perfectly. Recall that the action-value function measures the expected return after taking a given action in a given state. Now, the goal when solving an MDP is to find a policy that maximizes expected returns. Suppose you're in state <span class="math-container">$s$</span>. According to your action-value function, let's say actions <span class="math-container">$a$</span> maximizes the expected return. So, according to the goal of solving an MDP, the only action you would ever take from state <span class="math-container">$s$</span> is <span class="math-container">$a$</span>. In other words <span class="math-container">$\pi(a'\mid s) = \mathbf{1}[a'=a]$</span>, which is a deterministic policy.</p>
  <p>Now, you might argue that your action-value function will never be perfect. However, this just means you need more exploration, which can manifest itself as stochasticity in the policy. However, in the limit of infinite data, the optimal policy will be deterministic since the action-value function will have converged to the optimum.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-06T14:31:00.597Z
Answers.LastActivityDate: 2020-08-06T14:31:00.597Z
Answers.UserDisplayName: harwiltz
Body: <p>Could someone please help me gain some <strong>intuition</strong> as to why the optimal policy for a Markov Decision Process in the infinite horizon case (agent acts forever) is deterministic?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-06T11:11:54.150Z
Comments.Text: Maybe you should provide the quote, or the link to the article (or a reference to a book) that states it, just to contextualise.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><markov-decision-process><policies><optimal-policy><optimality>
Title: Why is the optimal policy for an infinite horizon MDP deterministic?
UserDisplayName: stoic-santiago
ThreadText: # Why is the optimal policy for an infinite horizon MDP deterministic?
  
  (Asked by: username_0 on 2020-08-06)
  
  Could someone please help me gain some **intuition** as to why the optimal policy for a Markov Decision Process in the infinite horizon case (agent acts forever) is deterministic?
  
  **Tags:** reinforcement-learning, markov-decision-process, policies, optimal-policy, optimality
  > **Comments:**
  > **username_1**: Maybe you should provide the quote, or the link to the article (or a reference to a book) that states it, just to contextualise.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2020-08-06. Score: 3 | Accepted answer
  
  Suppose you learned your action-value function perfectly. Recall that the action-value function measures the expected return after taking a given action in a given state. Now, the goal when solving an MDP is to find a policy that maximizes expected returns. Suppose you're in state $s$. According to your action-value function, let's say actions $a$ maximizes the expected return. So, according to the goal of solving an MDP, the only action you would ever take from state $s$ is $a$. In other words $\pi(a'\mid s) = \mathbf{1}[a'=a]$, which is a deterministic policy.
  
  Now, you might argue that your action-value function will never be perfect. However, this just means you need more exploration, which can manifest itself as stochasticity in the policy. However, in the limit of infinite data, the optimal policy will be deterministic since the action-value function will have converged to the optimum.
  
  
  ---
  
  ### Answer by username_2 on 2021-03-18. Score: 1
  
  The premise of this question is somewhat misleading. There is a deterministic optimal policy for a MDP, but this does not mean a stochastic optimal policy never exists. Talking about *the* optimal policy can be misleading, as there may be many different optimal policies.
  
  For example, certainly we could imagine an MDP where $Q^\*(s,a\_0) = Q^\*(s,a\_1)$ for two different actions $a\_1$ and $a\_2$ that both maximize the optimal action-value function $Q^\*$ at some state $s$. Then a stochastic policy choosing randomly between $a\_1$ and $a\_2$ at $s$ is optimal, but so is a deterministic policy that always picks $a\_1$ at $s$, and a deterministic policy that always picks $a\_2$ at $s$.
  
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/a/24694/2444) you have an answer that goes in a similar direction of what you say in your answer.  
  
  ---
  
  