Answers.Body: <blockquote>
  <p>Is it always a good idea to use deterministic policies during testing?</p>
  </blockquote>
  <p>No, it's not, and you correctly point out that in certain scenarios the optimal policy is stochastic.</p>
  <p>Specifically, the optimal policy in Rock-Paper-Scissors is a stochastic policy. However, this game can't really be represented as a stationary MDP, but as a <a href="https://courses.cs.duke.edu/spring07/cps296.3/littman94markov.pdf" rel="nofollow noreferrer">Markov Game</a> with 2 players. If you wanted to represent it as an MDP (Markov Game with 1 player), then it might not be stationary, i.e. the dynamics could change.</p>
  <p>Having said that, there's a known result in the theory of MDPs that says that there's at least one optimal policy that is stationary and deterministic, but there are other environments and models other than finite MDPs. You also have <a href="https://nbro.gitlab.io/blogging/2022/01/01/mdps-are-pomdps/" rel="nofollow noreferrer">POMDPs</a>, Markov Games, etc., which are generalisations of finite MDPs to different scenarios (partial observability of the state and multiple players, respectively).</p>
  <p>So, when should you use deterministic policies? Well, most likely, when you are sure that your environment is a finite MDP. It's possible that, when they use deterministic policies, people assume the environment can be represented as an MDP, while, in reality, maybe it's not. Then, of course, in practice, you lose all theoretical guarantees.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-16T10:00:16.070Z
Answers.LastActivityDate: 2023-02-16T10:07:05.610Z
Answers.LastEditDate: 2023-02-16T10:07:05.610Z
Answers.UserDisplayName: nbro
Body: <p>I frequently see people setting deterministic = True while testing an RL algorithm. But is this the right approach? For instance, what happens if the agent plays rock, paper, and scissors? In this case, as per game theory, a stochastic (random) policy is required (as per my understanding)</p>
  <p>Edit - Let me clarify my understanding of stochastic and deterministic policies. A deterministic policy will always select an action given some state. The stochastic policy will sample the action given a state. Therefore in the latter case, an optimal policy for Rock, Paper and Scissor will always choose an action with probability of 0.33.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-18T05:22:48.303Z
Comments.Text: It's worth noting deterministic policy doesn't necessarily mean it always select a unique action given some state, it's possible in finite MDP at some state several actions equally solves the Bellman optimality equation, thus the greedy deterministic optimal policy can randomly follow any such action.
Comments.UserDisplayName: cinch
Tags: <reinforcement-learning><deep-rl><policy-gradients>
Title: Is it always a good idea to use deterministic policies during testing?
UserDisplayName: desert_ranger
ThreadText: # Is it always a good idea to use deterministic policies during testing?
  
  (Asked by: username_0 on 2023-02-15)
  
  I frequently see people setting deterministic = True while testing an RL algorithm. But is this the right approach? For instance, what happens if the agent plays rock, paper, and scissors? In this case, as per game theory, a stochastic (random) policy is required (as per my understanding)
  
  Edit - Let me clarify my understanding of stochastic and deterministic policies. A deterministic policy will always select an action given some state. The stochastic policy will sample the action given a state. Therefore in the latter case, an optimal policy for Rock, Paper and Scissor will always choose an action with probability of 0.33.
  
  **Tags:** reinforcement-learning, deep-rl, policy-gradients
  
  > **Comments:**
  > **username_1**: It's worth noting deterministic policy doesn't necessarily mean it always select a unique action given some state, it's possible in finite MDP at some state several actions equally solves the Bellman optimality equation, thus the greedy deterministic optimal policy can randomly follow any such action.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-02-16. Score: 4 | Accepted answer
  
  > Is it always a good idea to use deterministic policies during testing?
  
  No, it's not, and you correctly point out that in certain scenarios the optimal policy is stochastic.
  
  Specifically, the optimal policy in Rock-Paper-Scissors is a stochastic policy. However, this game can't really be represented as a stationary MDP, but as a [Markov Game](https://courses.cs.duke.edu/spring07/cps296.3/littman94markov.pdf) with 2 players. If you wanted to represent it as an MDP (Markov Game with 1 player), then it might not be stationary, i.e. the dynamics could change.
  
  Having said that, there's a known result in the theory of MDPs that says that there's at least one optimal policy that is stationary and deterministic, but there are other environments and models other than finite MDPs. You also have [POMDPs](https://username_2.gitlab.io/blogging/2022/01/01/mdps-are-pomdps/), Markov Games, etc., which are generalisations of finite MDPs to different scenarios (partial observability of the state and multiple players, respectively).
  
  So, when should you use deterministic policies? Well, most likely, when you are sure that your environment is a finite MDP. It's possible that, when they use deterministic policies, people assume the environment can be represented as an MDP, while, in reality, maybe it's not. Then, of course, in practice, you lose all theoretical guarantees.
  
  
  ---
  
  