Body: <p>I'm training a reinforcement learning agent on simple two-player games (I'm doing Q-learning). I noticed that for a given state the value of each action is usually different. Thus, the greedy policy leads to a deterministic agent. Sometimes it may be useful to add randomness to the agent's policy.</p>
  <p>First, this makes for more interesting games when playing against a human. Second, if I want to test how strong the agent is or compare two agents, performing multiple tests appears to be helpful (but only if each test could potentially lead to a different outcome). Does this make sense?</p>
  <p>Is there some standard approach to add some randomness to the agent's behavior? I tried replacing greedy with softmax but this appears to decrease the agent's performance a lot.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-19T01:58:21.903Z
Comments.Text: Q-learning is off-policy, which means that it tries to learn a target policy, which happens to be greedy, that is different than the behaviour/exploration policy (usually, $\epsilon$-greedy but maybe also softmax). When you talk about the deterministic policy, are you referring to the target policy? I suppose so. Which game is this? The optimal policy may indeed be stochastic.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-19T02:30:46.123Z
Comments.Text: I'm referring to the target policy. I'm trying not to tie this to a specific game but I'm assuming it's a 2-player zero-sum perfect information game. Currently I'm testing with tic tac toe and reversi but it could be something like chess.
Comments.UserDisplayName: martinkunev
Tags: <reinforcement-learning>
Title: Non-deterministic play in reinforcement learning
UserDisplayName: martinkunev
ThreadText: # Non-deterministic play in reinforcement learning
  
  (Asked by: username_0 on 2023-02-18)
  
  I'm training a reinforcement learning agent on simple two-player games (I'm doing Q-learning). I noticed that for a given state the value of each action is usually different. Thus, the greedy policy leads to a deterministic agent. Sometimes it may be useful to add randomness to the agent's policy.
  
  First, this makes for more interesting games when playing against a human. Second, if I want to test how strong the agent is or compare two agents, performing multiple tests appears to be helpful (but only if each test could potentially lead to a different outcome). Does this make sense?
  
  Is there some standard approach to add some randomness to the agent's behavior? I tried replacing greedy with softmax but this appears to decrease the agent's performance a lot.
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: Q-learning is off-policy, which means that it tries to learn a target policy, which happens to be greedy, that is different than the behaviour/exploration policy (usually, $\epsilon$-greedy but maybe also softmax). When you talk about the deterministic policy, are you referring to the target policy? I suppose so. Which game is this? The optimal policy may indeed be stochastic.  
  > **username_0**: I'm referring to the target policy. I'm trying not to tie this to a specific game but I'm assuming it's a 2-player zero-sum perfect information game. Currently I'm testing with tic tac toe and reversi but it could be something like chess.  
  
  ---
  
  ## 0 answers
  
  