<p>I'm training a reinforcement learning agent on simple two-player games (I'm doing Q-learning). I noticed that for a given state the value of each action is usually different. Thus, the greedy policy leads to a deterministic agent. Sometimes it may be useful to add randomness to the agent's policy.</p>
<p>First, this makes for more interesting games when playing against a human. Second, if I want to test how strong the agent is or compare two agents, performing multiple tests appears to be helpful (but only if each test could potentially lead to a different outcome). Does this make sense?</p>
<p>Is there some standard approach to add some randomness to the agent's behavior? I tried replacing greedy with softmax but this appears to decrease the agent's performance a lot.</p>

CC BY-SA 4.0
2023-02-19T01:58:21.903Z
Q-learning is off-policy, which means that it tries to learn a target policy, which happens to be greedy, that is different than the behaviour/exploration policy (usually, $\epsilon$-greedy but maybe also softmax). When you talk about the deterministic policy, are you referring to the target policy? I suppose so. Which game is this? The optimal policy may indeed be stochastic.
nbro
CC BY-SA 4.0
2023-02-19T02:30:46.123Z
I'm referring to the target policy. I'm trying not to tie this to a specific game but I'm assuming it's a 2-player zero-sum perfect information game. Currently I'm testing with tic tac toe and reversi but it could be something like chess.
martinkunev
<reinforcement-learning>
Non-deterministic play in reinforcement learning
martinkunev
# Non-deterministic play in reinforcement learning

(Asked by: username_0 on 2023-02-18)

I'm training a reinforcement learning agent on simple two-player games (I'm doing Q-learning). I noticed that for a given state the value of each action is usually different. Thus, the greedy policy leads to a deterministic agent. Sometimes it may be useful to add randomness to the agent's policy.

First, this makes for more interesting games when playing against a human. Second, if I want to test how strong the agent is or compare two agents, performing multiple tests appears to be helpful (but only if each test could potentially lead to a different outcome). Does this make sense?

Is there some standard approach to add some randomness to the agent's behavior? I tried replacing greedy with softmax but this appears to decrease the agent's performance a lot.

**Tags:** reinforcement-learning

> **Comments:**
> **username_1**: Q-learning is off-policy, which means that it tries to learn a target policy, which happens to be greedy, that is different than the behaviour/exploration policy (usually, $\epsilon$-greedy but maybe also softmax). When you talk about the deterministic policy, are you referring to the target policy? I suppose so. Which game is this? The optimal policy may indeed be stochastic.  
> **username_0**: I'm referring to the target policy. I'm trying not to tie this to a specific game but I'm assuming it's a 2-player zero-sum perfect information game. Currently I'm testing with tic tac toe and reversi but it could be something like chess.  

---

## 0 answers

