Answers.Body: <p>This works, and is used as a standard approach for two player zero-sum games in reinforcement learning. As you stated, it is a combination of reinforcement learning with Minimax optimisation.</p>
  <p>A very similar approach is to have reward +1 for player A winning, and reward -1 for player B winning, with A's objective to maximise reward and B's objective to minimise reward, allowing them to share the same value function directly (with your formulation, the safest thing to do is maintain two value functions, although in some cases it may work with one). This approach leaves the definition of <span class="math-container">$Q(s,a)$</span> unchanged from standard RL, but changes the Bellman equation (and matching standard update rule) of <span class="math-container">$Q(s,a) = r + \gamma \text{min}_{a'}Q(s',a')$</span>.</p>
  <p>You may see another variation, where the action variable <span class="math-container">$a$</span> is dropped, and the agent selects the next state from all possibilities that the game engine allows. This is called the afterstate representation, and it works effectively for simple board games because it reduces the space needed for the value function compared to an action value approach.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-23T05:58:10.763Z
Answers.Text: Instead of having one player try to maximize the Q function and another trying to minimize it, could you just reverse the board (replace every O with an X and every X with an O) after every step, then get one player to choose their action based on the normal board and the other to choose an action based on the reversed board?
Answers.UserDisplayName: Mas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-23T14:20:22.617Z
Answers.Text: @Mas that would depend on the game. Probably not a good choice for TicTacToe because the reversed board would not be valid for O
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-20T10:11:55.287Z
Answers.LastActivityDate: 2024-03-20T10:11:55.287Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have been looking for ways to train a Q-learning agent for a multiplayer zero-sum game (a variation of Tic-Tac-Toe in my case). I came up with a learning strategy I haven't found anywhere else, and I want to know if it would work. Here it is:</p>
  <p>The agent learns through self-play. It receives a <code>+1</code> reward for winning <em>(no matter what player won, you'll see later why)</em> and <code>0</code> for every other state. The agent learns through a modified Bellman equation, where the expected future reward is SUBTRACTED and not added.</p>
  <blockquote>
  <p>Q(s<sub>t</sub>, a) = r - γ * max Q(s<sub>t+1</sub>, a)</p>
  </blockquote>
  <p>If you expand this equation, you should get this:</p>
  <blockquote>
  <p>Q(s<sub>t</sub>, a) = r<sub>t</sub> - γ * r<sub>t+1</sub> + γ<sup>2</sup> * r<sub>t+2</sub> - γ<sup>3</sup> * r<sub>t+3</sub> + ...</p>
  </blockquote>
  <p>Notice how the plus and minus signs alternate. Thus, your expected reward is a sum of the rewards from your turns minus the sum of your opponent's rewards. That is also why the reward for winning is always <code>+1</code>.</p>
  <p>This strategy is similar to the minimax tree search, where we want the best rewards for us and the worst for our opponent.</p>
  <p>I hope I got my point across. Please note that I'm not very good at maths and AI theory, so forgive me if I mixed some things up or have a wrong notation. Also, if this is not a good approach, which would you recommend?</p>
  
Tags: <reinforcement-learning><q-learning><dqn><multi-agent-systems><zero-sum-games>
Title: Q learning (DQN) strategy for a multiplayer zero-sum game
UserDisplayName: Vladislav Korecký
ThreadText: # Q learning (DQN) strategy for a multiplayer zero-sum game
  
  (Asked by: username_0 on 2024-03-20)
  
  I have been looking for ways to train a Q-learning agent for a multiplayer zero-sum game (a variation of Tic-Tac-Toe in my case). I came up with a learning strategy I haven't found anywhere else, and I want to know if it would work. Here it is:
  
  The agent learns through self-play. It receives a `+1` reward for winning *(no matter what player won, you'll see later why)* and `0` for every other state. The agent learns through a modified Bellman equation, where the expected future reward is SUBTRACTED and not added.
  
  > Q(st, a) = r - γ \* max Q(st+1, a)
  
  If you expand this equation, you should get this:
  
  > Q(st, a) = rt - γ \* rt+1 + γ2 \* rt+2 - γ3 \* rt+3 + ...
  
  Notice how the plus and minus signs alternate. Thus, your expected reward is a sum of the rewards from your turns minus the sum of your opponent's rewards. That is also why the reward for winning is always `+1`.
  
  This strategy is similar to the minimax tree search, where we want the best rewards for us and the worst for our opponent.
  
  I hope I got my point across. Please note that I'm not very good at maths and AI theory, so forgive me if I mixed some things up or have a wrong notation. Also, if this is not a good approach, which would you recommend?
  
  **Tags:** reinforcement-learning, q-learning, dqn, multi-agent-systems, zero-sum-games
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-03-20. Score: 2 | Accepted answer
  
  This works, and is used as a standard approach for two player zero-sum games in reinforcement learning. As you stated, it is a combination of reinforcement learning with Minimax optimisation.
  
  A very similar approach is to have reward +1 for player A winning, and reward -1 for player B winning, with A's objective to maximise reward and B's objective to minimise reward, allowing them to share the same value function directly (with your formulation, the safest thing to do is maintain two value functions, although in some cases it may work with one). This approach leaves the definition of $Q(s,a)$ unchanged from standard RL, but changes the Bellman equation (and matching standard update rule) of $Q(s,a) = r + \gamma \text{min}\_{a'}Q(s',a')$.
  
  You may see another variation, where the action variable $a$ is dropped, and the agent selects the next state from all possibilities that the game engine allows. This is called the afterstate representation, and it works effectively for simple board games because it reduces the space needed for the value function compared to an action value approach.
  
  > **Comments:**
  > **username_2**: Instead of having one player try to maximize the Q function and another trying to minimize it, could you just reverse the board (replace every O with an X and every X with an O) after every step, then get one player to choose their action based on the normal board and the other to choose an action based on the reversed board?  
  > **username_1**: @Mas that would depend on the game. Probably not a good choice for TicTacToe because the reversed board would not be valid for O  
  
  ---
  
  