Answers.Body: <p>There is a single neural network that guides self-plays in the Monte Carlo Tree Search algorithm. The neural network gets the current state of the board <span class="math-container">$s$</span> as an input and outputs current policy <span class="math-container">$\pi(a|s)$</span> and value <span class="math-container">$v(s)$</span>.</p>
  <p>The action probabilities are encoded in a (8,8,73) tensor. First two dimensions encode the coordinates of the figure to &quot;pick&quot; from the board. The third dimension encode where to move this figure: check out <a href="https://ai.stackexchange.com/questions/27336/how-does-the-alpha-zeros-move-encoding-work">this question</a> for a discussion on how all possible moves are encoded in a 73 dimensional vector.</p>
  <p>Similarly, the inputs of the network are organized in the (8, 8, 14 * 8 + 7 = 119) tensor. The first two 8 x 8 dimensions, again, encode the positions on the board. Then the positions of the figures one plane per 6 figure types: first 6 planes for player's figures, next 6 planes for opponent's figures and two <a href="https://ai.stackexchange.com/a/26656/20538">repetition planes</a>. The 14 planes are repeated 8 times supplying predecessor positions to the network. Finally, there are 7 extra planes encoding as a single uniform value over the board - castling rights (4 planes), total move count (2 planes) and the current player color (1 plane).</p>
  <p>Note that the positions of player's figures and opponent's figures are encoded in fixed layers of the state tensor. If you don't flip the board to the perspective of the player then the network will have very different training inputs for black and white states. It also will have to figure out which direction the pawns can move depending on the current player color. None of that it is impossible, of course - but that unnecessarily complicates something that is already a very hard problem for the DNN to learn.</p>
  <p>You can go further and completely split the training for white and black players, as you've described. But that'll essentially double the work you'll have to do train your nets (and, I suspect, there would be some stability troubles typical for adversarial training).</p>
  <p>To summarize - you are generally right - there is no fundamental need to flip the board. All the above details in state encoding are done to simplify the learning task for the deep neural network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-22T17:42:06.937Z
Answers.Text: Thanks for your explanation! (And I was the person who asked the question you linked haha, thanks again for all your help.)
Answers.UserDisplayName: Akshay Ghosh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-22T16:58:17.007Z
Answers.LastActivityDate: 2021-04-22T16:58:17.007Z
Answers.UserDisplayName: Kostya
Answers.Body: <p>I am not an expert in RL. I have been playing Go for some years.</p>
  <p>Let's quote from AlphaZero's paper first:</p>
  <blockquote>
  <p>Aside from
  komi, the rules of Go are also invariant to colour transposition; this knowledge is
  exploited by representing the board from the perspective of the current player (see
  Neural network architecture).</p>
  </blockquote>
  <p>In the Game of Go, the difference between Black and White except the board representation is the komi (the amount of points that Black has to compensate White in the final count for playing first). Except the presence of komi, there should be no difference in strategy under the same position if colours exchanged. In other words, given a state <span class="math-container">$s$</span> of black stones and white stones on the board, if the optimal policy of Black playing first is <span class="math-container">$\pi$</span>, then if colours of stones on the board exchanged and it is White's turn, the optimal policy for White should be the same as <span class="math-container">$\pi$</span>.</p>
  <p>With this in consideration, there are at least 2 advantages of using a network that represents the board in the perspective of Self/Opponent rather than Black/White.</p>
  <p>The first is that it prevents the network from the possibility of giving inconsistent strategies under two representations of the same state. Consider a network <span class="math-container">$f_\theta$</span> that accepts the board representation in the order of <span class="math-container">$(B,W)$</span>, and a state <span class="math-container">$s = (X_t,Y_t)$</span> in which <span class="math-container">$X_t$</span> is a feature map for black stones and <span class="math-container">$Y_t$</span> is a feature map for white stones and it is black's turn. Now consider a state <span class="math-container">$s' = (Y_t,X_t)$</span> (i.e. colours flipped) and it is white's turn. <span class="math-container">$s$</span> and <span class="math-container">$s'$</span> are essentially representation of the same state (except Komi which does not affect optimal policy). There could be a possibility that the network <span class="math-container">$f_\theta$</span> gives different policies for these two representations. However, if <span class="math-container">$f_\theta$</span> accepts the state as <span class="math-container">$(Self,Opponent)$</span>, the input to the network would be the same (except the komi feature).</p>
  <p>Therefore, this representation would significantly reduce number of states represented by the features vector <span class="math-container">$(X_t,Y_t)$</span>, which would be the second advantage to training the neural network. If we consider that in Go, the same local position could appear in exchanged colour in another position, the network could by this implementation, recognize them as the same position. A decrease in the number of states could mean a significant drop in parameters and power needed from the network.</p>
  <p>The same principle of making use of different representations of the same state is followed in AlphaGo's other training implementations as well, such as augmenting its training data to include rotations and reflections of the same board position.</p>
  <p>However, in the game of Chess, this would be a different case. For a chess position, if the pieces' colours are exchanged and it becomes opponent's turn, it would be a different state because the positions of the KING and the QUEEN are not the same for the two colours.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-22T17:43:15.870Z
Answers.Text: Thank you! I was just curious as to whether it was necessary or not, but now I realize the problems that are fixed when you flip the board.
Answers.UserDisplayName: Akshay Ghosh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-22T14:58:52.097Z
Answers.LastActivityDate: 2021-04-22T14:58:52.097Z
Answers.UserDisplayName: loct
Answers.Body: <p>In board games, whose turn is it play makes a significant difference in the outcome of the game. Many times its a difference of loss or a win.</p>
  <p>So this information has to be communicated to the neural network.
  There are two ways to communicate that information:</p>
  <ol>
  <li><p>To add extra input nodes which say whose turn it is to make move.</p>
  </li>
  <li><p>To simply flip the signs of the pieces.</p>
  </li>
  </ol>
  <p>Method 1 works but it increases the number of nodes hence making the neural network more complex and larger. So method 2 is preferred to keep the neural network simple without losing the needed information. This process is known as <strong>canonical form of the board</strong>.</p>
  <p>Flipping signs wouldn't work for games that have more than two players.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-27T21:36:01.707Z
Answers.Text: Note that the AlphaZero paper has both of these: the board is flipped to current player's orientation AND the current color is passed in.
Answers.UserDisplayName: Ben G
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-06T05:47:23.420Z
Answers.LastActivityDate: 2022-10-06T05:47:23.420Z
Answers.UserDisplayName: Dimanjan
Body: <p>While reading the AlphaZero paper in preparation to code my own RL algorithm to play Chess decently well, I saw that the</p>
  <blockquote>
  <p>&quot;The board is oriented to the perspective of the current player.&quot;</p>
  </blockquote>
  <p>I was wondering why this is the case if there are two agents (black and white). Is it because there is only one central DCNN network used for board and move evaluation (i.e. there aren't two separate networks/policies used for the respective players - black and white) in the algorithm AlphaZero uses to generate moves?</p>
  <p>If I were to implement a black move policy and a white move policy for the respective agents in my environment, would reflecting the board to match the perspective of the current player be necessary since theoretically the black agent should learn black's perspective of moves while the white agent should learn white's perspective of moves?</p>
  
Tags: <neural-networks><reinforcement-learning><alphazero><chess><multi-agent-systems>
Title: Why does Alpha Zero's Neural Network flip the board to be oriented towards the current player?
UserDisplayName: Akshay Ghosh
ThreadText: # Why does Alpha Zero's Neural Network flip the board to be oriented towards the current player?
  
  (Asked by: username_0 on 2021-04-22)
  
  While reading the AlphaZero paper in preparation to code my own RL algorithm to play Chess decently well, I saw that the
  
  > "The board is oriented to the perspective of the current player."
  
  I was wondering why this is the case if there are two agents (black and white). Is it because there is only one central DCNN network used for board and move evaluation (i.e. there aren't two separate networks/policies used for the respective players - black and white) in the algorithm AlphaZero uses to generate moves?
  
  If I were to implement a black move policy and a white move policy for the respective agents in my environment, would reflecting the board to match the perspective of the current player be necessary since theoretically the black agent should learn black's perspective of moves while the white agent should learn white's perspective of moves?
  
  **Tags:** neural-networks, reinforcement-learning, alphazero, chess, multi-agent-systems
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2021-04-22. Score: 2 | Accepted answer
  
  There is a single neural network that guides self-plays in the Monte Carlo Tree Search algorithm. The neural network gets the current state of the board $s$ as an input and outputs current policy $\pi(a|s)$ and value $v(s)$.
  
  The action probabilities are encoded in a (8,8,73) tensor. First two dimensions encode the coordinates of the figure to "pick" from the board. The third dimension encode where to move this figure: check out [this question](https://ai.stackexchange.com/questions/27336/how-does-the-alpha-zeros-move-encoding-work) for a discussion on how all possible moves are encoded in a 73 dimensional vector.
  
  Similarly, the inputs of the network are organized in the (8, 8, 14 \* 8 + 7 = 119) tensor. The first two 8 x 8 dimensions, again, encode the positions on the board. Then the positions of the figures one plane per 6 figure types: first 6 planes for player's figures, next 6 planes for opponent's figures and two [repetition planes](https://ai.stackexchange.com/a/26656/20538). The 14 planes are repeated 8 times supplying predecessor positions to the network. Finally, there are 7 extra planes encoding as a single uniform value over the board - castling rights (4 planes), total move count (2 planes) and the current player color (1 plane).
  
  Note that the positions of player's figures and opponent's figures are encoded in fixed layers of the state tensor. If you don't flip the board to the perspective of the player then the network will have very different training inputs for black and white states. It also will have to figure out which direction the pawns can move depending on the current player color. None of that it is impossible, of course - but that unnecessarily complicates something that is already a very hard problem for the DNN to learn.
  
  You can go further and completely split the training for white and black players, as you've described. But that'll essentially double the work you'll have to do train your nets (and, I suspect, there would be some stability troubles typical for adversarial training).
  
  To summarize - you are generally right - there is no fundamental need to flip the board. All the above details in state encoding are done to simplify the learning task for the deep neural network.
  
  > **Comments:**
  > **username_0**: Thanks for your explanation! (And I was the person who asked the question you linked haha, thanks again for all your help.)  
  
  ---
  
  ### Answer by username_2 on 2021-04-22. Score: 3
  
  I am not an expert in RL. I have been playing Go for some years.
  
  Let's quote from AlphaZero's paper first:
  
  > Aside from
  > komi, the rules of Go are also invariant to colour transposition; this knowledge is
  > exploited by representing the board from the perspective of the current player (see
  > Neural network architecture).
  
  In the Game of Go, the difference between Black and White except the board representation is the komi (the amount of points that Black has to compensate White in the final count for playing first). Except the presence of komi, there should be no difference in strategy under the same position if colours exchanged. In other words, given a state $s$ of black stones and white stones on the board, if the optimal policy of Black playing first is $\pi$, then if colours of stones on the board exchanged and it is White's turn, the optimal policy for White should be the same as $\pi$.
  
  With this in consideration, there are at least 2 advantages of using a network that represents the board in the perspective of Self/Opponent rather than Black/White.
  
  The first is that it prevents the network from the possibility of giving inconsistent strategies under two representations of the same state. Consider a network $f\_\theta$ that accepts the board representation in the order of $(B,W)$, and a state $s = (X\_t,Y\_t)$ in which $X\_t$ is a feature map for black stones and $Y\_t$ is a feature map for white stones and it is black's turn. Now consider a state $s' = (Y\_t,X\_t)$ (i.e. colours flipped) and it is white's turn. $s$ and $s'$ are essentially representation of the same state (except Komi which does not affect optimal policy). There could be a possibility that the network $f\_\theta$ gives different policies for these two representations. However, if $f\_\theta$ accepts the state as $(Self,Opponent)$, the input to the network would be the same (except the komi feature).
  
  Therefore, this representation would significantly reduce number of states represented by the features vector $(X\_t,Y\_t)$, which would be the second advantage to training the neural network. If we consider that in Go, the same local position could appear in exchanged colour in another position, the network could by this implementation, recognize them as the same position. A decrease in the number of states could mean a significant drop in parameters and power needed from the network.
  
  The same principle of making use of different representations of the same state is followed in AlphaGo's other training implementations as well, such as augmenting its training data to include rotations and reflections of the same board position.
  
  However, in the game of Chess, this would be a different case. For a chess position, if the pieces' colours are exchanged and it becomes opponent's turn, it would be a different state because the positions of the KING and the QUEEN are not the same for the two colours.
  
  > **Comments:**
  > **username_0**: Thank you! I was just curious as to whether it was necessary or not, but now I realize the problems that are fixed when you flip the board.  
  
  ---
  
  ### Answer by username_3 on 2022-10-06. Score: 0
  
  In board games, whose turn is it play makes a significant difference in the outcome of the game. Many times its a difference of loss or a win.
  
  So this information has to be communicated to the neural network.
  There are two ways to communicate that information:
  
  1. To add extra input nodes which say whose turn it is to make move.
  2. To simply flip the signs of the pieces.
  
  Method 1 works but it increases the number of nodes hence making the neural network more complex and larger. So method 2 is preferred to keep the neural network simple without losing the needed information. This process is known as **canonical form of the board**.
  
  Flipping signs wouldn't work for games that have more than two players.
  
  > **Comments:**
  > **username_4**: Note that the AlphaZero paper has both of these: the board is flipped to current player's orientation AND the current color is passed in.  
  
  ---
  
  