Answers.Body: <p>Let's do the code, so all the details are down.</p>
  <h3>Encoding dictionary:</h3>
  <pre><code>codes, i = {}, 0
  for nSquares in range(1,8):
      for direction in [&quot;N&quot;, &quot;NE&quot;, &quot;E&quot;, &quot;SE&quot;, &quot;S&quot;, &quot;SW&quot;, &quot;W&quot;, &quot;NW&quot;]:
          codes[(nSquares,direction)] = i
          i += 1
  </code></pre>
  <p>You'll see that the <code>codes</code> dictionary will have 56 entries in it for each <code>(nSquares,direction)</code> pair.</p>
  <p>The knight moves we'll encode as the long &quot;two&quot;-cell edge move first and the short &quot;one&quot;-cell edge second:</p>
  <pre><code>for two in [&quot;N&quot;,&quot;S&quot;]:
      for one in [&quot;E&quot;,&quot;W&quot;]:
          codes[(&quot;knight&quot;, two, one)] , i = i , i + 1
  for two in [&quot;E&quot;,&quot;W&quot;]:
      for one in [&quot;N&quot;,&quot;S&quot;]:
          codes[(&quot;knight&quot;, two, one)] , i = i , i + 1
  </code></pre>
  <p>Now we should have 64 codes. As I understand, the final 9 moves are when a pawn reaches the final rank and chosen to be <a href="https://en.wikipedia.org/wiki/Promotion_(chess)#Underpromotion" rel="noreferrer">underpromoted</a>. It can reach teh final rank either by moving <code>N</code>, or by capturing <code>NE</code>, <code>NW</code>. Underpromotion is possible to three pieces. Writing the code:</p>
  <pre><code>for move in [&quot;N&quot;,&quot;NW&quot;,&quot;NE&quot;]:
      for promote_to in [&quot;Rook&quot;,&quot;Knight&quot;,&quot;Bishop&quot;]:
          codes[(&quot;underpromotion&quot;, move, promote_to)] , i = i , i + 1
  </code></pre>
  <p>We get 73 codes as described.</p>
  <h3>Policy</h3>
  <p>The distribution over actions is a <code>(8,8,73)</code> tensor (it is not formally a &quot;policy&quot;, since policy should also depend on state, but lets cut this corner for this discussion):</p>
  <pre><code>policy = np.zeros((8,8,73))
  </code></pre>
  <p>Let's also do codes for columns for convenience:</p>
  <pre><code>columns = { k:v for v,k in enumerate(&quot;abcdefgh&quot;)}
  </code></pre>
  <blockquote>
  <p>How would one numerically represent the move 1. e4</p>
  </blockquote>
  <p>The first two dimensions choose the figure you are moving. So, that'd be <code>e2</code> pawn. And we move north 'N' by 2 cells.</p>
  <p>So, we put 1 into the tensor at the appropriate indices. Note that you have to subtract 1 from the row index, to make it zero-based.</p>
  <pre><code>e4policy = np.zeros((8,8,73))
  e4policy[ columns['e'] , 2 - 1 , codes[(2 , &quot;N&quot;)]] = 1
  </code></pre>
  <blockquote>
  <p>How would one numerically represent the move 1. NF3</p>
  </blockquote>
  <p>The first two dimensions choose the figure you are moving. So, that'd be <code>g1</code> knight. And we perform north-west <code>N</code>,<code>W</code> knight jump.</p>
  <pre><code>NF3policy = np.zeros((8,8,73))
  NF3policy[ columns['g'] , 1 - 1 , codes[(&quot;knight&quot;, 'N' , 'W')]] = 1
  </code></pre>
  <p>Generally, the policy is a probability distribution over all possible moves, so the policy tensor would have several non-zero probability values in it. For example an opening policy that does <code>1.e4</code> or <code>1.Nf3</code> with 50/50 probability would be:</p>
  <pre><code>openingPolicy = (e4policy + NF3policy) / 2 
  </code></pre>
  <p>Hope this clears things up.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-14T21:14:49.990Z
Answers.Text: Thanks for such a detailed explanation! I assume in order to make this more appropriate to sample, we could flatten the 8x8x73 array into a 1d 4762 array with each integer representing it's respective move? This would make more sense in the context of training via self play, correct?
Answers.UserDisplayName: Akshay Ghosh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-14T21:46:31.037Z
Answers.Text: @Akshay There are quite a lot of details there. This tensor is an output of  Policy Network, that gets trained by the whole MCTS machinery. You also have to make sure that probabilities of invalid moves are zero and that all probabilities sum to one. For more detailed explanation I suggest studying the paper and asking more questions.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-22T17:04:13.023Z
Answers.Text: thanks for great explanation for encoding the output of AlphaZero, however I struggle with decoding from move probabilities to real move. Do you have idea how to make it?
Answers.UserDisplayName: Filip Gacek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-14T18:51:06.517Z
Answers.LastActivityDate: 2021-04-14T18:51:06.517Z
Answers.UserDisplayName: Kostya
Body: <p>I am a beginner in AI. I'm trying to train a multi-agent RL algorithm to play chess. One issue that I ran into was representing the action space (legal moves/or honestly just moves in general) numerically. I looked up how Alpha Zero represented it, and they used an 8x8x73 array to encode all possible moves. I was wondering how it actually works since I got a bit confused in their explanation:</p>
  <blockquote>
  <p>A move in chess may be described in two parts: selecting the piece to move, and then selecting among the legal moves for that piece. We represent the policy <span class="math-container">$\pi(a \mid s)$</span> by a <span class="math-container">$8 \times 8 \times 73$</span> stack of planes encoding a probability distribution over 4,672 possible moves. Each of the <span class="math-container">$8 \times 8$</span> positions identifies the square from which to &quot;pick up&quot; a piece. The first 56 planes encode possible &quot;queen moves&quot; for any piece: a number of squares <span class="math-container">$[1..7]$</span> in which the piece will be moved, along one of eight relative compass directions {N, NE, E, SE, S, SW, W, NW}. The next 8 planes encode possible knight moves for that piece. The final 9 planes encode possible under-promotions for pawn moves or captures in two possible diagonals, to knight, bishop or rook respectively. Other pawn moves or captures from the seventh rank are promoted to a queen.</p>
  </blockquote>
  <p>How would one numerically represent the move 1. e4 or 1. NF3 (and how would the integer for 1. NF3 differ from 1. f3) for example? How do you tell what integer corresponds to which move? This is what I'm essentially asking.</p>
  
Tags: <reinforcement-learning><alphazero><chess><multi-agent-systems><action-spaces>
Title: How does the Alpha Zero's move encoding work?
UserDisplayName: Akshay Ghosh
ThreadText: # How does the Alpha Zero's move encoding work?
  
  (Asked by: username_0 on 2021-04-14)
  
  I am a beginner in AI. I'm trying to train a multi-agent RL algorithm to play chess. One issue that I ran into was representing the action space (legal moves/or honestly just moves in general) numerically. I looked up how Alpha Zero represented it, and they used an 8x8x73 array to encode all possible moves. I was wondering how it actually works since I got a bit confused in their explanation:
  
  > A move in chess may be described in two parts: selecting the piece to move, and then selecting among the legal moves for that piece. We represent the policy $\pi(a \mid s)$ by a $8 \times 8 \times 73$ stack of planes encoding a probability distribution over 4,672 possible moves. Each of the $8 \times 8$ positions identifies the square from which to "pick up" a piece. The first 56 planes encode possible "queen moves" for any piece: a number of squares $[1..7]$ in which the piece will be moved, along one of eight relative compass directions {N, NE, E, SE, S, SW, W, NW}. The next 8 planes encode possible knight moves for that piece. The final 9 planes encode possible under-promotions for pawn moves or captures in two possible diagonals, to knight, bishop or rook respectively. Other pawn moves or captures from the seventh rank are promoted to a queen.
  
  How would one numerically represent the move 1. e4 or 1. NF3 (and how would the integer for 1. NF3 differ from 1. f3) for example? How do you tell what integer corresponds to which move? This is what I'm essentially asking.
  
  **Tags:** reinforcement-learning, alphazero, chess, multi-agent-systems, action-spaces
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-14. Score: 6 | Accepted answer
  
  Let's do the code, so all the details are down.
  
  ### Encoding dictionary:
  
  ```
  codes, i = {}, 0
  for nSquares in range(1,8):
      for direction in ["N", "NE", "E", "SE", "S", "SW", "W", "NW"]:
          codes[(nSquares,direction)] = i
          i += 1
  
  ```
  
  You'll see that the `codes` dictionary will have 56 entries in it for each `(nSquares,direction)` pair.
  
  The knight moves we'll encode as the long "two"-cell edge move first and the short "one"-cell edge second:
  
  ```
  for two in ["N","S"]:
      for one in ["E","W"]:
          codes[("knight", two, one)] , i = i , i + 1
  for two in ["E","W"]:
      for one in ["N","S"]:
          codes[("knight", two, one)] , i = i , i + 1
  
  ```
  
  Now we should have 64 codes. As I understand, the final 9 moves are when a pawn reaches the final rank and chosen to be [underpromoted](https://en.wikipedia.org/wiki/Promotion_(chess)#Underpromotion). It can reach teh final rank either by moving `N`, or by capturing `NE`, `NW`. Underpromotion is possible to three pieces. Writing the code:
  
  ```
  for move in ["N","NW","NE"]:
      for promote_to in ["Rook","Knight","Bishop"]:
          codes[("underpromotion", move, promote_to)] , i = i , i + 1
  
  ```
  
  We get 73 codes as described.
  
  ### Policy
  
  The distribution over actions is a `(8,8,73)` tensor (it is not formally a "policy", since policy should also depend on state, but lets cut this corner for this discussion):
  
  ```
  policy = np.zeros((8,8,73))
  
  ```
  
  Let's also do codes for columns for convenience:
  
  ```
  columns = { k:v for v,k in enumerate("abcdefgh")}
  
  ```
  
  > How would one numerically represent the move 1. e4
  
  The first two dimensions choose the figure you are moving. So, that'd be `e2` pawn. And we move north 'N' by 2 cells.
  
  So, we put 1 into the tensor at the appropriate indices. Note that you have to subtract 1 from the row index, to make it zero-based.
  
  ```
  e4policy = np.zeros((8,8,73))
  e4policy[ columns['e'] , 2 - 1 , codes[(2 , "N")]] = 1
  
  ```
  
  > How would one numerically represent the move 1. NF3
  
  The first two dimensions choose the figure you are moving. So, that'd be `g1` knight. And we perform north-west `N`,`W` knight jump.
  
  ```
  NF3policy = np.zeros((8,8,73))
  NF3policy[ columns['g'] , 1 - 1 , codes[("knight", 'N' , 'W')]] = 1
  
  ```
  
  Generally, the policy is a probability distribution over all possible moves, so the policy tensor would have several non-zero probability values in it. For example an opening policy that does `1.e4` or `1.Nf3` with 50/50 probability would be:
  
  ```
  openingPolicy = (e4policy + NF3policy) / 2 
  
  ```
  
  Hope this clears things up.
  
  > **Comments:**
  > **username_0**: Thanks for such a detailed explanation! I assume in order to make this more appropriate to sample, we could flatten the 8x8x73 array into a 1d 4762 array with each integer representing it's respective move? This would make more sense in the context of training via self play, correct?  
  > **username_1**: @username_0 There are quite a lot of details there. This tensor is an output of  Policy Network, that gets trained by the whole MCTS machinery. You also have to make sure that probabilities of invalid moves are zero and that all probabilities sum to one. For more detailed explanation I suggest studying the paper and asking more questions.  
  > **username_2**: thanks for great explanation for encoding the output of AlphaZero, however I struggle with decoding from move probabilities to real move. Do you have idea how to make it?  
  
  ---
  
  