Answers.Body: <p>The output of the policy network is as described in the original paper:</p>
  
  <blockquote>
    <p>A move in chess may be described in two parts: selecting the piece to move, and then
    selecting among the legal moves for that piece. We represent the policy π(a|s) by a 8 × 8 × 73
    stack of planes encoding a probability distribution over 4,672 possible moves. Each of the 8×8
    positions identifies the square from which to “pick up” a piece. The first 56 planes encode
    possible ‘queen moves’ for any piece: a number of squares [1..7] in which the piece will be
    moved, along one of eight relative compass directions {N, NE, E, SE, S, SW, W, NW}. The
    next 8 planes encode possible knight moves for that piece. The final 9 planes encode possible
    underpromotions for pawn moves or captures in two possible diagonals, to knight, bishop or
    rook respectively. Other pawn moves or captures from the seventh rank are promoted to a
    queen.</p>
  </blockquote>
  
  <p>So each move selector scores the relative probability of selecting a piece in a given square and moving it in a specific way. For example, there is always one output dedicated to representing picking up the piece in A3 and moving it to A6. This representation includes selecting opponent pieces, selecting empty squares, making knight moves for rooks, making long diagonal moves for pawns. It also includes moves that take pieces off the board or through other blocking pieces. </p>
  
  <p>The typical branching factor in chess is around 35. The policy network described above always calculates discrete probabilities for 4672 moves.</p>
  
  <p>Clearly this <em>can</em> select many non-valid moves, if pieces are not available, or cannot move as suggested. In fact it does this all the time, even when fully trained, as nothing is ever learned about avoiding the non-valid moves during training - they do not receive positive or negative feedback, as there is never any experience gained relating to them. However, the benefit is that this structure is simple and fixed, both useful traits when building a neural network.</p>
  
  <p>The simple work-around is to filter out impossible moves logically, setting their effective probability to zero, and then re-normalise the probabilities for the remaining valid moves. That step involves asking the game engine for what the valid moves are - but that's fine, it's not "cheating".</p>
  
  <p>Whilst it might be possible to either have the agent learn to avoid non-valid moves, or some clever output structure that could only express valid moves, these would both distract from the core goal of learning how to play the game optimally.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-14T22:07:34.477Z
Answers.Text: Alphazero can play games where pieces move such as chess. How then would such an encoding work?
Answers.UserDisplayName: chessprogrammer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-15T08:11:49.987Z
Answers.Text: @chessprogrammer: I have updated the answer with details about the chess representation as used by AlphaZero
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-14T21:48:36.547Z
Answers.LastActivityDate: 2018-09-15T08:50:28.727Z
Answers.LastEditDate: 2018-09-15T08:50:28.727Z
Answers.UserDisplayName: Neil Slater
Body: <p>In AlphaZero, the <em>policy network</em> (or head of the network) maps game states to a distribution of the likelihood of taking each action. This distribution covers all possible actions from that state.</p>
  
  <p>How is such a network possible? The possible actions from each state are vastly different than subsequent states. So, how would each possible action from a given state be represented in the network's output, and what about the network design would stop the network from considering an illegal action?</p>
  
Tags: <neural-networks><reinforcement-learning><ai-design><alphazero><alphago-zero>
Title: Why does the policy network in AlphaZero work?
UserDisplayName: chessprogrammer
ThreadText: # Why does the policy network in AlphaZero work?
  
  (Asked by: username_0 on 2018-09-14)
  
  In AlphaZero, the *policy network* (or head of the network) maps game states to a distribution of the likelihood of taking each action. This distribution covers all possible actions from that state.
  
  How is such a network possible? The possible actions from each state are vastly different than subsequent states. So, how would each possible action from a given state be represented in the network's output, and what about the network design would stop the network from considering an illegal action?
  
  **Tags:** neural-networks, reinforcement-learning, ai-design, alphazero, alphago-zero
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-09-14. Score: 21 | Accepted answer
  
  The output of the policy network is as described in the original paper:
  
  > A move in chess may be described in two parts: selecting the piece to move, and then
  > selecting among the legal moves for that piece. We represent the policy π(a|s) by a 8 × 8 × 73
  > stack of planes encoding a probability distribution over 4,672 possible moves. Each of the 8×8
  > positions identifies the square from which to “pick up” a piece. The first 56 planes encode
  > possible ‘queen moves’ for any piece: a number of squares [1..7] in which the piece will be
  > moved, along one of eight relative compass directions {N, NE, E, SE, S, SW, W, NW}. The
  > next 8 planes encode possible knight moves for that piece. The final 9 planes encode possible
  > underpromotions for pawn moves or captures in two possible diagonals, to knight, bishop or
  > rook respectively. Other pawn moves or captures from the seventh rank are promoted to a
  > queen.
  
  So each move selector scores the relative probability of selecting a piece in a given square and moving it in a specific way. For example, there is always one output dedicated to representing picking up the piece in A3 and moving it to A6. This representation includes selecting opponent pieces, selecting empty squares, making knight moves for rooks, making long diagonal moves for pawns. It also includes moves that take pieces off the board or through other blocking pieces.
  
  The typical branching factor in chess is around 35. The policy network described above always calculates discrete probabilities for 4672 moves.
  
  Clearly this *can* select many non-valid moves, if pieces are not available, or cannot move as suggested. In fact it does this all the time, even when fully trained, as nothing is ever learned about avoiding the non-valid moves during training - they do not receive positive or negative feedback, as there is never any experience gained relating to them. However, the benefit is that this structure is simple and fixed, both useful traits when building a neural network.
  
  The simple work-around is to filter out impossible moves logically, setting their effective probability to zero, and then re-normalise the probabilities for the remaining valid moves. That step involves asking the game engine for what the valid moves are - but that's fine, it's not "cheating".
  
  Whilst it might be possible to either have the agent learn to avoid non-valid moves, or some clever output structure that could only express valid moves, these would both distract from the core goal of learning how to play the game optimally.
  
  > **Comments:**
  > **username_0**: Alphazero can play games where pieces move such as chess. How then would such an encoding work?  
  > **username_1**: @username_0: I have updated the answer with details about the chess representation as used by AlphaZero  
  
  ---
  
  