Answers.Body: <p>Other answers are right in that AlphaZero was implicitly told that castling is a move that exists, I'll elaborate on how exactly that happens by explaining the inputs and outputs of the neural network.</p>
  <h1>Input format</h1>
  <p><em>Note: The exact details and screenshots are from my own implementation, which is broadly similar to AlphaZero. At the end of this answer I note the differences between this input format and the one the real AlphaZero uses.</em></p>
  <p>At the core of AlphaZero is a neural network, which gets as an input the current board state. This board state is encoded as a stack of images, from the point of view of the current player. For the starting position, this looks like this:</p>
  <p><a href="https://i.sstatic.net/ky6ax.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ky6ax.png" alt="Input format visualization for the starting position" /></a></p>
  <p>From left to right:</p>
  <ul>
  <li>2 planes: Is it whites turn, is it blacks turn</li>
  <li>4 planes: Castling rights for us left and right, castling rights for the opponent left and right</li>
  <li>2 planes: Binary digits for the repetition counter</li>
  <li>6 planes: The side to move's pieces</li>
  <li>6 planes: The opponent pieces</li>
  <li>1 plane: The en-passant square if any</li>
  </ul>
  <p>In this context <a href="https://www.chessprogramming.org/Castling_Rights" rel="nofollow noreferrer">castling rights</a> means whether the king or rooks have moved in the past, not whether castling is currently available as a move.</p>
  <p>The network is never explicitly told that the 3rd to 6th planes have anything to do with castling or even moving the king, it has to figure that out itself during training.</p>
  <h1>Output move format</h1>
  <p>The network has two outputs, the first is the estimated <em>value</em> of the position and the second is an estimation of how good each possible move is, called the <em>policy</em> head. This second output is again a stack of images, and looks like this:</p>
  <p><a href="https://i.sstatic.net/fwrWd.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fwrWd.png" alt="Policy format visualization for the starting position" /></a></p>
  <p>In reality this is a stack of 72 images, I cut out a bunch of them in the center. The encoding here is that the square within the image is the square the move starts from, and the plane means what kind of move it is. For example, the first plane means &quot;move one square forward&quot;, the second &quot;move two squares forward&quot;, the 10th plane counting from the end means &quot;do a knight move forward and left&quot;. As a full example, in this position the bottom left square in the first image means &quot;move the rook on A1 to A2&quot;.</p>
  <p>This encoding does not include which piece actually makes the move. Pushing the e pawn two squares looks the same as moving a hypothetical queen from e2 to e4.</p>
  <p>The first row is the actual output of the network without any post processing. As you can see this output is full of nonsensical moves with starting squares on top of opponent pieces. To fix this only available moves are considered, the rest are set to zero and the output is re-normalized. The result is shown in the second row. As you can see in the starting position the network likes pushing the 4 center pawns two squares forward, and also moving the kingside knight to the center of the board.</p>
  <p><strong>Importantly this step of ignoring of invalid moves and renormalizing is the only place where knowledge of chess moves is put into the AlphaZero training process! The valid moves are not actually given as a network input or anywhere else, they're only considered when computing the network output. The network is not discouraged from suggesting invalid moves, these suggestions are simply ignored.</strong></p>
  <h1>Castling</h1>
  <p>Of course this question was actually about castling, so let's look at a position where the network wants to castle (and can). Like all other moves castling is encoded as moving from one square (e1) in certain way (two squares to the right for kingside castling).</p>
  <p>Fen string:</p>
  <p><code>r1bqkb1r/1ppp1ppp/p1n2n2/4p3/B3P3/5N2/PPPP1PPP/RNBQK2R w KQkq - 2 5</code></p>
  <p><a href="https://i.sstatic.net/dwhcU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/dwhcU.png" alt="Input &amp; output policy visualization for a position where castling is possible" /></a></p>
  <p>The first row is the board input, the second row the policy before removing the invalid moves and the last row is the final policy output. The bright white pixel in the bottom center encodes &quot;e1, two squares to the right&quot; which means castle kingside. Other moves the network is considering are pushing more center pawns and moving the f3 knight again.</p>
  <h1>Bonus</h1>
  <p>It's also possible to visualize the full network internals in this way, here are some examples:</p>
  <ul>
  <li><a href="https://i.sstatic.net/OyvDl.jpg" rel="nofollow noreferrer">The starting position</a></li>
  <li><a href="https://i.sstatic.net/ut4a3.jpg" rel="nofollow noreferrer">The castling position discussed before</a></li>
  </ul>
  <p>I don't think much insight can be gained from this but it looks cool!</p>
  <h1>Sources</h1>
  <p>Most of what is in this answer is also explained pretty well in the <a href="https://arxiv.org/pdf/1712.01815.pdf" rel="nofollow noreferrer">AlphaZero paper</a> itself, specifically the &quot;Representation&quot; section.</p>
  <p>The images were generated by my own work-in-progress implementation attempt, <a href="https://github.com/KarelPeeters/kZero" rel="nofollow noreferrer">kZero</a>. The network shown matches the actual AlphaZero network very closely, but was trained on the <a href="https://database.lichess.org/" rel="nofollow noreferrer">lichess database</a> instead of by real selfplay.</p>
  <h1>Differences with the real AlphaZero input format</h1>
  <ul>
  <li>In the real AlphaZero, the position of the pieces is given for each of the 8 previous board states. Chess is not fully described by the current board state, see eg. the <a href="https://en.wikipedia.org/wiki/Fifty-move_rule" rel="nofollow noreferrer">Fifty-move rule</a> or <a href="https://en.wikipedia.org/wiki/Threefold_repetition" rel="nofollow noreferrer">Threefold repetition</a>. Adding 8 steps of history helps, but it's still not really enough.</li>
  <li>AlphaZero doesn't need an en-passant plane since the network can just compare the previous two board states to see if en passant is possible.</li>
  <li>The are some slight differences around player colour and move counter planes, but these are not very significant.</li>
  </ul>
  <p>All together this means AlphaZero had 8 * 14 + 7 = 119 planes, whereas I only talked about 21 for most of this answer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-27T19:12:24.253Z
Answers.Text: Wonderful answer! Could you provide some sources? I'm very interested in learning more. In particular, where did the images come from?
Answers.UserDisplayName: Ben Mares
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-27T19:47:01.077Z
Answers.Text: @BenMares Thanks! I've added a section about my sources and how I made the images.
Answers.UserDisplayName: KarelPeeters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-30T14:39:15.697Z
Answers.Text: Is one of the depicted frames "move knight 2 forward and 1 right"? If so, it's interesting that AZ seems to have completely rejected 1. Nc3 as an opening move.
Answers.UserDisplayName: eyeballfrog
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-30T15:12:51.143Z
Answers.Text: @eyeballfrog No that one got cut out, the last 9 planes are for underpromotions and the eight before that are for knight moves. Here's how much it likes all possible opening moves: https://pastebin.com/raw/yU9yDGUF. Keep in mind that this is not the real AZ network, I just trained it to imitate high-elo lichess games. The linked paper does contain some details about what openings the real AZ likes.
Answers.UserDisplayName: KarelPeeters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-17T13:45:02.993Z
Answers.Text: @ToddSewell According to the AlphaZero paper, the input shape is 119x8x8, not 21x8x8.  What is your source for the change in input? The AlphaZero paper also doesn't mention the en passant square being included in the input. Is this a personal change you made?
Answers.UserDisplayName: zjeffer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-17T17:14:39.967Z
Answers.Text: @zjeffer You're right, I chose to simplify things somewhat (both for my implementation and this answer). I've added a note and some more details.
Answers.UserDisplayName: KarelPeeters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-25T12:59:32.980Z
Answers.LastActivityDate: 2022-04-17T17:14:05.557Z
Answers.LastEditDate: 2022-04-17T17:14:05.557Z
Answers.UserDisplayName: KarelPeeters
Answers.Body: <blockquote>
  <p>was AlphaZero just given this move</p>
  </blockquote>
  <p>It was given this move in the same way that it was given all other moves. It started with complete knowledge of the rules of the game. In machine learning terms, castling would be part of its baseline knowledge of <em>state transitions</em>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-24T16:24:34.383Z
Answers.LastActivityDate: 2021-11-24T16:24:34.383Z
Answers.UserDisplayName: Cleveland
Answers.Body: <p>The AlphaZero paper &quot;<a href="https://arxiv.org/abs/1712.01815" rel="noreferrer">Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm</a>&quot; by David Silver et al. contains this explanation</p>
  <blockquote>
  <p>Knowledge of the rules is also used to encode the input planes (i.e. castling, repetition,
  no-progress) and output planes (how pieces move, promotions, and piece drops in shogi).</p>
  </blockquote>
  <p>(An input and output plane are the terms of art for how the board and possible moves are represented to the AlphaZero model.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-25T02:47:15.867Z
Answers.LastActivityDate: 2021-11-25T02:47:15.867Z
Answers.UserDisplayName: Sycorax
Answers.Body: <blockquote>
  <p>was AlphaZero just given this move, in the same way it was told that players take turns and a win is better than a loss?</p>
  </blockquote>
  <p>Actually you've hit upon 3 very distinct ways that AlphaZero was programmed. Castling is a potential move that's built into the algorithm; AlphaZero doesn't need to learn how to do it, just when to do it. The authors list the all the knowledge encoded into the algorithm on page 12 of their <a href="https://arxiv.org/pdf/1712.01815.pdf" rel="nofollow noreferrer">paper</a>:</p>
  <blockquote>
  <p>Domain Knowledge</p>
  <ol>
  <li>The input features describing the position, and the output features describing the move, are structured as a set of planes; i.e. the
  neural network architecture is matched to the grid-structure of the
  board.</li>
  <li>AlphaZero is provided with perfect knowledge of the game rules. These are used during MCTS, to simulate the positions resulting from a
  sequence of moves, to determine game termination, and to score any
  simulations that reach a terminal state.</li>
  <li>Knowledge of the rules is also used to encode the input planes (i.e. castling, repetition, no-progress) and output planes (how pieces
  move, promotions, and piece drops in shogi).</li>
  <li>The typical number of legal moves is used to scale the exploration noise (see below).</li>
  <li>Chess and shogi games exceeding a maximum number of steps (determined by typical game length) were terminated and assigned a
  drawn outcome; Go games were terminated and scored with Tromp-Taylor
  rules, similarly to previous work (29).</li>
  </ol>
  <p>AlphaZero did not use any form of domain knowledge beyond the points listed above.</p>
  </blockquote>
  <p>In short, at every turn the neural network at the core of AlphaZero receives information about every piece on the board and outputs a number for every possible move allowed in chess. On top of that neural network there is an algorithm that controls the gameplay, and that zeroes out all currently impossible moves (like the neural net wanting to move a queen despite having no queen) then it picks the valid move with the highest number outputted by the neural net (sort of, the actual move is probabilistic). In this sense it was pre-programmed how to castle, and in fact at the beginning of training it has an equal chance of castling as it does moving a pawn (assuming the board state at that turn allows castling). During training the neural net will learn when castling leads to winning the game.</p>
  <p>The other answers already said as much, but I want to add to the other 2 things you mentioned. Taking turns is actually built into the algorithm that manages the game and it's such a central feature that the neural network was designed around it. It's actually quite hard to move away from turn based decision making when designing neural networks, and it's one of the features they address in their AlphaStar paper. AlphaStar plays Starcraft, a real time strategy based game where both players can make as many decisions as they want at any time. They actually don't really solve the continuous decision problem in AlphaStar, they just make the &quot;turns&quot; have a variable duration so the neural net can choose to make decisions every millisecond during intense battles or every 10 seconds if there's not much going on.</p>
  <p>Finally the win/loss thing is programmed into the optimizer side of AlphaZero's algorithm. The neural network at the core of AlphaZero is a mathematical function with lots of parameters, the value of the parameters determine what moves it thinks it's best. In essence you want to keep the parameters when it wins and you want to change parameters when it loses; but the exact way they make those changes is explained in the paper and it's quite ingenious. In any case it's actually quite easy to change the desired outcome (compared to the difficulty of moving away from turn based games). The authors desire the neural net to win at chess for obvious reasons, but they could very easily encode the desire to maximize pawn promotions, as an example.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-25T08:09:36.323Z
Answers.Text: "In this sense it was pre-programmed how to castle, and in fact at the beginning of training it has an equal chance of castling as it does moving a pawn (assuming the board state at that turn allows castling)." I think a more precise statement would be "each castling move has the same probability as each pawn move." If there is one castling move and three pawn moves, I would expect the p(pawn move) =3 p(castling). If there are two castling moves and one pawn move, I would expect p(pawn move) = p(castling)/2.
Answers.UserDisplayName: Acccumulation
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-25T16:01:10.913Z
Answers.Text: Todd Sewell's answer claims that machine doesn't get as an input the possible moves but just spits out a number of actions which are filtered for validity.
Answers.UserDisplayName: akostadinov
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-30T16:47:58.813Z
Answers.Text: @akostadinov yes that's right. The input is the board state, and the neural network output structure is fixed and is meant to encode every possible move. The layer on top of the neural net filters what moves are valid.
Answers.UserDisplayName: csiz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-25T04:05:01.733Z
Answers.LastActivityDate: 2021-11-30T16:45:53.377Z
Answers.LastEditDate: 2021-11-30T16:45:53.377Z
Answers.UserDisplayName: csiz
Answers.Body: <p>I do not understand your point. It may be that you don't understand how engines work. AlphaZero is nothing special to be honest. Let me explain.</p>
  <p>Engines do three things while searching for a move.</p>
  <ol>
  <li>Find out the possible moves.</li>
  <li>Trim out searching for moves which seem useless(in AlphaZero's case, it also trims out any invalid moves)</li>
  <li>Evaluate the position.</li>
  </ol>
  <p>(The neural network can actually evaluate any position but invalid moves are ignored and it figures out that those moves are not worth searching; so the neural network is never explicitly told that some moves are invalid and it has to figure it out itself but the engine itself is informed; this doesn't actually change anything because eventually illegal moves are trimmed out, just at a later stage)</p>
  <p>The first two things that AlphaZero does is the same as any other engine. Castling, is a possible move, so the way that it is generated is the same as any other non neural network engine.</p>
  <p>The thing is that when evaluating a position, other engines used to use a fixed tuned set of values. They would calculate king safety, space, mobility, etc using hardcoded values which were generated by years of tuning values. AlphaZero, on the other hand, uses neural networks for evaluating positions. It's all the difference. AlphaZero is no alien.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-25T17:54:09.443Z
Answers.Text: AlphaZero is different in the sense that the neural network is allowed to consider impossible moves as well.
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-26T12:41:23.703Z
Answers.Text: @Stef Wouldn't that be just inefficient?
Answers.UserDisplayName: Shambhav
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-26T12:43:51.340Z
Answers.Text: Inefficient in comparison to what? They got pretty good results with their method, and it's not obvious how to do it "more efficiently".
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-26T12:50:45.363Z
Answers.Text: Inefficient compared to normal engines... Uh? You could just search for positions using a normal search tree instead of like searching every position in existence. Do I not understand how AlphaZero works?
Answers.UserDisplayName: Shambhav
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-26T12:53:54.867Z
Answers.Text: AlphaZero does use a search tree. The neural networks outputs probabilities for every move, and these probabilities are used to decide how to explore the tree. Illegal moves are ignored when searching the tree. But the neural network isn't really aware of that. Although it probably learns very fast that illegal moves are bad, just like it eventually learns that bad moves are bad.
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-26T13:23:39.520Z
Answers.Text: @Stef What difference does that even make if the illegal moves are ignored by the search tree? Doesn't it mean the same thing. It's just a little quirk for how things are accomplished which doesn't affect the big picture. It's just like saying "Oooh an engine doesn't play a move, it just selects the best variation from the search tree". It's the same thing. What difference does it make if you prune the illegal moves during the generation of the moves or the searching. It means nothing.
Answers.UserDisplayName: Shambhav
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-26T13:26:25.320Z
Answers.Text: What difference between what and what? I don't understand what you're asking.
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-26T13:29:36.987Z
Answers.Text: Also note there is a difference between asking "how is the neural network used to explore the search tree during a game" and "how does the neural network learn".
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-26T14:17:20.340Z
Answers.Text: @Stef I have never asked for any difference. Now the misunderstandings are flooding in and I feel it's best to stop now before this becomes a heated debate caused due to bad communication.
Answers.UserDisplayName: Shambhav
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-23T00:15:46.463Z
Answers.Text: This answer is wrong; Read the other answers for OP’s question and the paper for how AlphaZero works.
Answers.UserDisplayName: Navin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-23T10:24:56.070Z
Answers.Text: @Navin Where is it wrong? Oh yeah, I'll correct it but I don't think the mistake that I found is the same one you found.
Answers.UserDisplayName: Shambhav
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-23T14:49:44.883Z
Answers.Text: This is still wrong. AlphaZero uses a totally different search algorithm than traditional engines. PUCT and AB have major differences.
Answers.UserDisplayName: Oscar Smith
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-25T15:58:06.080Z
Answers.LastActivityDate: 2021-12-23T10:32:50.587Z
Answers.LastEditDate: 2021-12-23T10:32:50.587Z
Answers.UserDisplayName: Shambhav
Body: <p>Since castling involves one player moving two pieces in one move, was AlphaZero just given this move, in the same way it was told that players take turns and a win is better than a loss? In other words, it didn't learn it on its own. (I guess it must have been; the range of possible two, three, four, or five piece moves, for all it knows, would just be too big to learn on its own.)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-25T22:46:49.100Z
Comments.Text: All the answers below make an unnecessary hash of a very simple misconception. You believe that AlphaZero deduced what the legal moves are in chess. This is not the case. AlphaZero was provided the full game rules. It is, effectively, given what the legal moves are every turn.
Comments.UserDisplayName: Mobeus Zoom
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-02T04:14:53.510Z
Comments.Text: @MobeusZoom is correct. In the book *Game Changer* by Sadler and Regan when discussing AlphaZero's learning, they clearly state that A0 began knowing the "rules of the game." Further reading in the book and the more academic papers makes this explicitly clear.  **AlphaZero started its training knowing how to castle.**
Comments.UserDisplayName: SecretAgentMan
Tags: <castling><alphazero>
Title: Was AlphaZero taught castling?
ThreadText: # Was AlphaZero taught castling?
  
  (Asked by: username_0 on 2021-11-24)
  
  Since castling involves one player moving two pieces in one move, was AlphaZero just given this move, in the same way it was told that players take turns and a win is better than a loss? In other words, it didn't learn it on its own. (I guess it must have been; the range of possible two, three, four, or five piece moves, for all it knows, would just be too big to learn on its own.)
  
  **Tags:** castling, alphazero
  
  > **Comments:**
  > **username_1**: All the answers below make an unnecessary hash of a very simple misconception. You believe that AlphaZero deduced what the legal moves are in chess. This is not the case. AlphaZero was provided the full game rules. It is, effectively, given what the legal moves are every turn.  
  > **username_2**: @MobeusZoom is correct. In the book *Game Changer* by Sadler and Regan when discussing AlphaZero's learning, they clearly state that A0 began knowing the "rules of the game." Further reading in the book and the more academic papers makes this explicitly clear.  **AlphaZero started its training knowing how to castle.**  
  
  ---
  
  ## 5 answers
  
  ### Answer by username_3 on 2021-11-25. Score: 54
  
  Other answers are right in that AlphaZero was implicitly told that castling is a move that exists, I'll elaborate on how exactly that happens by explaining the inputs and outputs of the neural network.
  
  # Input format
  
  *Note: The exact details and screenshots are from my own implementation, which is broadly similar to AlphaZero. At the end of this answer I note the differences between this input format and the one the real AlphaZero uses.*
  
  At the core of AlphaZero is a neural network, which gets as an input the current board state. This board state is encoded as a stack of images, from the point of view of the current player. For the starting position, this looks like this:
  
  [![Input format visualization for the starting position](https://i.sstatic.net/ky6ax.png)](https://i.sstatic.net/ky6ax.png)
  
  From left to right:
  
  * 2 planes: Is it whites turn, is it blacks turn
  * 4 planes: Castling rights for us left and right, castling rights for the opponent left and right
  * 2 planes: Binary digits for the repetition counter
  * 6 planes: The side to move's pieces
  * 6 planes: The opponent pieces
  * 1 plane: The en-passant square if any
  
  In this context [castling rights](https://www.chessprogramming.org/Castling_Rights) means whether the king or rooks have moved in the past, not whether castling is currently available as a move.
  
  The network is never explicitly told that the 3rd to 6th planes have anything to do with castling or even moving the king, it has to figure that out itself during training.
  
  # Output move format
  
  The network has two outputs, the first is the estimated *value* of the position and the second is an estimation of how good each possible move is, called the *policy* head. This second output is again a stack of images, and looks like this:
  
  [![Policy format visualization for the starting position](https://i.sstatic.net/fwrWd.png)](https://i.sstatic.net/fwrWd.png)
  
  In reality this is a stack of 72 images, I cut out a bunch of them in the center. The encoding here is that the square within the image is the square the move starts from, and the plane means what kind of move it is. For example, the first plane means "move one square forward", the second "move two squares forward", the 10th plane counting from the end means "do a knight move forward and left". As a full example, in this position the bottom left square in the first image means "move the rook on A1 to A2".
  
  This encoding does not include which piece actually makes the move. Pushing the e pawn two squares looks the same as moving a hypothetical queen from e2 to e4.
  
  The first row is the actual output of the network without any post processing. As you can see this output is full of nonsensical moves with starting squares on top of opponent pieces. To fix this only available moves are considered, the rest are set to zero and the output is re-normalized. The result is shown in the second row. As you can see in the starting position the network likes pushing the 4 center pawns two squares forward, and also moving the kingside knight to the center of the board.
  
  **Importantly this step of ignoring of invalid moves and renormalizing is the only place where knowledge of chess moves is put into the AlphaZero training process! The valid moves are not actually given as a network input or anywhere else, they're only considered when computing the network output. The network is not discouraged from suggesting invalid moves, these suggestions are simply ignored.**
  
  # Castling
  
  Of course this question was actually about castling, so let's look at a position where the network wants to castle (and can). Like all other moves castling is encoded as moving from one square (e1) in certain way (two squares to the right for kingside castling).
  
  Fen string:
  
  `r1bqkb1r/1ppp1ppp/p1n2n2/4p3/B3P3/5N2/PPPP1PPP/RNBQK2R w KQkq - 2 5`
  
  [![Input & output policy visualization for a position where castling is possible](https://i.sstatic.net/dwhcU.png)](https://i.sstatic.net/dwhcU.png)
  
  The first row is the board input, the second row the policy before removing the invalid moves and the last row is the final policy output. The bright white pixel in the bottom center encodes "e1, two squares to the right" which means castle kingside. Other moves the network is considering are pushing more center pawns and moving the f3 knight again.
  
  # Bonus
  
  It's also possible to visualize the full network internals in this way, here are some examples:
  
  * [The starting position](https://i.sstatic.net/OyvDl.jpg)
  * [The castling position discussed before](https://i.sstatic.net/ut4a3.jpg)
  
  I don't think much insight can be gained from this but it looks cool!
  
  # Sources
  
  Most of what is in this answer is also explained pretty well in the [AlphaZero paper](https://arxiv.org/pdf/1712.01815.pdf) itself, specifically the "Representation" section.
  
  The images were generated by my own work-in-progress implementation attempt, [kZero](https://github.com/username_3/kZero). The network shown matches the actual AlphaZero network very closely, but was trained on the [lichess database](https://database.lichess.org/) instead of by real selfplay.
  
  # Differences with the real AlphaZero input format
  
  * In the real AlphaZero, the position of the pieces is given for each of the 8 previous board states. Chess is not fully described by the current board state, see eg. the [Fifty-move rule](https://en.wikipedia.org/wiki/Fifty-move_rule) or [Threefold repetition](https://en.wikipedia.org/wiki/Threefold_repetition). Adding 8 steps of history helps, but it's still not really enough.
  * AlphaZero doesn't need an en-passant plane since the network can just compare the previous two board states to see if en passant is possible.
  * The are some slight differences around player colour and move counter planes, but these are not very significant.
  
  All together this means AlphaZero had 8 \* 14 + 7 = 119 planes, whereas I only talked about 21 for most of this answer.
  
  > **Comments:**
  > **username_4**: Wonderful answer! Could you provide some sources? I'm very interested in learning more. In particular, where did the images come from?  
  > **username_3**: @BenMares Thanks! I've added a section about my sources and how I made the images.  
  > **username_5**: Is one of the depicted frames "move knight 2 forward and 1 right"? If so, it's interesting that AZ seems to have completely rejected 1. Nc3 as an opening move.  
  > **username_3**: @username_5 No that one got cut out, the last 9 planes are for underpromotions and the eight before that are for knight moves. Here's how much it likes all possible opening moves: https://pastebin.com/raw/yU9yDGUF. Keep in mind that this is not the real AZ network, I just trained it to imitate high-elo lichess games. The linked paper does contain some details about what openings the real AZ likes.  
  > **username_6**: @ToddSewell According to the AlphaZero paper, the input shape is 119x8x8, not 21x8x8.  What is your source for the change in input? The AlphaZero paper also doesn't mention the en passant square being included in the input. Is this a personal change you made?  
  > **username_3**: @username_6 You're right, I chose to simplify things somewhat (both for my implementation and this answer). I've added a note and some more details.  
  
  ---
  
  ### Answer by username_7 on 2021-11-24. Score: 50
  
  > was AlphaZero just given this move
  
  It was given this move in the same way that it was given all other moves. It started with complete knowledge of the rules of the game. In machine learning terms, castling would be part of its baseline knowledge of *state transitions*.
  
  
  ---
  
  ### Answer by username_8 on 2021-11-25. Score: 12
  
  The AlphaZero paper "[Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://arxiv.org/abs/1712.01815)" by David Silver et al. contains this explanation
  
  > Knowledge of the rules is also used to encode the input planes (i.e. castling, repetition,
  > no-progress) and output planes (how pieces move, promotions, and piece drops in shogi).
  
  (An input and output plane are the terms of art for how the board and possible moves are represented to the AlphaZero model.)
  
  
  ---
  
  ### Answer by username_9 on 2021-11-25. Score: 2
  
  > was AlphaZero just given this move, in the same way it was told that players take turns and a win is better than a loss?
  
  Actually you've hit upon 3 very distinct ways that AlphaZero was programmed. Castling is a potential move that's built into the algorithm; AlphaZero doesn't need to learn how to do it, just when to do it. The authors list the all the knowledge encoded into the algorithm on page 12 of their [paper](https://arxiv.org/pdf/1712.01815.pdf):
  
  > Domain Knowledge
  >
  > 1. The input features describing the position, and the output features describing the move, are structured as a set of planes; i.e. the
  >    neural network architecture is matched to the grid-structure of the
  >    board.
  > 2. AlphaZero is provided with perfect knowledge of the game rules. These are used during MCTS, to simulate the positions resulting from a
  >    sequence of moves, to determine game termination, and to score any
  >    simulations that reach a terminal state.
  > 3. Knowledge of the rules is also used to encode the input planes (i.e. castling, repetition, no-progress) and output planes (how pieces
  >    move, promotions, and piece drops in shogi).
  > 4. The typical number of legal moves is used to scale the exploration noise (see below).
  > 5. Chess and shogi games exceeding a maximum number of steps (determined by typical game length) were terminated and assigned a
  >    drawn outcome; Go games were terminated and scored with Tromp-Taylor
  >    rules, similarly to previous work (29).
  >
  > AlphaZero did not use any form of domain knowledge beyond the points listed above.
  
  In short, at every turn the neural network at the core of AlphaZero receives information about every piece on the board and outputs a number for every possible move allowed in chess. On top of that neural network there is an algorithm that controls the gameplay, and that zeroes out all currently impossible moves (like the neural net wanting to move a queen despite having no queen) then it picks the valid move with the highest number outputted by the neural net (sort of, the actual move is probabilistic). In this sense it was pre-programmed how to castle, and in fact at the beginning of training it has an equal chance of castling as it does moving a pawn (assuming the board state at that turn allows castling). During training the neural net will learn when castling leads to winning the game.
  
  The other answers already said as much, but I want to add to the other 2 things you mentioned. Taking turns is actually built into the algorithm that manages the game and it's such a central feature that the neural network was designed around it. It's actually quite hard to move away from turn based decision making when designing neural networks, and it's one of the features they address in their AlphaStar paper. AlphaStar plays Starcraft, a real time strategy based game where both players can make as many decisions as they want at any time. They actually don't really solve the continuous decision problem in AlphaStar, they just make the "turns" have a variable duration so the neural net can choose to make decisions every millisecond during intense battles or every 10 seconds if there's not much going on.
  
  Finally the win/loss thing is programmed into the optimizer side of AlphaZero's algorithm. The neural network at the core of AlphaZero is a mathematical function with lots of parameters, the value of the parameters determine what moves it thinks it's best. In essence you want to keep the parameters when it wins and you want to change parameters when it loses; but the exact way they make those changes is explained in the paper and it's quite ingenious. In any case it's actually quite easy to change the desired outcome (compared to the difficulty of moving away from turn based games). The authors desire the neural net to win at chess for obvious reasons, but they could very easily encode the desire to maximize pawn promotions, as an example.
  
  > **Comments:**
  > **username_10**: "In this sense it was pre-programmed how to castle, and in fact at the beginning of training it has an equal chance of castling as it does moving a pawn (assuming the board state at that turn allows castling)." I think a more precise statement would be "each castling move has the same probability as each pawn move." If there is one castling move and three pawn moves, I would expect the p(pawn move) =3 p(castling). If there are two castling moves and one pawn move, I would expect p(pawn move) = p(castling)/2.  
  > **username_11**: Todd Sewell's answer claims that machine doesn't get as an input the possible moves but just spits out a number of actions which are filtered for validity.  
  > **username_9**: @username_11 yes that's right. The input is the board state, and the neural network output structure is fixed and is meant to encode every possible move. The layer on top of the neural net filters what moves are valid.  
  
  ---
  
  ### Answer by username_12 on 2021-11-25. Score: 1
  
  I do not understand your point. It may be that you don't understand how engines work. AlphaZero is nothing special to be honest. Let me explain.
  
  Engines do three things while searching for a move.
  
  1. Find out the possible moves.
  2. Trim out searching for moves which seem useless(in AlphaZero's case, it also trims out any invalid moves)
  3. Evaluate the position.
  
  (The neural network can actually evaluate any position but invalid moves are ignored and it figures out that those moves are not worth searching; so the neural network is never explicitly told that some moves are invalid and it has to figure it out itself but the engine itself is informed; this doesn't actually change anything because eventually illegal moves are trimmed out, just at a later stage)
  
  The first two things that AlphaZero does is the same as any other engine. Castling, is a possible move, so the way that it is generated is the same as any other non neural network engine.
  
  The thing is that when evaluating a position, other engines used to use a fixed tuned set of values. They would calculate king safety, space, mobility, etc using hardcoded values which were generated by years of tuning values. AlphaZero, on the other hand, uses neural networks for evaluating positions. It's all the difference. AlphaZero is no alien.
  
  > **Comments:**
  > **username_13**: AlphaZero is different in the sense that the neural network is allowed to consider impossible moves as well.  
  > **username_12**: @username_13 Wouldn't that be just inefficient?  
  > **username_13**: Inefficient in comparison to what? They got pretty good results with their method, and it's not obvious how to do it "more efficiently".  
  > **username_12**: Inefficient compared to normal engines... Uh? You could just search for positions using a normal search tree instead of like searching every position in existence. Do I not understand how AlphaZero works?  
  > **username_13**: AlphaZero does use a search tree. The neural networks outputs probabilities for every move, and these probabilities are used to decide how to explore the tree. Illegal moves are ignored when searching the tree. But the neural network isn't really aware of that. Although it probably learns very fast that illegal moves are bad, just like it eventually learns that bad moves are bad.  
  > **username_12**: @username_13 What difference does that even make if the illegal moves are ignored by the search tree? Doesn't it mean the same thing. It's just a little quirk for how things are accomplished which doesn't affect the big picture. It's just like saying "Oooh an engine doesn't play a move, it just selects the best variation from the search tree". It's the same thing. What difference does it make if you prune the illegal moves during the generation of the moves or the searching. It means nothing.  
  > **username_13**: What difference between what and what? I don't understand what you're asking.  
  > **username_13**: Also note there is a difference between asking "how is the neural network used to explore the search tree during a game" and "how does the neural network learn".  
  > **username_12**: @username_13 I have never asked for any difference. Now the misunderstandings are flooding in and I feel it's best to stop now before this becomes a heated debate caused due to bad communication.  
  > **username_14**: This answer is wrong; Read the other answers for OP’s question and the paper for how AlphaZero works.  
  > **username_12**: @username_14 Where is it wrong? Oh yeah, I'll correct it but I don't think the mistake that I found is the same one you found.  
  > **username_15**: This is still wrong. AlphaZero uses a totally different search algorithm than traditional engines. PUCT and AB have major differences.  
  
  ---
  
  