Answers.Body: <p>Minimax and related algorithms <em>are</em> used to play chess. That is how chess programs have worked for many years (with some additions such as standard opening playbooks). They do not need to process the whole game tree. There are a few different techniques used to reduce the effective search space. One of the most the most impactful is to truncate search after a certain depth if it has not resolved the game, and to <em>score</em> the position reached using some form of heuristic function that takes the board state as input and outputs a number. The impact from this has two parts:</p>
  <ul>
  <li><p>The search space is reduced by a large factor.</p>
  </li>
  <li><p>The strength of the resulting automated player is limited by how effective the heuristic is at reflecting the strength of the game position. A perfect heuristic could allow a very limited search (just one ply look ahead, or up to some point within the game) to play perfectly. However, it is a practical impossibility for many games - if a game is &quot;unsolved&quot; that means by extension that there is no known perfect heuristic.</p>
  </li>
  </ul>
  <p>There are other ways to reduce search space or search it more efficiently, but this search depth plus heuristic pattern is very common in turn-based games. This is how IBM's Deep Blue worked for example, and it did not use neural networks. The heuristic function in Deep Blue was constructed by experts extending already well-studied basics of chess playing, such as assigning simple points value to each side's remaining pieces (e.g. a rook is worth more than a bishop).</p>
  <blockquote>
  <p><strong>What exactly would the Neural Network use as a Loss Function?</strong></p>
  </blockquote>
  <p>There are two main ways that a neural network can be applied in a game like chess:</p>
  <h3>1. To predict the chances of winning from a given position</h3>
  <p>(or equivalently score a heuristic for the position)</p>
  <p>In reinforcement learning terms, this would be the value function for that state (game position).</p>
  <p>This can be used alongside tree searches, the main difference being that instead of an expert <em>designing</em> the heuristic, the neural network will have learned one from observing games. Those games could be a database of games played by humans, or generated by a learning agent playing against itself.</p>
  <p>The loss function that drives learning here would typically be a cross-entropy loss comparing the eventual win/loss with the predicted probability. This is noisy of course, because the same position may lead to win, draw or loss depending on relative strengths of the players. However, as the agent player becomes stronger, the prediction will start to match it, and more closely approximate the probabilities from optimal play.</p>
  <h3>2. To predict the best move from a given position</h3>
  <p>This is also called a policy function.</p>
  <p>This is a multi-class prediction problem, that would use the normal multiclass log loss function. Similar to learning the value function, it can be learned from observation assuming all action choices that lead to a win are the best choices. Again this is noisy, but should converge on an approximation of optimal play provided the input is from master-level play database, or is from a working self-play system.</p>
  <blockquote>
  <p>I don't see how the Loss Function in this case is continuous, and thus how could Gradient Descent be used on such a Loss Function?</p>
  </blockquote>
  <p>Once you move away from trying to have the neural networks process a game tree directly, the problem becomes clearer. You need to run the game tree processing using suitable tree search algorithm (minimax, negamax, MCTS etc), and have neural networks for the value and/or policy function estimations at each node, which they can learn from any relevant data.</p>
  <p>If you want to create self-learning agents, as opposed to training them from a pre-existing database of games, then you will need to study systems that can generate suitable data from experiencing game play directly. This adds another outer layer of logic alongside the tree search algorithm that helps decide what data to collect and how to score it so it can be used as training data for the neural networks.</p>
  <p>There is quite a lot you could study about these kinds of learning systems. I recommend for the basics that you look into reinforcement learning (RL), and the go-to book for many people there is Sutton &amp; Barto's <a href="http://incompleteideas.net/book/the-book.html" rel="noreferrer">Reinforcement Learning: An Introduction</a>, which lays out the theory and terminology used in RL and a variety of basic learning algorithms. It would be a diversion from building a game-playing program to study it all, and it does not delve into all elements of game-playing agents in great detail, but it is IMO a very good grounding into the <em>learning</em> parts of any such agent.</p>
  <p>In addition, one state-of-the-art self-play learning algorithm is Deep Mind's Alpha Zero. It might be a bit of a steep learning curve to dive straight into how it works, but the algorithm is at its core surprisingly simple and elegant. It may be possible, given some basic knoweldge of terms from RL (so you know what a policy is), to get straight into it based on some tutorials like <a href="https://web.stanford.edu/%7Esurag/posts/alphazero.html" rel="noreferrer">this one that learns Othello</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-18T20:52:02.153Z
Answers.Text: Thank you for this answer! I will take some time and try to digest it!
Answers.UserDisplayName: stats_noob
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-18T16:44:44.237Z
Answers.LastActivityDate: 2022-03-18T16:51:39.427Z
Answers.LastEditDate: 2022-03-18T16:51:39.427Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>This is a good question. your understanding in general is correct. Indeed, data can be used to construct a proper evaluation of a move/board position and recommended moves based on its history (at least alphago does).</p>
  <p>Regarding your first point, it is possible that scenarios that never occurred in historical data could occur, but that's not a problem if your valuation procedure is strong. Typically this is modeled by some continuous function that takes your input board and gives some scalar value. Doing so implicitly assumes you learn the patterns among data pretty well so that ideally you can extrapolate sufficiently in new scenarios. If you had used a lookup table for every board state, you would be screwed in both of the respects you mentioned above.</p>
  <p>Regarding your second point, problems related to storing the tree itself and efficiently querying has other solutions that essentially involve a clever pruning method (minimax, and its variants) and/or an efficient cache, like zobrist hashing (see chessprogramming wiki for more techniques).</p>
  <p>Now to the core of the question, how do you evaluate and setup a loss function?
  There are several ways to do this though: roughly you can think about this as falling into the following buckets</p>
  <ul>
  <li><p>(i) a reinforcement learning approach, which   Alphago uses (<a href="https://www.chess.com/article/view/whats-inside-alphazeros-brain" rel="noreferrer">https://www.chess.com/article/view/whats-inside-alphazeros-brain</a>) . This is interesting because RL is quite a different paradigm of learning than supervised learning. I'd recommend reading through some of sutton/barto (<a href="http://www.incompleteideas.net/book/the-book-2nd.html" rel="noreferrer">http://www.incompleteideas.net/book/the-book-2nd.html</a>) to see how questions of policy learning and move recommendations can be derived in general; this is quite a readable book. Then read about alphazero and alphago <a href="https://www.science.org/doi/10.1126/science.aar6404" rel="noreferrer">https://www.science.org/doi/10.1126/science.aar6404</a>.</p>
  </li>
  <li><p>(ii) supervised learning.
  When I initially tried this out, I trained a neural net to output a value at least as good as the moves outputted by a grandmaster (see <a href="https://ai.stackexchange.com/questions/4725/should-i-use-neural-networks-or-genetic-algorithms-to-solve-gomoku/4727#4727">Should I use neural networks or genetic algorithms to solve Gomoku?</a> for more details). Another way can be based on whether the player won or lost at the end, assuming that in general their moves were better when they won than when they lost (a little strong of an assumption to me personally). See <a href="https://www.cs.tau.ac.il/%7Ewolf/papers/deepchess.pdf" rel="noreferrer">https://www.cs.tau.ac.il/~wolf/papers/deepchess.pdf</a> - They randomly sample moves from a game white won and lost, and learns to choose between the moves with a neural network.</p>
  </li>
  </ul>
  <p>To be explicit, once you setup the appropriate loss or policy optimization problem, you can perform gradient descent as usual. what you are optimizing would of course differ based on if you choose the RL route or the supervised learning route.</p>
  <p>Since this is in the first part of your question, I will additionally say that NNs help in chess because of this complex problem of how to valuate a position based on some unique progression and state of the game, and the combinatorial explosion (which is somewhat rectified by a move recommender in alphago, and/or intuitively to me, better evaluating positions may allow for more robust move selection given less plies). Without a NN, some more static evaluator or less complex function approximator may be too simplistic for robustly learning `good moves' in chess.</p>
  <p>It is interesting to see that some methods do not require you to actually encode which moves are legal and which are not.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-18T16:43:27.737Z
Answers.LastActivityDate: 2022-03-18T17:08:29.220Z
Answers.LastEditDate: 2022-03-18T17:08:29.220Z
Answers.UserDisplayName: sma
Body: <p>I have been spending a few days trying to wrap my head around how and why neural networks are used to play chess.</p>
  <p>Although I know very little about how the game of chess works, I can understand the following idea. Theoretically, we could make a &quot;tree&quot; that included every possible outcome of a chess game. Through knowledge provided by chess experts, we could identify how &quot;favorable&quot; certain parts of this tree are compared to other parts of the tree. We could also use this tree to &quot;rank&quot; optimal chess moves based on how the chess board appears in the current turn (e.g. which pieces you and your opponent have left and where these pieces are situated).</p>
  <p>The problem is, this tree would be so enormous that it would be impossible to create, store and &quot;search&quot; (e.g. with the MinMax algorithm):</p>
  <p><a href="https://i.sstatic.net/IXbIc.png" rel="noreferrer"><img src="https://i.sstatic.net/IXbIc.png" alt="enter image description here" /></a></p>
  <p>I understand that perhaps this tree can be created using data to limit the size of the tree based on scenarios that are more likely to appear compared to all possible scenarios. For example, if a player wanted they could spend the whole game aimlessly shifting their &quot;Rook&quot; back and forth - theoretically, this outcome could occur but no player (in their sane mind) would ever do this. Thus, the tree could be constructed using actual data from millions of chess games. This for example could tell us : Based on historical data and given the current setup of the chess board, 21% of games were won when the immediate next move involved moving the Queen to &quot;F5&quot; vs only 3% of games were won when the immediate next move involved moving the Knight to &quot;F5&quot;. I suppose at each move, the data based tree could be queried to rank the optimality of each next move by checking the proportion of &quot;terminal nodes&quot; that resulted in wins for each possible move given the current chess board.</p>
  <p>However, I still see 2 problems with this approach:</p>
  <ul>
  <li><p>It is possible that we might run into a scenario(s) that never occurred within the historical data, rendering the tree useless in this scenario</p>
  </li>
  <li><p>This tree still might be too large to efficiently store and query.</p>
  </li>
  </ul>
  <p>This is probably why neural networks are being used to play chess - I tried to do some readings about this topic, but I can't seem to fully understand it. In this case, <strong>what exactly would the neural network use as a loss function? I don't see how the loss function in this case is continuous, and thus how could gradient descent be used on such a loss function?</strong></p>
  <p>Could someone please recommend some sources (e.g. YouTube Videos, Blogs, etc.) that show how a neural network can be used to play chess.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-18T23:10:00.270Z
Comments.Text: You're raising multiple issues here. First, you're asking what the loss function would be. Another issue that you raise is: what to do if we have no data for a particular situation. I understand that the overall issue is how can neural networks be used in the context of chess, but you should narrow the scope of your posts to just one **specific question**. Is your main question "What the loss function would be in this case?" or is the question in the title (which is broad)? You don't describe what the input and output of the neural network would be, so it's not clear if you know that or not.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-19T02:10:16.593Z
Comments.Text: I think its also worth pointing to the [chess programming wiki](https://www.chessprogramming.org/Main_Page) which covers the basics (and not so basics) of chess engines. Very useful resource if you're planning on developing your own engine be it using a hand-crafted or NN-based evaluation function. Specifically [NNUE](https://www.chessprogramming.org/NNUE) may be what you're looking for.
Comments.UserDisplayName: jodag
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-19T07:36:10.200Z
Comments.Text: For the large game tree analysis, you could analyze it with combinatorial game theory—no need to get into statistics and historical data. As you say though, it's a moot point since the game tree would be unfathomably big.
Comments.UserDisplayName: The Guy with The Hat
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-19T13:16:45.127Z
Comments.Text: "It is possible that we might run into a scenario(s) that never occurred within the historical data, rendering the tree useless in this scenario" - the ability to generalise to unseen data is one of the main reasons AI is used in the first place. One could easily generate the tree dynamically (because it's trivial to generate all the moves you can make from a specific board state), with each board state / node in the tree (at some depth) being evaluated by the neural network to generate a score used by minimax. So unseen data shouldn't be a problem at all (if you've seen *similar enough* data)
Comments.UserDisplayName: NotThatGuy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-02T16:15:14.717Z
Comments.Text: Just a comment on your tree. It actually exists when there are 6 or maybe by now even 7 pieces left. And due to this tree they found out that some chess rules are actually wrong. There is a 50 move rule to prevent games from going for ever. If no piece has been taken and no pawn advanced in 50 moves, the game is considered to be a draw because no progress can be made to win the game. But due to the tree they found out, that in certain situations there can be a combination of 200 moves without taking a piece and advancing a pawn that leads to a win. So the rule was based on a bad guess.
Comments.UserDisplayName: Jefe infiltrado
Tags: <neural-networks><reference-request><chess><board-games>
Title: How do neural networks play chess?
UserDisplayName: stats_noob
ThreadText: # How do neural networks play chess?
  
  (Asked by: username_0 on 2022-03-18)
  
  I have been spending a few days trying to wrap my head around how and why neural networks are used to play chess.
  
  Although I know very little about how the game of chess works, I can understand the following idea. Theoretically, we could make a "tree" that included every possible outcome of a chess game. Through knowledge provided by chess experts, we could identify how "favorable" certain parts of this tree are compared to other parts of the tree. We could also use this tree to "rank" optimal chess moves based on how the chess board appears in the current turn (e.g. which pieces you and your opponent have left and where these pieces are situated).
  
  The problem is, this tree would be so enormous that it would be impossible to create, store and "search" (e.g. with the MinMax algorithm):
  
  [![enter image description here](https://i.sstatic.net/IXbIc.png)](https://i.sstatic.net/IXbIc.png)
  
  I understand that perhaps this tree can be created using data to limit the size of the tree based on scenarios that are more likely to appear compared to all possible scenarios. For example, if a player wanted they could spend the whole game aimlessly shifting their "Rook" back and forth - theoretically, this outcome could occur but no player (in their sane mind) would ever do this. Thus, the tree could be constructed using actual data from millions of chess games. This for example could tell us : Based on historical data and given the current setup of the chess board, 21% of games were won when the immediate next move involved moving the Queen to "F5" vs only 3% of games were won when the immediate next move involved moving the Knight to "F5". I suppose at each move, the data based tree could be queried to rank the optimality of each next move by checking the proportion of "terminal nodes" that resulted in wins for each possible move given the current chess board.
  
  However, I still see 2 problems with this approach:
  
  * It is possible that we might run into a scenario(s) that never occurred within the historical data, rendering the tree useless in this scenario
  * This tree still might be too large to efficiently store and query.
  
  This is probably why neural networks are being used to play chess - I tried to do some readings about this topic, but I can't seem to fully understand it. In this case, **what exactly would the neural network use as a loss function? I don't see how the loss function in this case is continuous, and thus how could gradient descent be used on such a loss function?**
  
  Could someone please recommend some sources (e.g. YouTube Videos, Blogs, etc.) that show how a neural network can be used to play chess.
  
  **Tags:** neural-networks, reference-request, chess, board-games
  
  > **Comments:**
  > **username_1**: You're raising multiple issues here. First, you're asking what the loss function would be. Another issue that you raise is: what to do if we have no data for a particular situation. I understand that the overall issue is how can neural networks be used in the context of chess, but you should narrow the scope of your posts to just one **specific question**. Is your main question "What the loss function would be in this case?" or is the question in the title (which is broad)? You don't describe what the input and output of the neural network would be, so it's not clear if you know that or not.  
  > **username_2**: I think its also worth pointing to the [chess programming wiki](https://www.chessprogramming.org/Main_Page) which covers the basics (and not so basics) of chess engines. Very useful resource if you're planning on developing your own engine be it using a hand-crafted or NN-based evaluation function. Specifically [NNUE](https://www.chessprogramming.org/NNUE) may be what you're looking for.  
  > **username_3**: For the large game tree analysis, you could analyze it with combinatorial game theory—no need to get into statistics and historical data. As you say though, it's a moot point since the game tree would be unfathomably big.  
  > **username_4**: "It is possible that we might run into a scenario(s) that never occurred within the historical data, rendering the tree useless in this scenario" - the ability to generalise to unseen data is one of the main reasons AI is used in the first place. One could easily generate the tree dynamically (because it's trivial to generate all the moves you can make from a specific board state), with each board state / node in the tree (at some depth) being evaluated by the neural network to generate a score used by minimax. So unseen data shouldn't be a problem at all (if you've seen *similar enough* data)  
  > **username_5**: Just a comment on your tree. It actually exists when there are 6 or maybe by now even 7 pieces left. And due to this tree they found out that some chess rules are actually wrong. There is a 50 move rule to prevent games from going for ever. If no piece has been taken and no pawn advanced in 50 moves, the game is considered to be a draw because no progress can be made to win the game. But due to the tree they found out, that in certain situations there can be a combination of 200 moves without taking a piece and advancing a pawn that leads to a win. So the rule was based on a bad guess.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_6 on 2022-03-18. Score: 20
  
  Minimax and related algorithms *are* used to play chess. That is how chess programs have worked for many years (with some additions such as standard opening playbooks). They do not need to process the whole game tree. There are a few different techniques used to reduce the effective search space. One of the most the most impactful is to truncate search after a certain depth if it has not resolved the game, and to *score* the position reached using some form of heuristic function that takes the board state as input and outputs a number. The impact from this has two parts:
  
  * The search space is reduced by a large factor.
  * The strength of the resulting automated player is limited by how effective the heuristic is at reflecting the strength of the game position. A perfect heuristic could allow a very limited search (just one ply look ahead, or up to some point within the game) to play perfectly. However, it is a practical impossibility for many games - if a game is "unsolved" that means by extension that there is no known perfect heuristic.
  
  There are other ways to reduce search space or search it more efficiently, but this search depth plus heuristic pattern is very common in turn-based games. This is how IBM's Deep Blue worked for example, and it did not use neural networks. The heuristic function in Deep Blue was constructed by experts extending already well-studied basics of chess playing, such as assigning simple points value to each side's remaining pieces (e.g. a rook is worth more than a bishop).
  
  > **What exactly would the Neural Network use as a Loss Function?**
  
  There are two main ways that a neural network can be applied in a game like chess:
  
  ### 1. To predict the chances of winning from a given position
  
  (or equivalently score a heuristic for the position)
  
  In reinforcement learning terms, this would be the value function for that state (game position).
  
  This can be used alongside tree searches, the main difference being that instead of an expert *designing* the heuristic, the neural network will have learned one from observing games. Those games could be a database of games played by humans, or generated by a learning agent playing against itself.
  
  The loss function that drives learning here would typically be a cross-entropy loss comparing the eventual win/loss with the predicted probability. This is noisy of course, because the same position may lead to win, draw or loss depending on relative strengths of the players. However, as the agent player becomes stronger, the prediction will start to match it, and more closely approximate the probabilities from optimal play.
  
  ### 2. To predict the best move from a given position
  
  This is also called a policy function.
  
  This is a multi-class prediction problem, that would use the normal multiclass log loss function. Similar to learning the value function, it can be learned from observation assuming all action choices that lead to a win are the best choices. Again this is noisy, but should converge on an approximation of optimal play provided the input is from master-level play database, or is from a working self-play system.
  
  > I don't see how the Loss Function in this case is continuous, and thus how could Gradient Descent be used on such a Loss Function?
  
  Once you move away from trying to have the neural networks process a game tree directly, the problem becomes clearer. You need to run the game tree processing using suitable tree search algorithm (minimax, negamax, MCTS etc), and have neural networks for the value and/or policy function estimations at each node, which they can learn from any relevant data.
  
  If you want to create self-learning agents, as opposed to training them from a pre-existing database of games, then you will need to study systems that can generate suitable data from experiencing game play directly. This adds another outer layer of logic alongside the tree search algorithm that helps decide what data to collect and how to score it so it can be used as training data for the neural networks.
  
  There is quite a lot you could study about these kinds of learning systems. I recommend for the basics that you look into reinforcement learning (RL), and the go-to book for many people there is Sutton & Barto's [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/the-book.html), which lays out the theory and terminology used in RL and a variety of basic learning algorithms. It would be a diversion from building a game-playing program to study it all, and it does not delve into all elements of game-playing agents in great detail, but it is IMO a very good grounding into the *learning* parts of any such agent.
  
  In addition, one state-of-the-art self-play learning algorithm is Deep Mind's Alpha Zero. It might be a bit of a steep learning curve to dive straight into how it works, but the algorithm is at its core surprisingly simple and elegant. It may be possible, given some basic knoweldge of terms from RL (so you know what a policy is), to get straight into it based on some tutorials like [this one that learns Othello](https://web.stanford.edu/%7Esurag/posts/alphazero.html).
  
  > **Comments:**
  > **username_0**: Thank you for this answer! I will take some time and try to digest it!  
  
  ---
  
  ### Answer by username_7 on 2022-03-18. Score: 12
  
  This is a good question. your understanding in general is correct. Indeed, data can be used to construct a proper evaluation of a move/board position and recommended moves based on its history (at least alphago does).
  
  Regarding your first point, it is possible that scenarios that never occurred in historical data could occur, but that's not a problem if your valuation procedure is strong. Typically this is modeled by some continuous function that takes your input board and gives some scalar value. Doing so implicitly assumes you learn the patterns among data pretty well so that ideally you can extrapolate sufficiently in new scenarios. If you had used a lookup table for every board state, you would be screwed in both of the respects you mentioned above.
  
  Regarding your second point, problems related to storing the tree itself and efficiently querying has other solutions that essentially involve a clever pruning method (minimax, and its variants) and/or an efficient cache, like zobrist hashing (see chessprogramming wiki for more techniques).
  
  Now to the core of the question, how do you evaluate and setup a loss function?
  There are several ways to do this though: roughly you can think about this as falling into the following buckets
  
  * (i) a reinforcement learning approach, which Alphago uses (<https://www.chess.com/article/view/whats-inside-alphazeros-brain>) . This is interesting because RL is quite a different paradigm of learning than supervised learning. I'd recommend reading through some of sutton/barto (<http://www.incompleteideas.net/book/the-book-2nd.html>) to see how questions of policy learning and move recommendations can be derived in general; this is quite a readable book. Then read about alphazero and alphago <https://www.science.org/doi/10.1126/science.aar6404>.
  * (ii) supervised learning.
    When I initially tried this out, I trained a neural net to output a value at least as good as the moves outputted by a grandmaster (see [Should I use neural networks or genetic algorithms to solve Gomoku?](https://ai.stackexchange.com/questions/4725/should-i-use-neural-networks-or-genetic-algorithms-to-solve-gomoku/4727#4727) for more details). Another way can be based on whether the player won or lost at the end, assuming that in general their moves were better when they won than when they lost (a little strong of an assumption to me personally). See [https://www.cs.tau.ac.il/~wolf/papers/deepchess.pdf](https://www.cs.tau.ac.il/%7Ewolf/papers/deepchess.pdf) - They randomly sample moves from a game white won and lost, and learns to choose between the moves with a neural network.
  
  To be explicit, once you setup the appropriate loss or policy optimization problem, you can perform gradient descent as usual. what you are optimizing would of course differ based on if you choose the RL route or the supervised learning route.
  
  Since this is in the first part of your question, I will additionally say that NNs help in chess because of this complex problem of how to valuate a position based on some unique progression and state of the game, and the combinatorial explosion (which is somewhat rectified by a move recommender in alphago, and/or intuitively to me, better evaluating positions may allow for more robust move selection given less plies). Without a NN, some more static evaluator or less complex function approximator may be too simplistic for robustly learning `good moves' in chess.
  
  It is interesting to see that some methods do not require you to actually encode which moves are legal and which are not.
  
  
  ---
  
  