Answers.Body: <p>Are you talking about static evaluations? No it’s not sufficient to play a good game. In chess, even a single pawn push might change the evaluation of the position. There are too many possibilities, so no engine will be successful without a search algorithm. What’s the point of a good pawn structure if your opponent can force a checkmate? A good model can guide a search algorithm better but it’s not a replacement.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T18:14:45.257Z
Answers.Text: No, I am trying to figure out if an engine that was trained against itself would eventually learn to look at future moves to more efficiently get its reward. And if so, would that be more valuable then using a hardcoded algorithm to dig through the future move tree>
Answers.UserDisplayName: OldAmmo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T03:29:42.897Z
Answers.LastActivityDate: 2023-01-08T03:29:42.897Z
Answers.UserDisplayName: SmallChess
Answers.Body: <p>Pure reinforcement learning with current neural network architectures for chess will, if done well, produce an opponent that is surprisingly strong; I can certainly see such a program beat master level players, especially if they are unaware of whom they are playing against. It will however fall far short of chess programs that incorporate some form of search. Typical machine learning algorithms (here: neural networks) are fundamentally limited by being algorithms that process a given input in a fixed time, i.e. they cannot by themselves take advantage of more time to think. Algorithms that combine good static evaluation with look-ahead, on the other hand, will get a lot better when allowed to do some search. Competitive programs therefore use search.</p>
  <p>That said, it is possible to try out in practice just how well a naked neural network can play, for instance by playing LC0 while requiring an instant response at each turn. To illustrate, I have just generated a game of LC0 under such conditions against the <em>hard</em> level of <em>Shredder Chess Online</em>; I would estimate that that opponent is roughly low candidate master level:</p>
  <pre><code>[FEN &quot;&quot;]
  [Event &quot;Test game&quot;]
  [Date &quot;2023.01.08&quot;]
  [Round &quot;&quot;]
  [White &quot;Player&quot;]
  [Black &quot;LC0&quot;]
  [Result &quot;1/2 - 1/2&quot;]
  
  1.e4 c5 2.c3 Nf6 3.e5 Nd5 4.Nf3 Nc6 5.d4 cxd4 6.Bc4 Nb6 7.Bb3 d5 8.exd6 
  Qxd6 9.O-O Be6 10.Na3 Bxb3 11.Qxb3 e6 12.Nb5 Qd7 13.Nxa7 Rxa7 14.Qxb6 dxc3
  15.Be3 Ra6 16.Qb3 cxb2 17.Qxb2 f6 18.Rfd1 Qc8 19.Rd3 Be7 20.Rb3 Na5 21.Rc1
  Qd7 22.Rbc3 O-O 23.Rc7 Qd8 24.Qb5 Ba3 25.R1c3 Qd1+ 26.Qf1 Qxf1+ 27.Kxf1 
  Bd6 28.Rd7 Rf7 29.Rc8+ Rf8 30.Rc1 Rf7 31.Rc8+ Rf8 32.Rc1 Rf7 33.Rc8+ 1/2-1/2
  </code></pre>
  <p>The Maiabots on lichess are, as far as I know, another example of pure policy network opponents that are publicly available. They do, however, play significantly weaker than the LC0 policy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T23:31:53.757Z
Answers.Text: Would the downvoter explain why they downvoted?
Answers.UserDisplayName: Polytropos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T18:12:51.497Z
Answers.Text: interesting, I guess my question then is, would it be possible for a model like this to weigh future moves with a high depth without having another algorithm dig through the tree of future boards to find the best moves? for example, what if you just took a model like a maiabot and train it against the highest level stockfish,  In order to beat stockfish and get the reward the model would have to look into the future list of moves and the only way to do that would be for it to come up with its own algorithm to do that. could the bot then be trained against itself to be better then stockfish?
Answers.UserDisplayName: OldAmmo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T22:33:32.277Z
Answers.Text: Training against the highest level of Stockfish will just fail, because you never get a reward (at least with the naive reward model). But independently of how you train, I would guess that your network will learn to spot common tactics and how to avoid tactical problems by making some prophylactic moves, but fail to see a lot of uncommon tactics. Constant time algorithms (such as NN) cannot fully replace search in chess. You'll fall far short of the playing strength of Stockfish, considering that even full LC0 with search is just about even with Stockfish on top hardware.
Answers.UserDisplayName: Polytropos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T04:04:54.583Z
Answers.LastActivityDate: 2023-01-08T04:50:27.473Z
Answers.LastEditDate: 2023-01-08T04:50:27.473Z
Answers.UserDisplayName: Polytropos
Body: <p>I have recently been studying up on Machine Learning based chess engines and have begun to develop one of my own. I was wondering, realistically, doesn't the board technically contain all of the data used to predict any number of moves into the future? From what I can tell, models like AlphaGo use a search algorithm to pick which moves it should investigate further in order to make its move.</p>
  <p>what if a RL model were to just be trained on the current position, not any other data besides what anyone could see from looking at the board. through training, would the model develop its own method of searching for future moves in order to maximize its reward potential in a game naturally?</p>
  <p>I assume that if the model developed its own algo it would be significantly faster and more optimized at returning good moves with a depth level that would make it competent at the game.</p>
  <p>Is there something I am missing?</p>
  
Tags: <engines><learning><alphazero><chess-algorithms><chess-ai>
Title: Does a reinforcement learning style model actually need to be combined with a search algo to produce the best moves?
ThreadText: # Does a reinforcement learning style model actually need to be combined with a search algo to produce the best moves?
  
  (Asked by: username_0 on 2023-01-06)
  
  I have recently been studying up on Machine Learning based chess engines and have begun to develop one of my own. I was wondering, realistically, doesn't the board technically contain all of the data used to predict any number of moves into the future? From what I can tell, models like AlphaGo use a search algorithm to pick which moves it should investigate further in order to make its move.
  
  what if a RL model were to just be trained on the current position, not any other data besides what anyone could see from looking at the board. through training, would the model develop its own method of searching for future moves in order to maximize its reward potential in a game naturally?
  
  I assume that if the model developed its own algo it would be significantly faster and more optimized at returning good moves with a depth level that would make it competent at the game.
  
  Is there something I am missing?
  
  **Tags:** engines, learning, alphazero, chess-algorithms, chess-ai
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-01-08. Score: 1
  
  Are you talking about static evaluations? No it’s not sufficient to play a good game. In chess, even a single pawn push might change the evaluation of the position. There are too many possibilities, so no engine will be successful without a search algorithm. What’s the point of a good pawn structure if your opponent can force a checkmate? A good model can guide a search algorithm better but it’s not a replacement.
  
  > **Comments:**
  > **username_0**: No, I am trying to figure out if an engine that was trained against itself would eventually learn to look at future moves to more efficiently get its reward. And if so, would that be more valuable then using a hardcoded algorithm to dig through the future move tree>  
  
  ---
  
  ### Answer by username_2 on 2023-01-08. Score: 1
  
  Pure reinforcement learning with current neural network architectures for chess will, if done well, produce an opponent that is surprisingly strong; I can certainly see such a program beat master level players, especially if they are unaware of whom they are playing against. It will however fall far short of chess programs that incorporate some form of search. Typical machine learning algorithms (here: neural networks) are fundamentally limited by being algorithms that process a given input in a fixed time, i.e. they cannot by themselves take advantage of more time to think. Algorithms that combine good static evaluation with look-ahead, on the other hand, will get a lot better when allowed to do some search. Competitive programs therefore use search.
  That said, it is possible to try out in practice just how well a naked neural network can play, for instance by playing LC0 while requiring an instant response at each turn. To illustrate, I have just generated a game of LC0 under such conditions against the hard level of Shredder Chess Online; I would estimate that that opponent is roughly low candidate master level:
  [FEN ""]
  [Event "Test game"]
  [Date "2023.01.08"]
  [Round ""]
  [White "Player"]
  [Black "LC0"]
  [Result "1/2 - 1/2"]
  
  1.e4 c5 2.c3 Nf6 3.e5 Nd5 4.Nf3 Nc6 5.d4 cxd4 6.Bc4 Nb6 7.Bb3 d5 8.exd6 
  Qxd6 9.O-O Be6 10.Na3 Bxb3 11.Qxb3 e6 12.Nb5 Qd7 13.Nxa7 Rxa7 14.Qxb6 dxc3
  15.Be3 Ra6 16.Qb3 cxb2 17.Qxb2 f6 18.Rfd1 Qc8 19.Rd3 Be7 20.Rb3 Na5 21.Rc1
  Qd7 22.Rbc3 O-O 23.Rc7 Qd8 24.Qb5 Ba3 25.R1c3 Qd1+ 26.Qf1 Qxf1+ 27.Kxf1 
  Bd6 28.Rd7 Rf7 29.Rc8+ Rf8 30.Rc1 Rf7 31.Rc8+ Rf8 32.Rc1 Rf7 33.Rc8+ 1/2-1/2
  
  The Maiabots on lichess are, as far as I know, another example of pure policy network opponents that are publicly available. They do, however, play significantly weaker than the LC0 policy.
  
  > **Comments:**
  > **username_2**: Would the downvoter explain why they downvoted?  
  > **username_0**: interesting, I guess my question then is, would it be possible for a model like this to weigh future moves with a high depth without having another algorithm dig through the tree of future boards to find the best moves? for example, what if you just took a model like a maiabot and train it against the highest level stockfish,  In order to beat stockfish and get the reward the model would have to look into the future list of moves and the only way to do that would be for it to come up with its own algorithm to do that. could the bot then be trained against itself to be better then stockfish?  
  > **username_2**: Training against the highest level of Stockfish will just fail, because you never get a reward (at least with the naive reward model). But independently of how you train, I would guess that your network will learn to spot common tactics and how to avoid tactical problems by making some prophylactic moves, but fail to see a lot of uncommon tactics. Constant time algorithms (such as NN) cannot fully replace search in chess. You'll fall far short of the playing strength of Stockfish, considering that even full LC0 with search is just about even with Stockfish on top hardware.  
  
  ---
  
  