Answers.Body: <h1>First intuition</h1>
  <p>I wrote a quick script to test the outcome distribution of random chess games: I get about 7% wins for black and white each, so 14% non-drawn games. The MCTS should still be able to find simple mate in one moves even with a fully uninitialized network, so I would expect it to draw even less.</p>
  <p>Your 97% draw rate is significantly higher than that, so I suspect something else is going wrong. Some ideas to try:</p>
  <ul>
  <li>Do your own random moves test to confirm that you also get 7% win for each player, otherwise something went wrong with your board implementation.</li>
  <li>Run MCTS with a random NN on a mate-in-one position, and ensure that it finds the mate and picks it as the best (highest visit count) move after a couple thousand visits.</li>
  </ul>
  <p>I think in theory 3% decisive games is already enough to kickstart the learning process, but specifically for chess it probably means something in the implementation is wrong.</p>
  <h1>Actual data</h1>
  <p>Here's the win/draw/loss plot from a chess run of my own AlphaZero implementation, <a href="https://github.com/KarelPeeters/kZero" rel="nofollow noreferrer">kZero</a>. The horizontal axis corresponds to about 300k selfplay games played. This is with 400 visits/move, and settings otherwise mostly similar to the AlphaZero paper.</p>
  <p>It looks like the behavior is as follows:</p>
  <ul>
  <li>The initial games generated with a random NN have only 17% draws, so MCTS is finding a lot of mates</li>
  <li>Very quickly the drawrate goes up to 35%, as the NN learns to avoid mate in one</li>
  <li>After that the drawrate drops back down to 8%, as the NN learns to finish games from winning positions.</li>
  <li>For the rest of the run the drawrate slowly climbs again, as is typical for stronger chess players and engines.</li>
  </ul>
  <p><a href="https://i.sstatic.net/vVsP6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vVsP6.png" alt="WDL plot" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-24T19:41:42.250Z
Answers.Text: Thanks a lot for your quick and comprehensive answer!
Answers.UserDisplayName: Oliver Ewald
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-24T19:57:54.950Z
Answers.Text: 1. I will do new random checks (without MCTS) to see if my win ratios will go up towards your 14%
  2. I just checked that my MCTS clearly detects "mate in one" 
  3. What I find surprising is that your MCTS on a fully uninitialized NN only finds 17% draws. That's far off from my 97%
  4. Currently I define a game as "drawn" if there is no mate after 100 moves. What rules do you apply to avoid endless games?
  5. I will answer back here once I have 1. completed with (possibly) adjusted rules how to define draws.
Answers.UserDisplayName: Oliver Ewald
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-25T22:23:43.577Z
Answers.Text: @OliverEwald I only terminate games based on 3-fold repetition, the 50 move rule and insufficient material (only king vs king). For this run the average game length started at 100, peaked at 300 and then dropped back down to 110. Cutting of games at 100 moves indeed causes lots of draws, so that's probably your main issue!
Answers.UserDisplayName: KarelPeeters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T08:09:29.047Z
Answers.Text: Thanks a lot for clarification! I will test with modified termination rules.
Answers.UserDisplayName: Oliver Ewald
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-23T21:17:20.940Z
Answers.LastActivityDate: 2024-01-23T21:17:20.940Z
Answers.UserDisplayName: KarelPeeters
Body: <p>I am experimenting with a self-programmed version of AZ Chess following the described methodology in the official paper. I am experiencing that at the beginning of the self-play (when the ANN weights are is still randomly initialized) a very high portion of all games end with a &quot;draw&quot; (ca. 97% of all games). This is not surprising because if two pure random players play against each other, it is very unlikely that white or black achieve a win/lose. As a result, ca. 97% of all board states visited during the sell-play are trained towards a target value of 0 (instead towards +1/-1 for white to win/lose). This means that there is very litte training data (3% of all training data generated via self-play) that helps the ANN to learn how to win and to get stronger.</p>
  <p>Does anyone know how DeepMind has overcome this problem? I have found nothing about this in the www.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-23T16:22:52.893Z
Comments.Text: Hello and welcome to AI Stack Exchange! How many games have you currently run during training?
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-23T16:41:58.757Z
Comments.Text: Also, are you using a MCTS?
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-23T21:04:39.270Z
Comments.Text: I have already run ca. 1000 self-play games. But that is enough to see that the vast majority end with a "draw" which does not generate any training effect.
Comments.UserDisplayName: Oliver Ewald
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-23T21:13:35.443Z
Comments.Text: Yes, I general I use MCTS during self-play, but at the beginning I set "number of roll-outs" to just 1 in order to speed up the process (that led to the 97% draws). But during the last 24 hours I set "number of roll-outs" to 800 and observed a decrease of "draws" (down from 97% to 80%). But the 80% still seem far suboptimal for effective learning. I am wondering if it was really the huge number of self-play games that solved the problem in the original version of AlphaZero, or do I still miss something conceptionally-wise?
Comments.UserDisplayName: Oliver Ewald
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-25T22:24:14.797Z
Comments.Text: @OliverEwald AlphaZero training with only a single visit is a bit strange, how would the policy head ever learning anything? You definitely need at least _some_ rollouts!
Comments.UserDisplayName: KarelPeeters
Tags: <reinforcement-learning><alphazero><chess><board-games>
Title: AlphaZero Chess: High portion of "draws" during first rounds of self-play hampers learning
UserDisplayName: Oliver Ewald
ThreadText: # AlphaZero Chess: High portion of "draws" during first rounds of self-play hampers learning
  
  (Asked by: username_0 on 2024-01-22)
  
  I am experimenting with a self-programmed version of AZ Chess following the described methodology in the official paper. I am experiencing that at the beginning of the self-play (when the ANN weights are is still randomly initialized) a very high portion of all games end with a "draw" (ca. 97% of all games). This is not surprising because if two pure random players play against each other, it is very unlikely that white or black achieve a win/lose. As a result, ca. 97% of all board states visited during the sell-play are trained towards a target value of 0 (instead towards +1/-1 for white to win/lose). This means that there is very litte training data (3% of all training data generated via self-play) that helps the ANN to learn how to win and to get stronger.
  
  Does anyone know how DeepMind has overcome this problem? I have found nothing about this in the www.
  
  **Tags:** reinforcement-learning, alphazero, chess, board-games
  
  > **Comments:**
  > **username_1**: Hello and welcome to AI Stack Exchange! How many games have you currently run during training?  
  > **username_1**: Also, are you using a MCTS?  
  > **username_0**: I have already run ca. 1000 self-play games. But that is enough to see that the vast majority end with a "draw" which does not generate any training effect.  
  > **username_0**: Yes, I general I use MCTS during self-play, but at the beginning I set "number of roll-outs" to just 1 in order to speed up the process (that led to the 97% draws). But during the last 24 hours I set "number of roll-outs" to 800 and observed a decrease of "draws" (down from 97% to 80%). But the 80% still seem far suboptimal for effective learning. I am wondering if it was really the huge number of self-play games that solved the problem in the original version of AlphaZero, or do I still miss something conceptionally-wise?  
  > **username_2**: @username_0 AlphaZero training with only a single visit is a bit strange, how would the policy head ever learning anything? You definitely need at least _some_ rollouts!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-01-23. Score: 1 | Accepted answer
  
  # First intuition
  
  I wrote a quick script to test the outcome distribution of random chess games: I get about 7% wins for black and white each, so 14% non-drawn games. The MCTS should still be able to find simple mate in one moves even with a fully uninitialized network, so I would expect it to draw even less.
  
  Your 97% draw rate is significantly higher than that, so I suspect something else is going wrong. Some ideas to try:
  
  * Do your own random moves test to confirm that you also get 7% win for each player, otherwise something went wrong with your board implementation.
  * Run MCTS with a random NN on a mate-in-one position, and ensure that it finds the mate and picks it as the best (highest visit count) move after a couple thousand visits.
  
  I think in theory 3% decisive games is already enough to kickstart the learning process, but specifically for chess it probably means something in the implementation is wrong.
  
  # Actual data
  
  Here's the win/draw/loss plot from a chess run of my own AlphaZero implementation, [kZero](https://github.com/username_2/kZero). The horizontal axis corresponds to about 300k selfplay games played. This is with 400 visits/move, and settings otherwise mostly similar to the AlphaZero paper.
  
  It looks like the behavior is as follows:
  
  * The initial games generated with a random NN have only 17% draws, so MCTS is finding a lot of mates
  * Very quickly the drawrate goes up to 35%, as the NN learns to avoid mate in one
  * After that the drawrate drops back down to 8%, as the NN learns to finish games from winning positions.
  * For the rest of the run the drawrate slowly climbs again, as is typical for stronger chess players and engines.
  
  [![WDL plot](https://i.sstatic.net/vVsP6.png)](https://i.sstatic.net/vVsP6.png)
  
  > **Comments:**
  > **username_0**: Thanks a lot for your quick and comprehensive answer!  
  > **username_0**: 1. I will do new random checks (without MCTS) to see if my win ratios will go up towards your 14% 2. I just checked that my MCTS clearly detects "mate in one"  3. What I find surprising is that your MCTS on a fully uninitialized NN only finds 17% draws. That's far off from my 97% 4. Currently I define a game as "drawn" if there is no mate after 100 moves. What rules do you apply to avoid endless games? 5. I will answer back here once I have 1. completed with (possibly) adjusted rules how to define draws.  
  > **username_2**: @username_0 I only terminate games based on 3-fold repetition, the 50 move rule and insufficient material (only king vs king). For this run the average game length started at 100, peaked at 300 and then dropped back down to 110. Cutting of games at 100 moves indeed causes lots of draws, so that's probably your main issue!  
  > **username_0**: Thanks a lot for clarification! I will test with modified termination rules.  
  
  ---
  
  