Answers.Body: <p>You're right, it would not be great to keep training on games played early during the selfplay process, since they would just hurt the network playing strength.</p>
  <p>The <a href="https://www.science.org/doi/10.1126/science.aar6404" rel="nofollow noreferrer">AlphaZero paper</a> (<a href="https://arxiv.org/abs/1712.01815.pdf" rel="nofollow noreferrer">free access</a>) itself does not elaborate, but it has this sneaky sentence:</p>
  <blockquote>
  <p>Unless
  otherwise specified, the training and search algorithm and parameters are identical to <em>AlphaGo Zero</em></p>
  </blockquote>
  <p>The <a href="https://www.nature.com/articles/nature24270.pdf" rel="nofollow noreferrer">AlphaGo Zero paper</a> (<a href="https://discovery.ucl.ac.uk/id/eprint/10045895/1/agz_unformatted_nature.pdf" rel="nofollow noreferrer">free access</a>) is more helpful:</p>
  <blockquote>
  <p>Optimization: [...] Each mini­batch of data is sampled uniformly at random from all positions of the most recent 500,000 games of self­play.</p>
  </blockquote>
  <p>For context, in total AlphaGoZero played 4.9 million games, so the buffer rotated about 10 times.</p>
  <p>As another example, <a href="https://arxiv.org/abs/1902.10565" rel="nofollow noreferrer">KataGo</a> started with a small buffer early in training and then gradually grew it:</p>
  <blockquote>
  <p>Samples are drawn uniformly from a growing moving window of the most recent data, with window size beginning at 250,000 samples and increasing to about 22 million by the end of the main run. See Appendix C for details.</p>
  </blockquote>
  <p>(note that this is counting positions, not games like the AlphaZero paper)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-24T14:58:15.713Z
Answers.LastActivityDate: 2022-10-11T20:07:30.980Z
Answers.LastEditDate: 2022-10-11T20:07:30.980Z
Answers.UserDisplayName: KarelPeeters
Body: <p>Regarding the <a href="https://arxiv.org/pdf/1712.01815.pdf" rel="nofollow noreferrer">AlphaZero paper</a>, it is not clear to me when the Monte Carlo Tree Search (MCTS) results will be cleaned up.</p>
  <p>I assume this has to happen at some point, since mixing results could lead to lower quality results? Imagine in the self-play the Neural Network (NN) is updated to a new version and evaluates certain patterns differently by detecting a new trick. Many iterations must follow to outperform the old best choice (visit-count). I imagine discarding old MCTS results should be done about between an episode and the next NN weight updates.</p>
  <p>I feel that a wrong decision here could have a strong negative impact on the overall learning process.</p>
  
Tags: <reinforcement-learning><alphazero>
Title: At what point are MCTS results discarded in AlphaZero Training?
UserDisplayName: Daniel
ThreadText: # At what point are MCTS results discarded in AlphaZero Training?
  
  (Asked by: username_0 on 2021-07-30)
  
  Regarding the [AlphaZero paper](https://arxiv.org/pdf/1712.01815.pdf), it is not clear to me when the Monte Carlo Tree Search (MCTS) results will be cleaned up.
  
  I assume this has to happen at some point, since mixing results could lead to lower quality results? Imagine in the self-play the Neural Network (NN) is updated to a new version and evaluates certain patterns differently by detecting a new trick. Many iterations must follow to outperform the old best choice (visit-count). I imagine discarding old MCTS results should be done about between an episode and the next NN weight updates.
  
  I feel that a wrong decision here could have a strong negative impact on the overall learning process.
  
  **Tags:** reinforcement-learning, alphazero
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-01-24. Score: 4 | Accepted answer
  
  You're right, it would not be great to keep training on games played early during the selfplay process, since they would just hurt the network playing strength.
  
  The [AlphaZero paper](https://www.science.org/doi/10.1126/science.aar6404) ([free access](https://arxiv.org/abs/1712.01815.pdf)) itself does not elaborate, but it has this sneaky sentence:
  
  > Unless
  > otherwise specified, the training and search algorithm and parameters are identical to *AlphaGo Zero*
  
  The [AlphaGo Zero paper](https://www.nature.com/articles/nature24270.pdf) ([free access](https://discovery.ucl.ac.uk/id/eprint/10045895/1/agz_unformatted_nature.pdf)) is more helpful:
  
  > Optimization: [...] Each mini­batch of data is sampled uniformly at random from all positions of the most recent 500,000 games of self­play.
  
  For context, in total AlphaGoZero played 4.9 million games, so the buffer rotated about 10 times.
  
  As another example, [KataGo](https://arxiv.org/abs/1902.10565) started with a small buffer early in training and then gradually grew it:
  
  > Samples are drawn uniformly from a growing moving window of the most recent data, with window size beginning at 250,000 samples and increasing to about 22 million by the end of the main run. See Appendix C for details.
  
  (note that this is counting positions, not games like the AlphaZero paper)
  
  
  ---
  
  