Answers.Body: <p>During the self-play training process, AlphaZero does not greedily play only the moves it thinks are "best" (which would normally be the move with the highest visit count leading out of the root node of the MCTS search tree). Instead, for the purpose of generating a more diverse set of experience, it samples moves <strong>proportionally</strong> to the visit counts. This means that in any given situation encountered during self-play, the move considered to be "optimal" will still have the largest probability of getting picked, but other moves will also have smaller probabilities of getting picked. In theory, it might even sometimes pick the move that it expects to be the <strong>worst</strong> one (very rarely)!</p>
  
  <p>If I recall correctly, they only did what I describe above for the first 30 moves of any game, and afterwards move on to greedy play. This still results in a very diverse set of 30-move-starts for every game it experiences though. I'm not 100% sure if I remember this detail correctly though, maybe they only did this earlier (in AlphaGo Zero for example), and no longer do it in AlphaZero. Would have to check the paper to make sure.</p>
  
  <p>Additionally, whenever they start a new search process, they perturb the prior probabilities assigned by the learned policy network to all the moves available in the root node. This is done in a non-deterministic way using Dirichlet noise. I think this is not really explicitly mentioned in the AlphaZero paper, but it is in the Supplementary Materials (and also in the AlphaGo Zero paper?). Anyway, this also means that if precisely the same game state is encountered twice in two different games of self-play, the search behaviour may be slightly different due to the introduced stochasticity, and hence it may come to a different conclusion as to what move is "optimal".</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-28T08:49:14.597Z
Answers.LastActivityDate: 2019-08-28T08:49:14.597Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>If AlphaZero was always playing the best moves it would just generate the same training game over and over again. So where does the randomness come from? When does it decide not to play the most optimal move?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-27T21:02:04.727Z
Comments.Text: AlphaZero does not know the optimal move. It has not "solved" the games of go or of chess etc. So could you clarify that you are looking for how the AlphaZero algorithm explores state/action space during training only (technically it explores during non-training play as well)?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-27T22:20:00.400Z
Comments.Text: The optimal move according to the network. By playing the moves it thinks are best it would just play the same moves. For example it plays first move e4 because it is the best move then it sees the new position and thinks that e5 is the best for black and it would just repeat this singe game over and over without producing any meaningful training data
Comments.UserDisplayName: Bojidar Ivanov
Tags: <reinforcement-learning><deep-rl><alphazero><greedy-ai>
Title: When does AlphaZero play suboptimal moves?
UserDisplayName: Bojidar Ivanov
ThreadText: # When does AlphaZero play suboptimal moves?
  
  (Asked by: username_0 on 2019-08-27)
  
  If AlphaZero was always playing the best moves it would just generate the same training game over and over again. So where does the randomness come from? When does it decide not to play the most optimal move?
  
  **Tags:** reinforcement-learning, deep-rl, alphazero, greedy-ai
  
  > **Comments:**
  > **username_1**: AlphaZero does not know the optimal move. It has not "solved" the games of go or of chess etc. So could you clarify that you are looking for how the AlphaZero algorithm explores state/action space during training only (technically it explores during non-training play as well)?  
  > **username_0**: The optimal move according to the network. By playing the moves it thinks are best it would just play the same moves. For example it plays first move e4 because it is the best move then it sees the new position and thinks that e5 is the best for black and it would just repeat this singe game over and over without producing any meaningful training data  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-08-28. Score: 3 | Accepted answer
  
  During the self-play training process, AlphaZero does not greedily play only the moves it thinks are "best" (which would normally be the move with the highest visit count leading out of the root node of the MCTS search tree). Instead, for the purpose of generating a more diverse set of experience, it samples moves **proportionally** to the visit counts. This means that in any given situation encountered during self-play, the move considered to be "optimal" will still have the largest probability of getting picked, but other moves will also have smaller probabilities of getting picked. In theory, it might even sometimes pick the move that it expects to be the **worst** one (very rarely)!
  
  If I recall correctly, they only did what I describe above for the first 30 moves of any game, and afterwards move on to greedy play. This still results in a very diverse set of 30-move-starts for every game it experiences though. I'm not 100% sure if I remember this detail correctly though, maybe they only did this earlier (in AlphaGo Zero for example), and no longer do it in AlphaZero. Would have to check the paper to make sure.
  
  Additionally, whenever they start a new search process, they perturb the prior probabilities assigned by the learned policy network to all the moves available in the root node. This is done in a non-deterministic way using Dirichlet noise. I think this is not really explicitly mentioned in the AlphaZero paper, but it is in the Supplementary Materials (and also in the AlphaGo Zero paper?). Anyway, this also means that if precisely the same game state is encountered twice in two different games of self-play, the search behaviour may be slightly different due to the introduced stochasticity, and hence it may come to a different conclusion as to what move is "optimal".
  
  
  ---
  
  