Answers.Body: <blockquote>
  <p>Did AlphaGo and AlphaGo [Zero] play 100 repetitions of the same sequence of boards, or were there 100 different games?</p>
  </blockquote>
  <p>There were 100 different games. <a href="https://www.alphago-games.com/#agzero_vs_aglee" rel="nofollow noreferrer">You can view some example games between AlphaGo [Lee] and AlphaGo Zero here</a>. They are clearly all different.</p>
  <p>This statement in the question shows a misunderstanding:</p>
  <blockquote>
  <p>My understanding of AlphaGo and AlphaGo [Zero] are that they are deterministic, not stochastic.</p>
  </blockquote>
  <p>The <a href="https://en.wikipedia.org/wiki/Monte_Carlo_tree_search" rel="nofollow noreferrer">Monte Carlo Tree Search (MCTS)</a> algorithm used for look-ahead planning in AlphaGo and Alpha Zero is inherently stochastic. It samples from the huge tree of possibilities in a game like Go by making weighted random choices at all branch points. That means play can progress stochastically with two such agents opposing each other, as many board states will resolve into selecting semi-randomly between &quot;best&quot; moves that would be very closely ranked by each agent in the limit of very long search times.</p>
  <p>Whilst this solves the main point of your question, it is worth noting that there can be a related effect in self-play algorithms, even if they are partially stochastic. That is, it is possible to have one agent that develops a specific weakness by chance, that another agent consistently takes advantage of, such that agent A consistently beats agent B, and wins in a very similar fashion each time (maybe deterministically, maybe across a range of different games all with a similar mistake). However it may be the case that also:</p>
  <ul>
  <li><p>Neither agent is strong in general.</p>
  </li>
  <li><p>Another agent C can beat B consistently, but will lose to A consistently. There would then be no clear way to rank agents A, B, and C without further measurements.</p>
  </li>
  </ul>
  <p>Agents trained through self play therefore do need to be trained and tested against a wide range of opponents to verify this is not happening and that the  skill level assessment is valid more generally. I believe this was done with all the AlphaGo variants built by DeepMind.</p>
  <p>The MCTS algorithm does help a little with this scenario as it can correct for weaknesses in how a trained neural network rates early board positions. The look-ahead planning of MCTS makes initial ratings less relevant to eventual action selection - effectively it refines those learned ratings using the samples from current position.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-24T14:36:39.970Z
Answers.Text: A big part of this answer is wrong, MCTS as used in Alpha(Go)Zero is not inherently stochastic. In selfplay they force game diversity by injecting additional noise into the network policy output and by (weighed) randomly sampling the move to played during the early game. In tournament play only the best move is played, diversity is forced either by using an opening book or again by random sampling during the first couple of moves.
Answers.UserDisplayName: KarelPeeters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-24T14:41:17.713Z
Answers.Text: @ToddSewell I think this follows on from discussion on another question? I will need to look into it with more detail. However, AlphaGo [Lee] did have random rollouts using a simplied rollout policy IIRC? So it is possible for me to be wrong in my comments on your answer, but more correct here (at least regarding AlphaGo [Lee]). Anyway, I think I will need to research in more detail before either commenting on your answer or editing this one
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-24T15:01:30.697Z
Answers.Text: Ah I didn't realize you were the same person I was discussing with elsewhere, my bad! I was looking around for other sources and came across this answer, but failed to realize this was about the Lee version. I'll delete my comment here since I don't know enough about it to make these strong claims.
Answers.UserDisplayName: KarelPeeters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-21T19:17:21.453Z
Answers.LastActivityDate: 2020-10-21T21:36:57.400Z
Answers.LastEditDate: 2020-10-21T21:36:57.400Z
Answers.UserDisplayName: Neil Slater
Body: <p><strong>tl;dr</strong><br />
  Did AlphaGo and AlphaGo play 100 repetitions of the same sequence of boards, or were there 100 different games?</p>
  <p><strong>Background:</strong><br />
  Alphago was the first superhuman go player, but it had human tuning and training.</p>
  <p>AlphaGo zero learned to be more superhuman than superhuman.  Its supremacy was shown by how it beat AlphaGo perfectly in 100 games.</p>
  <p>My understanding of AlphaGo and AlphaGo are that they are deterministic, not stochastic.</p>
  <p>If they are deterministic, then given a board position they will always make the same move.</p>
  <p>The way that mathematicians count the possible games in chess is to account for different board positions.  As I understand it, and I could be wrong, if they have the exact same sequence of board positions then it does not count as a different game.</p>
  <p>If they make the same sequence of moves 100 times, then they did not play 100 different games, but played one game for 100 repetitions.</p>
  <p><strong>Question:</strong><br />
  So, using the mathematical definition, did AlphaGo and AlphaGo Zero play only one game for 100 iterations or did they play 100 different games?</p>
  <p><strong>References:</strong></p>
  <ul>
  <li><a href="https://www.scientificamerican.com/article/ai-versus-ai-self-taught-alphago-zero-vanquishes-its-predecessor/" rel="nofollow noreferrer">https://www.scientificamerican.com/article/ai-versus-ai-self-taught-alphago-zero-vanquishes-its-predecessor/</a></li>
  <li><a href="https://deepmind.com/blog/article/alphago-zero-starting-scratch" rel="nofollow noreferrer">https://deepmind.com/blog/article/alphago-zero-starting-scratch</a></li>
  <li><a href="https://mathworld.wolfram.com/Stochastic.html" rel="nofollow noreferrer">https://mathworld.wolfram.com/Stochastic.html</a></li>
  </ul>
  
Tags: <alphago-zero><alphago><deterministic-policy><stochastic-policy>
Title: Did Alphago zero actually beat Alphago 100 games to 0?
UserDisplayName: EngrStudent
ThreadText: # Did Alphago zero actually beat Alphago 100 games to 0?
  
  (Asked by: username_0 on 2020-10-21)
  
  **tl;dr**  
  Did AlphaGo and AlphaGo play 100 repetitions of the same sequence of boards, or were there 100 different games?
  
  **Background:**  
  Alphago was the first superhuman go player, but it had human tuning and training.
  
  AlphaGo zero learned to be more superhuman than superhuman. Its supremacy was shown by how it beat AlphaGo perfectly in 100 games.
  
  My understanding of AlphaGo and AlphaGo are that they are deterministic, not stochastic.
  
  If they are deterministic, then given a board position they will always make the same move.
  
  The way that mathematicians count the possible games in chess is to account for different board positions. As I understand it, and I could be wrong, if they have the exact same sequence of board positions then it does not count as a different game.
  
  If they make the same sequence of moves 100 times, then they did not play 100 different games, but played one game for 100 repetitions.
  
  **Question:**  
  So, using the mathematical definition, did AlphaGo and AlphaGo Zero play only one game for 100 iterations or did they play 100 different games?
  
  **References:**
  
  * <https://www.scientificamerican.com/article/ai-versus-ai-self-taught-alphago-zero-vanquishes-its-predecessor/>
  * <https://deepmind.com/blog/article/alphago-zero-starting-scratch>
  * <https://mathworld.wolfram.com/Stochastic.html>
  
  **Tags:** alphago-zero, alphago, deterministic-policy, stochastic-policy
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-10-21. Score: 7
  
  > Did AlphaGo and AlphaGo [Zero] play 100 repetitions of the same sequence of boards, or were there 100 different games?
  
  There were 100 different games. [You can view some example games between AlphaGo [Lee] and AlphaGo Zero here](https://www.alphago-games.com/#agzero_vs_aglee). They are clearly all different.
  
  This statement in the question shows a misunderstanding:
  
  > My understanding of AlphaGo and AlphaGo [Zero] are that they are deterministic, not stochastic.
  
  The [Monte Carlo Tree Search (MCTS)](https://en.wikipedia.org/wiki/Monte_Carlo_tree_search) algorithm used for look-ahead planning in AlphaGo and Alpha Zero is inherently stochastic. It samples from the huge tree of possibilities in a game like Go by making weighted random choices at all branch points. That means play can progress stochastically with two such agents opposing each other, as many board states will resolve into selecting semi-randomly between "best" moves that would be very closely ranked by each agent in the limit of very long search times.
  
  Whilst this solves the main point of your question, it is worth noting that there can be a related effect in self-play algorithms, even if they are partially stochastic. That is, it is possible to have one agent that develops a specific weakness by chance, that another agent consistently takes advantage of, such that agent A consistently beats agent B, and wins in a very similar fashion each time (maybe deterministically, maybe across a range of different games all with a similar mistake). However it may be the case that also:
  
  * Neither agent is strong in general.
  * Another agent C can beat B consistently, but will lose to A consistently. There would then be no clear way to rank agents A, B, and C without further measurements.
  
  Agents trained through self play therefore do need to be trained and tested against a wide range of opponents to verify this is not happening and that the skill level assessment is valid more generally. I believe this was done with all the AlphaGo variants built by DeepMind.
  
  The MCTS algorithm does help a little with this scenario as it can correct for weaknesses in how a trained neural network rates early board positions. The look-ahead planning of MCTS makes initial ratings less relevant to eventual action selection - effectively it refines those learned ratings using the samples from current position.
  
  > **Comments:**
  > **username_2**: A big part of this answer is wrong, MCTS as used in Alpha(Go)Zero is not inherently stochastic. In selfplay they force game diversity by injecting additional noise into the network policy output and by (weighed) randomly sampling the move to played during the early game. In tournament play only the best move is played, diversity is forced either by using an opening book or again by random sampling during the first couple of moves.  
  > **username_1**: @ToddSewell I think this follows on from discussion on another question? I will need to look into it with more detail. However, AlphaGo [Lee] did have random rollouts using a simplied rollout policy IIRC? So it is possible for me to be wrong in my comments on your answer, but more correct here (at least regarding AlphaGo [Lee]). Anyway, I think I will need to research in more detail before either commenting on your answer or editing this one  
  > **username_2**: Ah I didn't realize you were the same person I was discussing with elsewhere, my bad! I was looking around for other sources and came across this answer, but failed to realize this was about the Lee version. I'll delete my comment here since I don't know enough about it to make these strong claims.  
  
  ---
  
  