Body: <p>In the vanilla <a href="https://en.wikipedia.org/wiki/Monte_Carlo_tree_search" rel="nofollow noreferrer">Monte Carlo tree search (MCTS)</a> implementation, the rollout is usually implemented following a uniform random policy, that is, it takes random actions until the game is finished and only then the information gathered is backed up.</p>
  <p>I have read the AlphaZero paper (and the AlphaGo Zero too) and I didn't find any information on how the rollout is implemented (maybe I missed it).</p>
  <p>How is the rollout from the MCTS implemented in both the AlphaGo Zero and the AlphaZero algorithms?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-03T07:40:06.163Z
Comments.Text: AlphaZero/Go don't have rollout. Rollout replaced with neural network value estimation
Comments.UserDisplayName: mirror2image
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-03T09:03:45.110Z
Comments.Text: @mi: I thought that original Alpha Go *did* have a rollout policy, but that one of the simplifying changes in AlphaZero was to completely remove it?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-03T13:53:07.980Z
Comments.Text: @NeilSlater That's correct. The question seems to be about AlphaGo Zero and AlphaZero though, not about Alpha Go :) **AlphaGo**: uses a fast rollout policy (trained like the policy network, but is not a large DNN, just a single layer + softmax I believe). **AlphaGo Zero**: no rollouts. **AlphaZero**: no rollouts.
Comments.UserDisplayName: Dennis Soemers
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-03T17:57:52.330Z
Comments.Text: @mirror2image even though it was only a line, you answered the question -- why not make it slightly more verbose and make it an answer
Comments.UserDisplayName: mshlis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-03T23:18:40.597Z
Comments.Text: Thanks for the answers. However, I'm still confused. I understand that they use the V value returned from the network to simulate who won, but how do they know which value of V a player won or lost? Because as far as I understand, in the weight update (in the loss function), they will compare the probability distribution gathered from the self-plays and the difference between the real winner (from MCTS) and the value V.
Comments.UserDisplayName: ihavenoidea
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-03T23:19:30.217Z
Comments.Text: In short, I'm confused about the V value range returned from the network. Will it be in the [-1,1] range?
Comments.UserDisplayName: ihavenoidea
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-29T19:16:42.040Z
Comments.Text: @ihavenoidea  I got the same question have you figured it out?
Comments.UserDisplayName: Daniel
Tags: <monte-carlo-tree-search><alphazero><implementation><alphago-zero>
Title: How is the rollout from the MCTS implemented in both of the AlphaGo Zero and the AlphaZero algorithms?
UserDisplayName: ihavenoidea
ThreadText: # How is the rollout from the MCTS implemented in both of the AlphaGo Zero and the AlphaZero algorithms?
  
  (Asked by: username_0 on 2019-11-03)
  
  In the vanilla [Monte Carlo tree search (MCTS)](https://en.wikipedia.org/wiki/Monte_Carlo_tree_search) implementation, the rollout is usually implemented following a uniform random policy, that is, it takes random actions until the game is finished and only then the information gathered is backed up.
  
  I have read the AlphaZero paper (and the AlphaGo Zero too) and I didn't find any information on how the rollout is implemented (maybe I missed it).
  
  How is the rollout from the MCTS implemented in both the AlphaGo Zero and the AlphaZero algorithms?
  
  **Tags:** monte-carlo-tree-search, alphazero, implementation, alphago-zero
  > **Comments:**
  > **username_1**: AlphaZero/Go don't have rollout. Rollout replaced with neural network value estimation  
  > **username_2**: @mi: I thought that original Alpha Go *did* have a rollout policy, but that one of the simplifying changes in AlphaZero was to completely remove it?  
  > **username_3**: @NeilSlater That's correct. The question seems to be about AlphaGo Zero and AlphaZero though, not about Alpha Go :) **AlphaGo**: uses a fast rollout policy (trained like the policy network, but is not a large DNN, just a single layer + softmax I believe). **AlphaGo Zero**: no rollouts. **AlphaZero**: no rollouts.  
  > **username_4**: @mirror2image even though it was only a line, you answered the question -- why not make it slightly more verbose and make it an answer  
  > **username_0**: Thanks for the answers. However, I'm still confused. I understand that they use the V value returned from the network to simulate who won, but how do they know which value of V a player won or lost? Because as far as I understand, in the weight update (in the loss function), they will compare the probability distribution gathered from the self-plays and the difference between the real winner (from MCTS) and the value V.  
  > **username_0**: In short, I'm confused about the V value range returned from the network. Will it be in the [-1,1] range?  
  > **username_5**: @ihavenoidea  I got the same question have you figured it out?  
  
  ---
  
  ## 0 answers
  
  