Answers.Body: <p><strong>TL;DR:</strong> Alpha Zero removed rollout altogether from MCTS and just used current DNN estimates instead.</p>
  <hr />
  <p>The single Deep Neural Network has 2 heads:</p>
  <ul>
  <li>A Value Head (which assigns a score to each state).</li>
  <li>And a Policy Head (which predicts the score for all possible moves).</li>
  </ul>
  <p>Instead of doing rollouts to determine the outcome, it uses the DNN estimations, so it don't need to explore too deep.</p>
  <p>By relying on the DNN, the MCTS gets even simpler:</p>
  <ul>
  <li><strong>Probability</strong> of each action is a simple normalization of previously obtained Policy Head.</li>
  <li><strong>Selection</strong> chooses move with “low count, high move
  probability, and high value”</li>
  <li><strong>Expansion</strong> is done by DNN, outputing <em>Value</em> and <em>Policy</em>.</li>
  <li><strong>Simulation</strong> with rollouts: no longer needed.</li>
  <li><strong>Back-propagation</strong> updates nodes using DNN's value (instead of a rollout outcome).</li>
  </ul>
  <p><a href="https://www.biostat.wisc.edu/%7Ecraven/cs760/lectures/AlphaZero.pdf" rel="nofollow noreferrer">Source</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-21T06:25:51.660Z
Answers.LastActivityDate: 2021-08-21T06:25:51.660Z
Answers.UserDisplayName: Andre Goulart
Body: <p>According to <a href="https://deepmind.com/blog/article/alphago-zero-starting-scratch" rel="nofollow noreferrer">a blog post by DeepMind</a>, AlphaZero doesn't have a real rollout.</p>
  <blockquote>
  <p>AlphaGo Zero does not use &quot;rollouts&quot; - fast, random games used by other Go programs to predict which player will win from the current board position. Instead, it relies on its high quality neural networks to evaluate positions.</p>
  </blockquote>
  <p>Instead, I assume it just interprets the winner at a given state by the NN values head result. This replaces the rollout. So the computation time saved could be used for many expansions instead. Evaluating a state from a root node would then be the best action derived from the visit count in MCTS, which is only based on the predictions of the NN value heads. (no current score, no policy?)</p>
  <p>With policy, I mean the NN's policy head (softmax).</p>
  <p>This would mean that the NN policy is only used in the loss calculation and nowhere else?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-21T06:00:01.650Z
Comments.Text: You have lot's of questions and interpretations there. So I'll do my best to explain it below.
Comments.UserDisplayName: Andre Goulart
Tags: <reinforcement-learning><alphazero>
Title: What does it mean there is no rollout in AlphaZero's training?
UserDisplayName: Daniel
ThreadText: # What does it mean there is no rollout in AlphaZero's training?
  
  (Asked by: username_0 on 2021-07-30)
  
  According to [a blog post by DeepMind](https://deepmind.com/blog/article/alphago-zero-starting-scratch), AlphaZero doesn't have a real rollout.
  
  > AlphaGo Zero does not use "rollouts" - fast, random games used by other Go programs to predict which player will win from the current board position. Instead, it relies on its high quality neural networks to evaluate positions.
  
  Instead, I assume it just interprets the winner at a given state by the NN values head result. This replaces the rollout. So the computation time saved could be used for many expansions instead. Evaluating a state from a root node would then be the best action derived from the visit count in MCTS, which is only based on the predictions of the NN value heads. (no current score, no policy?)
  
  With policy, I mean the NN's policy head (softmax).
  
  This would mean that the NN policy is only used in the loss calculation and nowhere else?
  
  **Tags:** reinforcement-learning, alphazero
  > **Comments:**
  > **username_1**: You have lot's of questions and interpretations there. So I'll do my best to explain it below.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-21. Score: 2 | Accepted answer
  
  **TL;DR:** Alpha Zero removed rollout altogether from MCTS and just used current DNN estimates instead.
  
  ---
  
  The single Deep Neural Network has 2 heads:
  
  * A Value Head (which assigns a score to each state).
  * And a Policy Head (which predicts the score for all possible moves).
  
  Instead of doing rollouts to determine the outcome, it uses the DNN estimations, so it don't need to explore too deep.
  
  By relying on the DNN, the MCTS gets even simpler:
  
  * **Probability** of each action is a simple normalization of previously obtained Policy Head.
  * **Selection** chooses move with “low count, high move
    probability, and high value”
  * **Expansion** is done by DNN, outputing *Value* and *Policy*.
  * **Simulation** with rollouts: no longer needed.
  * **Back-propagation** updates nodes using DNN's value (instead of a rollout outcome).
  
  [Source](https://www.biostat.wisc.edu/%7Ecraven/cs760/lectures/AlphaZero.pdf)
  
  
  ---
  
  