Answers.Body: <blockquote>
  <p>However, I’m not sure which policy is saved</p>
  </blockquote>
  <p>The policy from the Monte Carlo tree search is stored, as we can get the policy estimate from the network later by passing the given state through the network, which is used to calculate the cross entropy loss to update the network's policy (summed with Mean squared error loss between value head's prediction and the actual value/reward).</p>
  <blockquote>
  <p>Wouldn’t it be more logical to choose the move with the highest probability calculated by the policy head?</p>
  </blockquote>
  <p>Depends on the number of searches you've performed, after thousands of simulations the MCTS would give better results, as it approximates the minimax tree.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T18:40:01.037Z
Answers.Text: To be more precise, I would say that "the visit counts" of the actions from the root noe of the Monte Carlo tree search are stored. Just "the policy" might be a bit too vague. Also, welcome to the site :)
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-30T06:07:07.420Z
Answers.Text: @DennisSoemers but aren't the visit counts of each action used to calculate the policy?
Answers.UserDisplayName: Abhishek Kaushik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-30T10:02:13.203Z
Answers.Text: Sure, but "policy" can mean a lot of things. It can mean what you intend it to mean, you're not wrong. But the strategy we use in the "Selection" phase (like PUCT) is also often called a "tree policy". In Reinforcement Learning, a policy would be a mapping from states to a probability distribution over actions. You could view the visit counts (after normalising) as being such a (discrete) probability disrtibution.. but it could also refer to the policy network being trained! Hence, there's room for confusion. Explicitly saying that the visit counts gets stored leaves less room for confusion :)
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T18:30:55.510Z
Answers.LastActivityDate: 2020-07-29T18:30:55.510Z
Answers.UserDisplayName: Abhishek Kaushik
Body: <p>I've been doing some research on the principles behind AlphaZero.
  Especially this <a href="https://adspassets.blob.core.windows.net/website/content/alpha_go_zero_cheat_sheet.png" rel="nofollow noreferrer">cheat sheet</a> (1) and this <a href="https://towardsdatascience.com/from-scratch-implementation-of-alphazero-for-connect4-f73d4554002a" rel="nofollow noreferrer">implementation</a> (2) (in Connect 4) were very useful.</p>
  <p>Yet, I still have two important questions:</p>
  <ol>
  <li><p>How is the policy network updated? In (2), board positions are saved in a dataset of tuples (state, policy, value). The value is derived from the result of the self-played game. However, I'm not sure which policy is saved: the number of times that each move has been played, the prior probabilities for each move (I guess not), or something else?</p>
  </li>
  <li><p>The cheat sheet says that (for competitive play) the move is chosen with the greatest N (=most visited). Wouldn't it be more logical to choose the move with the highest probability calculated by the policy head?</p>
  </li>
  </ol>
  
Tags: <reinforcement-learning><alphazero><self-play>
Title: In AlphaZero, which policy is saved in the dataset, and how is the move chosen?
UserDisplayName: Jonas De Schouwer
ThreadText: # In AlphaZero, which policy is saved in the dataset, and how is the move chosen?
  
  (Asked by: username_0 on 2020-01-10)
  
  I've been doing some research on the principles behind AlphaZero.
  Especially this [cheat sheet](https://adspassets.blob.core.windows.net/website/content/alpha_go_zero_cheat_sheet.png) (1) and this [implementation](https://towardsdatascience.com/from-scratch-implementation-of-alphazero-for-connect4-f73d4554002a) (2) (in Connect 4) were very useful.
  
  Yet, I still have two important questions:
  
  1. How is the policy network updated? In (2), board positions are saved in a dataset of tuples (state, policy, value). The value is derived from the result of the self-played game. However, I'm not sure which policy is saved: the number of times that each move has been played, the prior probabilities for each move (I guess not), or something else?
  2. The cheat sheet says that (for competitive play) the move is chosen with the greatest N (=most visited). Wouldn't it be more logical to choose the move with the highest probability calculated by the policy head?
  
  **Tags:** reinforcement-learning, alphazero, self-play
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-29. Score: 1
  
  > However, I’m not sure which policy is saved
  
  The policy from the Monte Carlo tree search is stored, as we can get the policy estimate from the network later by passing the given state through the network, which is used to calculate the cross entropy loss to update the network's policy (summed with Mean squared error loss between value head's prediction and the actual value/reward).
  
  > Wouldn’t it be more logical to choose the move with the highest probability calculated by the policy head?
  
  Depends on the number of searches you've performed, after thousands of simulations the MCTS would give better results, as it approximates the minimax tree.
  
  > **Comments:**
  > **username_2**: To be more precise, I would say that "the visit counts" of the actions from the root noe of the Monte Carlo tree search are stored. Just "the policy" might be a bit too vague. Also, welcome to the site :)  
  > **username_1**: @DennisSoemers but aren't the visit counts of each action used to calculate the policy?  
  > **username_2**: Sure, but "policy" can mean a lot of things. It can mean what you intend it to mean, you're not wrong. But the strategy we use in the "Selection" phase (like PUCT) is also often called a "tree policy". In Reinforcement Learning, a policy would be a mapping from states to a probability distribution over actions. You could view the visit counts (after normalising) as being such a (discrete) probability disrtibution.. but it could also refer to the policy network being trained! Hence, there's room for confusion. Explicitly saying that the visit counts gets stored leaves less room for confusion :)  
  
  ---
  
  