Answers.Body: <p>The training target for the policy is always exactly the one described by the pseudocode; distribution proportional to visit counts, without any other kind of scaling.</p>
  <p>The softmax sampling with the temperature (different for first 30 moves than after that) is not used for the policy target, but is used for the &quot;real&quot; move selection by the agent in the self-play game (i.e. after it has run its search of 800 or 1600 or however many iterations it was for the root game state encountered in the self-play game).</p>
  <hr />
  <p>Generally, in my experience, you really wouldn't want the policy target to become <em>too</em> close to deterministic, except if really your entire MCTS already is extremely convinced about one move being clearly the best one all by itself (without getting pushed towards something more deterministic by a low-temperature softmax). If your policy becomes too deterministic too quickly, it destroys the exploration that we normally want MCTS to do inside its selection phase. Using a low-temperature softmax for the policy training target would introduce quite a big risk of this happening.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-09T20:55:42.910Z
Answers.LastActivityDate: 2021-12-09T20:55:42.910Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>In AlphaZero's attached pseudocode, they create a training target for the policy network in this way.</p>
  <pre><code>def store_search_statistics(self, root):
      sum_visits = sum(child.visit_count for child in root.children.itervalues())
      self.child_visits.append([
          root.children[a].visit_count / sum_visits if a in root.children else 0
          for a in range(self.num_actions)
      ])
  </code></pre>
  <p>In other words, the training target probability for a certain move is proportional to its visit count.</p>
  <p>However, in the paper, they describe the usage of softmax sampling of visit counts with temperature. This temperate is equal to 1 for the first 30 moves (in this case the policy training target is the same as in the pseudocode above) and for subsequent moves they set infinitesimal temperature -&gt; 0, which essentially means they are picking the move with the highest visit count.</p>
  <p>Since these are 2 different things (if the game has more than 30 moves), my question is: which approach should be used for creating the training target for the policy?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-08T07:54:27.860Z
Comments.Text: I've been so confused about this for the longest time. But yes, it seems that DeepMind make an error in their Nature paper. Yes, Policy is matched to the Visit count. Policy is fundamentally a predictor of final Visit count, and based on PUCT, the algorithm uses the Prior Policy when selecting child nodes in MCTS. In other words, PUCT will cause MCTS to start visiting nodes in the distribution that the neural network _predicts_ the final visit distribution will be, but MCTS will deviate from the NN's prediction if one of the paths has a more attractive Q.
Comments.UserDisplayName: Nicholas Pipitone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-08T08:08:43.897Z
Comments.Text: For real-life usage of Dennis's answer, [Lc0](https://github.com/LeelaChessZero/lc0/blob/master/src/trainingdata/trainingdata.cc)
Comments.UserDisplayName: Nicholas Pipitone
Tags: <reinforcement-learning><training><policy-gradients><monte-carlo-tree-search><alphazero>
Title: What is a policy training target in AlphaZero?
UserDisplayName: Druudik
ThreadText: # What is a policy training target in AlphaZero?
  
  (Asked by: username_0 on 2021-12-09)
  
  In AlphaZero's attached pseudocode, they create a training target for the policy network in this way.
  def store_search_statistics(self, root):
      sum_visits = sum(child.visit_count for child in root.children.itervalues())
      self.child_visits.append([
          root.children[a].visit_count / sum_visits if a in root.children else 0
          for a in range(self.num_actions)
      ])
  
  In other words, the training target probability for a certain move is proportional to its visit count.
  However, in the paper, they describe the usage of softmax sampling of visit counts with temperature. This temperate is equal to 1 for the first 30 moves (in this case the policy training target is the same as in the pseudocode above) and for subsequent moves they set infinitesimal temperature -> 0, which essentially means they are picking the move with the highest visit count.
  Since these are 2 different things (if the game has more than 30 moves), my question is: which approach should be used for creating the training target for the policy?
  
  **Tags:** reinforcement-learning, training, policy-gradients, monte-carlo-tree-search, alphazero
  > **Comments:**
  > **username_1**: I've been so confused about this for the longest time. But yes, it seems that DeepMind make an error in their Nature paper. Yes, Policy is matched to the Visit count. Policy is fundamentally a predictor of final Visit count, and based on PUCT, the algorithm uses the Prior Policy when selecting child nodes in MCTS. In other words, PUCT will cause MCTS to start visiting nodes in the distribution that the neural network _predicts_ the final visit distribution will be, but MCTS will deviate from the NN's prediction if one of the paths has a more attractive Q.  
  > **username_1**: For real-life usage of Dennis's answer, [Lc0](https://github.com/LeelaChessZero/lc0/blob/master/src/trainingdata/trainingdata.cc)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-12-09. Score: 3 | Accepted answer
  
  The training target for the policy is always exactly the one described by the pseudocode; distribution proportional to visit counts, without any other kind of scaling.
  
  The softmax sampling with the temperature (different for first 30 moves than after that) is not used for the policy target, but is used for the "real" move selection by the agent in the self-play game (i.e. after it has run its search of 800 or 1600 or however many iterations it was for the root game state encountered in the self-play game).
  
  ---
  
  Generally, in my experience, you really wouldn't want the policy target to become *too* close to deterministic, except if really your entire MCTS already is extremely convinced about one move being clearly the best one all by itself (without getting pushed towards something more deterministic by a low-temperature softmax). If your policy becomes too deterministic too quickly, it destroys the exploration that we normally want MCTS to do inside its selection phase. Using a low-temperature softmax for the policy training target would introduce quite a big risk of this happening.
  
  
  ---
  
  