Answers.Body: <p>The <em>AlphaZero</em> algorithm uses the <strong>neural network for non-terminal states</strong> and the <strong>game rules for terminal states</strong> when determining the value <span class="math-container">$v$</span> of a node during MCTS expansion. The supplementary material of the <a href="https://www.science.org/doi/10.1126/science.aar6404" rel="nofollow noreferrer">paper</a> (<a href="https://arxiv.org/abs/1712.01815" rel="nofollow noreferrer">arXiv link</a>) states the following (see Domain Knowledge, item 2):</p>
  <blockquote>
  <p><em>AlphaZero</em> is provided with perfect knowledge of the game rules. These are used during MCTS, to simulate the positions resulting from a sequence of moves, to determine game termination, and <strong>to score any simulations that reach a terminal state</strong>.</p>
  </blockquote>
  <p>During a game, <em>AlphaZero</em> stores the game outcome <span class="math-container">$z$</span> (determined by the game rules) and predicted outcomes <span class="math-container">$v_t$</span> of each encountered state (determined by the neural network). One aim of the loss function is to minimize the error between <span class="math-container">$z$</span> and <span class="math-container">$v_t$</span> (see equation 1 in the paper).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-01T01:24:25.903Z
Answers.LastActivityDate: 2022-12-01T01:24:25.903Z
Answers.UserDisplayName: DeepQZero
Body: <p>According to what I understand from the <em>AlphaZero</em> algorithm, a neural network is used to set value and prior probability for a node during the expansion stage of MCTS. On the other hand, according to the rules of the game, we need to set the value equal to +1 a win, -1 for a loss, or 0 for a draw.</p>
  <p><strong>Question</strong>: When do we use the neural network to predict the value and when do we use the rules of the game to set the value during the expansion stage of MCTS in <em>AlphaZero</em>?</p>
  
Tags: <alphazero><alphago-zero><alphago>
Title: When do we use the neural network to predict value during the expansion stage of MCTS in the AlphaZero algorithm?
UserDisplayName: Mahdi Hosseini
ThreadText: # When do we use the neural network to predict value during the expansion stage of MCTS in the AlphaZero algorithm?
  
  (Asked by: username_0 on 2022-11-30)
  
  According to what I understand from the *AlphaZero* algorithm, a neural network is used to set value and prior probability for a node during the expansion stage of MCTS. On the other hand, according to the rules of the game, we need to set the value equal to +1 a win, -1 for a loss, or 0 for a draw.
  
  **Question**: When do we use the neural network to predict the value and when do we use the rules of the game to set the value during the expansion stage of MCTS in *AlphaZero*?
  
  **Tags:** alphazero, alphago-zero, alphago
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-12-01. Score: 0
  
  The *AlphaZero* algorithm uses the **neural network for non-terminal states** and the **game rules for terminal states** when determining the value $v$ of a node during MCTS expansion. The supplementary material of the [paper](https://www.science.org/doi/10.1126/science.aar6404) ([arXiv link](https://arxiv.org/abs/1712.01815)) states the following (see Domain Knowledge, item 2):
  
  > *AlphaZero* is provided with perfect knowledge of the game rules. These are used during MCTS, to simulate the positions resulting from a sequence of moves, to determine game termination, and **to score any simulations that reach a terminal state**.
  
  During a game, *AlphaZero* stores the game outcome $z$ (determined by the game rules) and predicted outcomes $v\_t$ of each encountered state (determined by the neural network). One aim of the loss function is to minimize the error between $z$ and $v\_t$ (see equation 1 in the paper).
  
  
  ---
  
  