Answers.Body: <p>Your observation is accurate. The use of the term &quot;simulation&quot; in the AlphaGoZero paper can indeed be confusing. It does not refer to the simulation or rollout phase of the Monte Carlo Tree Search. As you correctly pointed out, the AlphaGoZero model removes the rollout, which in the AlphaGo paper was a shallow neural network trained on supervised data. Instead, it uses a single neural network that outputs both policies and values for the given position in the expansion phase.</p>
  <p>In the excerpt you have provided, the term &quot;simulation&quot; refers to the exploration and play within the game tree. I would normally refer to the simulation phase as &quot;rollout&quot; to avoid confusion, but that is up to you. Otherwise, your understanding is correct.</p>
  <p>I hope this clarifies your doubts. If I have not made myself clear, please let me know.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-03T16:29:52.293Z
Answers.Text: Thank you for your reply. I am a little unsure of the word "exploration" here. Why do you think it's just exploration that's going on during simulation?
Answers.UserDisplayName: desert_ranger
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-03T20:34:13.643Z
Answers.Text: When I say "exploration," I am referring to the fact that the tree is increasing in size because it is exploring more nodes in the game. Reinforcement learning (RL) has the concept of exploration vs. exploitation, which comes from the idea of balancing between repeating what seems to be good actions and exploring new states and actions that have not been explored before. The MCTS algorithm is constructed in a way to make this balance efficient for an RL agent. However, when I said "exploration," I was referring to executing an MCTS iteration and expanding the tree.
Answers.UserDisplayName: Cesar Ruiz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-02T03:03:45.213Z
Answers.LastActivityDate: 2023-08-02T03:10:51.767Z
Answers.LastEditDate: 2023-08-02T03:10:51.767Z
Answers.UserDisplayName: Cesar Ruiz
Body: <p>I am familiar with the Monte Carlo Tree Search, which can be broken down into 4 parts - Selection, Expansion, Simulation and Backpropogation. In this case, the simulation step denotes performing a random rollout from the expanded node. From my understanding, the AlphaGoZero paper removes the MCTS simulation step and replaces it with a neural network that takes the leaf node as input and outputs a policy vector and a value.</p>
  <p>However, I am not sure what the <a href="https://www.nature.com/articles/nature24270" rel="nofollow noreferrer">AlphaGoZero</a> paper means when they use the term simulation. It contains the following paragraph - &quot;<em>The MCTS uses the neural network fθ to guide its simulations (see Fig. 2). Each edge (s, a) in the search tree stores a prior probability P(s, a), a visit count N(s, a), and an action value Q(s, a). Each simulation starts from the root state and iteratively selects moves that maximize an upper confidence bound Q(s, a)+U(s, a), where U(s, a)∝P(s, a)/(1+N(s, a)) (refs 12, 24), until a leaf node s′ is encountered.</em>&quot;</p>
  
Tags: <reinforcement-learning><deep-rl><alphago-zero>
Title: What do simulations mean in the context of the AlphaGoZero paper?
UserDisplayName: desert_ranger
ThreadText: # What do simulations mean in the context of the AlphaGoZero paper?
  
  (Asked by: username_0 on 2023-08-01)
  
  I am familiar with the Monte Carlo Tree Search, which can be broken down into 4 parts - Selection, Expansion, Simulation and Backpropogation. In this case, the simulation step denotes performing a random rollout from the expanded node. From my understanding, the AlphaGoZero paper removes the MCTS simulation step and replaces it with a neural network that takes the leaf node as input and outputs a policy vector and a value.
  
  However, I am not sure what the [AlphaGoZero](https://www.nature.com/articles/nature24270) paper means when they use the term simulation. It contains the following paragraph - "*The MCTS uses the neural network fθ to guide its simulations (see Fig. 2). Each edge (s, a) in the search tree stores a prior probability P(s, a), a visit count N(s, a), and an action value Q(s, a). Each simulation starts from the root state and iteratively selects moves that maximize an upper confidence bound Q(s, a)+U(s, a), where U(s, a)∝P(s, a)/(1+N(s, a)) (refs 12, 24), until a leaf node s′ is encountered.*"
  
  **Tags:** reinforcement-learning, deep-rl, alphago-zero
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-02. Score: 2 | Accepted answer
  
  Your observation is accurate. The use of the term "simulation" in the AlphaGoZero paper can indeed be confusing. It does not refer to the simulation or rollout phase of the Monte Carlo Tree Search. As you correctly pointed out, the AlphaGoZero model removes the rollout, which in the AlphaGo paper was a shallow neural network trained on supervised data. Instead, it uses a single neural network that outputs both policies and values for the given position in the expansion phase.
  
  In the excerpt you have provided, the term "simulation" refers to the exploration and play within the game tree. I would normally refer to the simulation phase as "rollout" to avoid confusion, but that is up to you. Otherwise, your understanding is correct.
  
  I hope this clarifies your doubts. If I have not made myself clear, please let me know.
  
  > **Comments:**
  > **username_0**: Thank you for your reply. I am a little unsure of the word "exploration" here. Why do you think it's just exploration that's going on during simulation?  
  > **username_1**: When I say "exploration," I am referring to the fact that the tree is increasing in size because it is exploring more nodes in the game. Reinforcement learning (RL) has the concept of exploration vs. exploitation, which comes from the idea of balancing between repeating what seems to be good actions and exploring new states and actions that have not been explored before. The MCTS algorithm is constructed in a way to make this balance efficient for an RL agent. However, when I said "exploration," I was referring to executing an MCTS iteration and expanding the tree.  
  
  ---
  
  