Answers.Body: <p>The algorithm in the <strong>2012 survey article</strong> (your second link) is the <strong>most common / standard implementation</strong>. Whenever someone mentions using MCTS or UCT, without explicitly stating any other info, it's safe to assume that that pseudocode is what they're using.</p>
  
  <p>The paper by <strong>Kocsis and Szepesvári from 2006</strong> (your first link) is (one of) the <strong>original publication(s) on UCT</strong>. It is very similar to the "standard" implementation as described in the survey paper. If I recall correctly, the only important difference is that the algorithm as described in 2006 keeps track of at which point in time during an episode a reward is observed, and accounts for that timing in the backpropagation phase (i.e. if a reward is observed at time <span class="math-container">$t$</span> in an episode, credit for that reward is not assigned to states/action after time <span class="math-container">$t$</span>). It also used a discounting factor <span class="math-container">$\gamma$</span> to discount the importance of reward depending on temporal distance, which is uncommon otherwise in MCTS literature.</p>
  
  <p>Both of those differences are due to the original, 2006 paper having more of a "Markov decision process" or "Reinforcement Learning" flavour, whereas otherwise MCTS has especially become popular in AI for (board) games where we often by default assume that there is only a single nonzero reward (win or lose) at the end of every episode anyway, which makes those differences less meaningful.</p>
  
  <hr>
  
  <p>Both of the AlphaGo papers (AlphaGo and AlphaGo Zero) use a "foundation" of MCTS that is mostly similar to the one from the 2012 survey article. The core components are all the same. Both of those systems add on a lot of (very important, and some quite complex) bells and whistles though. </p>
  
  <p>Going off the top of my head (should be mostly accurate, but best details are in the original source of course!) <strong>AlphaGo</strong> (your third link) added Neural Networks trained in various ways to output policies (mapping from states to probability distributions over actions) and value functions (mapping from states to "value estimates", or estimates of win percentage). Trained policy networks were used in a variant of the "selection" phase (no longer UCB1 strategy as in UCT) to bias selection. A different (simpler function, not a deep net) policy was used to run play-outs (no long uniform at random action selection as in UCT). A combination of observed reward at end of play-out + value function estimate at start of play-out was used for backpropagation (no longer only backpropagating reward observed in play-out as in UCT).</p>
  
  <p><strong>AlphaGo Zero</strong> is, if we're purely looking at the MCTS part, quite similar to AlphaGo. The Neural Networks were a bit different (a single one, with multiple outputs for policy + value), and play-outs were no longer used at all (just immediate backpropagation of value function estimates after MCTS' selection + expansion phases). Apart from that, the primary differences going from AlphaGo to AlphaGo Zero were in the learning process used to train the Neural Networks really, not so much in the MCTS side of things.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-01T18:10:04.220Z
Answers.LastActivityDate: 2019-04-01T18:10:04.220Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I am trying to understand the similarities and differences between: (i) the UCT algorithm in Kocsis and Szepesvári (2006); (ii) the UCT algorithm in Section 3.3 of Browne et al (2012); (iii) the MCTS algorithm in Silver et al. (2016); (iv) the MCTS algorithm in Silver et al. (2017).</p>
  
  <p>I would be really grateful for some help identifying the similarities and differences in these papers, I am doing some research and really struggling right now.</p>
  
  <p>(i) <a href="http://ggp.stanford.edu/readings/uct.pdf" rel="nofollow noreferrer">http://ggp.stanford.edu/readings/uct.pdf</a> </p>
  
  <p>(ii) <a href="http://mcts.ai/pubs/mcts-survey-master.pdf" rel="nofollow noreferrer">http://mcts.ai/pubs/mcts-survey-master.pdf</a> (Section 3.3)</p>
  
  <p>(iii) <a href="https://storage.googleapis.com/deepmind-media/alphago/AlphaGoNaturePaper.pdf" rel="nofollow noreferrer">https://storage.googleapis.com/deepmind-media/alphago/AlphaGoNaturePaper.pdf</a></p>
  
  <p>(iv) <a href="https://deepmind.com/documents/119/agz_unformatted_nature.pdf" rel="nofollow noreferrer">https://deepmind.com/documents/119/agz_unformatted_nature.pdf</a></p>
  
Tags: <algorithm><monte-carlo-tree-search><alphago><alphazero><monte-carlo-methods>
Title: Similarities and differences between UCT algorithms in (i), (ii), (iii) and (iv)?
UserDisplayName: Proof Reading
ThreadText: # Similarities and differences between UCT algorithms in (i), (ii), (iii) and (iv)?
  
  (Asked by: username_0 on 2019-03-31)
  
  I am trying to understand the similarities and differences between: (i) the UCT algorithm in Kocsis and Szepesvári (2006); (ii) the UCT algorithm in Section 3.3 of Browne et al (2012); (iii) the MCTS algorithm in Silver et al. (2016); (iv) the MCTS algorithm in Silver et al. (2017).
  
  I would be really grateful for some help identifying the similarities and differences in these papers, I am doing some research and really struggling right now.
  
  (i) <http://ggp.stanford.edu/readings/uct.pdf>
  
  (ii) <http://mcts.ai/pubs/mcts-survey-master.pdf> (Section 3.3)
  
  (iii) <https://storage.googleapis.com/deepmind-media/alphago/AlphaGoNaturePaper.pdf>
  
  (iv) <https://deepmind.com/documents/119/agz_unformatted_nature.pdf>
  
  **Tags:** algorithm, monte-carlo-tree-search, alphago, alphazero, monte-carlo-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-04-01. Score: 2
  
  The algorithm in the **2012 survey article** (your second link) is the **most common / standard implementation**. Whenever someone mentions using MCTS or UCT, without explicitly stating any other info, it's safe to assume that that pseudocode is what they're using.
  
  The paper by **Kocsis and Szepesvári from 2006** (your first link) is (one of) the **original publication(s) on UCT**. It is very similar to the "standard" implementation as described in the survey paper. If I recall correctly, the only important difference is that the algorithm as described in 2006 keeps track of at which point in time during an episode a reward is observed, and accounts for that timing in the backpropagation phase (i.e. if a reward is observed at time $t$ in an episode, credit for that reward is not assigned to states/action after time $t$). It also used a discounting factor $\gamma$ to discount the importance of reward depending on temporal distance, which is uncommon otherwise in MCTS literature.
  
  Both of those differences are due to the original, 2006 paper having more of a "Markov decision process" or "Reinforcement Learning" flavour, whereas otherwise MCTS has especially become popular in AI for (board) games where we often by default assume that there is only a single nonzero reward (win or lose) at the end of every episode anyway, which makes those differences less meaningful.
  
  ---
  
  Both of the AlphaGo papers (AlphaGo and AlphaGo Zero) use a "foundation" of MCTS that is mostly similar to the one from the 2012 survey article. The core components are all the same. Both of those systems add on a lot of (very important, and some quite complex) bells and whistles though.
  
  Going off the top of my head (should be mostly accurate, but best details are in the original source of course!) **AlphaGo** (your third link) added Neural Networks trained in various ways to output policies (mapping from states to probability distributions over actions) and value functions (mapping from states to "value estimates", or estimates of win percentage). Trained policy networks were used in a variant of the "selection" phase (no longer UCB1 strategy as in UCT) to bias selection. A different (simpler function, not a deep net) policy was used to run play-outs (no long uniform at random action selection as in UCT). A combination of observed reward at end of play-out + value function estimate at start of play-out was used for backpropagation (no longer only backpropagating reward observed in play-out as in UCT).
  
  **AlphaGo Zero** is, if we're purely looking at the MCTS part, quite similar to AlphaGo. The Neural Networks were a bit different (a single one, with multiple outputs for policy + value), and play-outs were no longer used at all (just immediate backpropagation of value function estimates after MCTS' selection + expansion phases). Apart from that, the primary differences going from AlphaGo to AlphaGo Zero were in the learning process used to train the Neural Networks really, not so much in the MCTS side of things.
  
  
  ---
  
  