Answers.Body: <p>There is an element in MCTS, in which you select to explore &quot;promising&quot; directions. You also give a chance for branches that were not explored enough yet. So, you can use for that Bayesian methods (to keep and maintain the distribution of the expected rewards).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-07T03:19:18.410Z
Answers.Text: I'm curious if you know if anyone has tried using MCMC for this?
Answers.UserDisplayName: profPlum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-08T17:07:25.567Z
Answers.Text: AlphaGo as far as I can tell at this moment, is basically MCTS + maintaining and updating a value function and using it for making decision where to spend the next iterations. The vanilla MCTS makes use of Upper Confidence Bound (UCB), which is a handy / simple surrogate for a distribution.
Answers.UserDisplayName: Oren
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-28T13:46:19.697Z
Answers.LastActivityDate: 2023-06-28T13:46:19.697Z
Answers.UserDisplayName: Oren
Body: <p><a href="https://en.wikipedia.org/wiki/Monte_Carlo_tree_search" rel="nofollow noreferrer">Monte-Carlo Tree Search</a> was the method used for AlphaGo my understanding is: it would randomly search the state space of possible moves where the probability of choosing a move was proportional to the perceived Value of the resulting state (this rollout data was then used to train the value function further).</p>
  <p>This seems very similar to <a href="https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo" rel="nofollow noreferrer">MCMC</a> to me. Except that it includes a mechanism for observing ground truth likelihood (winners/losers in GO) &amp; updating the likelihood function (aka Value function) in the process.</p>
  <p>Is this true? And to what degree in general have Bayesian methods been used in RL?
  For example have people tried using one of the various specialized MCMC algorithms to improve Monte-Carlo tree search?</p>
  <p>Sorry if the question is open-ended/vague.</p>
  
Tags: <reinforcement-learning><comparison><monte-carlo-tree-search><monte-carlo-methods><markov-chain-monte-carlo>
Title: How does Monte-Carlo Tree Search Compare to MCMC?
UserDisplayName: profPlum
ThreadText: # How does Monte-Carlo Tree Search Compare to MCMC?
  
  (Asked by: username_0 on 2023-05-11)
  
  [Monte-Carlo Tree Search](https://en.wikipedia.org/wiki/Monte_Carlo_tree_search) was the method used for AlphaGo my understanding is: it would randomly search the state space of possible moves where the probability of choosing a move was proportional to the perceived Value of the resulting state (this rollout data was then used to train the value function further).
  
  This seems very similar to [MCMC](https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo) to me. Except that it includes a mechanism for observing ground truth likelihood (winners/losers in GO) & updating the likelihood function (aka Value function) in the process.
  
  Is this true? And to what degree in general have Bayesian methods been used in RL?
  For example have people tried using one of the various specialized MCMC algorithms to improve Monte-Carlo tree search?
  
  Sorry if the question is open-ended/vague.
  
  **Tags:** reinforcement-learning, comparison, monte-carlo-tree-search, monte-carlo-methods, markov-chain-monte-carlo
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-06-28. Score: 1
  
  There is an element in MCTS, in which you select to explore "promising" directions. You also give a chance for branches that were not explored enough yet. So, you can use for that Bayesian methods (to keep and maintain the distribution of the expected rewards).
  
  > **Comments:**
  > **username_0**: I'm curious if you know if anyone has tried using MCMC for this?  
  > **username_1**: AlphaGo as far as I can tell at this moment, is basically MCTS + maintaining and updating a value function and using it for making decision where to spend the next iterations. The vanilla MCTS makes use of Upper Confidence Bound (UCB), which is a handy / simple surrogate for a distribution.  
  
  ---
  
  