Answers.Body: <p>The hard to work around traits of Monte Carlo Tree Search (MCTS) are:</p>
  <ul>
  <li><p>It is model-based, and due to using a long look-ahead distance it can be sensitive to errors in the model (for learned models).</p>
  </li>
  <li><p>It requires compute resources at the time of making a decision. In some domains this is an advantage (the compute resource is efficiently applied to a required decision), in others it is a liability where it may be hard to see much benefit on a real time system.</p>
  </li>
  </ul>
  <p>The first issue is strongly limiting - you need a good predictive model of the environment, and this is often not available. Even simple physical systems are hard to model accurately a significant amount into the future.</p>
  <p>The second issue might be worked around by e.g. only using MCTS as a training assistant to develop a policy function - as MCTS evaluates and chooses an action more accurately than the current value and policy functions, it can be useful as updates to those functions (this is basically how AlphaGo training works). You would expect that policy function to perform worse without MCTS for look ahead planning than during training, but it could still be an effective agent.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-04T19:44:08.233Z
Answers.Text: The last paragraph is not clear. Could you please better explain it?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-05T07:25:36.237Z
Answers.Text: @DSPinfinity I added an explanation of how such training works
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-04T07:40:07.220Z
Answers.LastActivityDate: 2024-06-05T07:24:37.690Z
Answers.LastEditDate: 2024-06-05T07:24:37.690Z
Answers.UserDisplayName: Neil Slater
Body: <p>What are disadvantages/limitations of Monte Carlo Tree Search in RL, and hence for what kind of applications might its use not be appropriate?</p>
  
Tags: <reinforcement-learning><applications><monte-carlo-tree-search>
Title: What are disadvantages/limitations of Monte Carlo Tree Search in RL?
UserDisplayName: DSPinfinity
ThreadText: # What are disadvantages/limitations of Monte Carlo Tree Search in RL?
  
  (Asked by: username_0 on 2024-06-04)
  
  What are disadvantages/limitations of Monte Carlo Tree Search in RL, and hence for what kind of applications might its use not be appropriate?
  
  **Tags:** reinforcement-learning, applications, monte-carlo-tree-search
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-06-04. Score: 1 | Accepted answer
  
  The hard to work around traits of Monte Carlo Tree Search (MCTS) are:
  
  * It is model-based, and due to using a long look-ahead distance it can be sensitive to errors in the model (for learned models).
  * It requires compute resources at the time of making a decision. In some domains this is an advantage (the compute resource is efficiently applied to a required decision), in others it is a liability where it may be hard to see much benefit on a real time system.
  
  The first issue is strongly limiting - you need a good predictive model of the environment, and this is often not available. Even simple physical systems are hard to model accurately a significant amount into the future.
  
  The second issue might be worked around by e.g. only using MCTS as a training assistant to develop a policy function - as MCTS evaluates and chooses an action more accurately than the current value and policy functions, it can be useful as updates to those functions (this is basically how AlphaGo training works). You would expect that policy function to perform worse without MCTS for look ahead planning than during training, but it could still be an effective agent.
  
  > **Comments:**
  > **username_0**: The last paragraph is not clear. Could you please better explain it?  
  > **username_1**: @username_0 I added an explanation of how such training works  
  
  ---
  
  