Answers.Body: <blockquote>
    <p>Is this the only big difference? Or in other words, can I do the same implementation as in a simple MCTS with the 4 stages: selection, expansion, simulation and backpropagation, where the result of the simulation is the accumulated reward instead of a value between 0 and 1? How would the UCT selection be adjusted in this case?</p>
  </blockquote>
  
  <p>No, this is not the only difference. The important differences between the "original UCT" (as described in the paper you linked), and the "standard UCT" (as typically implemented in game AI research) are nicely summarised in Subsection 2.4 of <a href="https://www.jair.org/index.php/jair/article/view/11099/26289" rel="nofollow noreferrer">"On Monte Carlo Tree Search and Reinforcement Learning"</a>. I'd recommend reading the entire paper, or large parts of it, if you have the time, but to summarise the key differences:</p>
  
  <ul>
  <li>Original stores as many nodes as allowed by memory limitations per iteration, standard only expands one node per iteration.</li>
  <li>Original uses transpositions, standard implementation does not</li>
  <li>Original uses discount factor <span class="math-container">$\gamma$</span>, standard does not (equivalent to picking <span class="math-container">$\gamma = 1$</span>)</li>
  <li>Original takes into account at which point in time rewards were observed, standard does not</li>
  </ul>
  
  <p>That last point is the one your question mainly seems to be about. Your idea of <strong>using a standard implementation, but accumulating any intermediate rewards and treating them as one large terminal reward, would not be the same</strong>. Actually taking into account the times at which rewards were observed, as done in the original UCT, can lead to more accurate value estimates in nodes in the middle of the tree (or at least to finding them more quickly). Your idea would be equivalent to learning from Monte-Carlo backups in Reinforcement Learning literature (e.g. Sutton and Barto's book), whereas the original UCT implementation would be more similar to learning from TD(<span class="math-container">$0$</span>) backups.</p>
  
  <p>Note that, when talking about the "standard" implementation of UCT, that's very often in papers about two-player zero-sum games in which all intermediate rewards have a value of <span class="math-container">$0$</span>, discounting is considered to be unimportant (i.e. <span class="math-container">$\gamma = 1$</span>), and only terminal game states have a non-zero reward. In this special case, the two ideas do become equivalent.</p>
  
  <hr>
  
  <blockquote>
    <p>What does the UpdateValue function in line 12 does exactly? In the text it says it is used to adjust the state-action pair value at a given depth, will this be used to the selection? How is this calculated exactly?</p>
  </blockquote>
  
  <p>This would be done in the "standard" way. The visit count is incremented, and <span class="math-container">$q$</span> could be added to a sum of all <span class="math-container">$q$</span> values ever observed during the search for that state-action pair, such that the average score can be computed as the sum of <span class="math-container">$q$</span> scores divided by visit count. This average score is the <span class="math-container">$\bar{X}$</span> used in Selection.</p>
  
  <hr>
  
  <blockquote>
    <p>What is the depth parameter needed for? Is it related with the UpdateValue?</p>
  </blockquote>
  
  <p>I think it's usually not really needed. In the paper they describe that it may be considered to have a depth cut-off. From paper:</p>
  
  <blockquote>
    <p>"This can be the reach of a terminal state, <strong>or episodes can be cut at a certain depth (line 8)</strong>."</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-16T16:05:13.700Z
Answers.Text: if that corresponds to the average score, how can we guarantee it will be bounded between 0 and 1? Even if each individual reward is in between those bounds we cannot say that the sum $R + \gamma R_1 +... $ will be in the same bounds right?
Answers.UserDisplayName: Miguel Saraiva
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-16T16:40:57.477Z
Answers.Text: @MiguelSaraiva In practice that may be difficult, yes. In theory, you can always guarantee it if you have a fully defined MDP with $\gamma < 1$ or with a theoretical limit on the episode durations. Then you can always find at least some theoretical upper bound on the magnitude of such a discounted sum, and map it to lie in $[0, 1]$ by dividing all rewards by that upper bound. Those $[0, 1]$ bounds are normally only really important for the theoretical proofs anyway, in practice it's fine if we violate them (though sticking to them might make hyperparam tuning easier).
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-16T15:50:54.180Z
Answers.LastActivityDate: 2019-07-16T15:50:54.180Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>In <a href="https://link.springer.com/content/pdf/10.1007%2F11871842_29.pdf" rel="nofollow noreferrer">Bandit Based Monte-Carlo Planning</a>, the article where UCT is introduced as a planning algorithm, there is an algorithm description in page 285 (4 of the pdf).</p>
  <p>Comparing this implementation of UCT (a specific type of MCTS algorithm) to the application normally used in games, there is one major difference. Here, the rewards are calculated for every state, instead of only doing an evaluation at the end of the simulation.</p>
  <p>My questions are (they are all related to each other):</p>
  <ul>
  <li><p>Is this the only big difference? Or in other words, can I do the same implementation as in MCTS for games with the 4 stages: selection, expansion, simulation and backpropagation, where the result of the simulation is the accumulated reward instead of a value between 0 and 1? How would the UCT selection be adjusted in this case?</p>
  </li>
  <li><p>What does the UpdateValue function in line 12 does exactly? In the text it says it is used to adjust the state-action pair value at a given depth, will this be used to the selection? How is this calculated exactly?</p>
  </li>
  <li><p>What is the depth parameter needed for? Is it related with the UpdateValue?</p>
  </li>
  </ul>
  <p>Finally I would like to ask if you know any other papers where a clear implementation of UCT for planning is used with multiple rewards, not only on the end of the simulation.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-16T22:48:59.267Z
Comments.Text: Hi Miguel. Please, next time ask one question per post. In the future, it unlikely that someone will have exactly the same questions, so this post will be useful only for you. Furthermore, you facilitate the life of the answerers: some people might know the answer to some question, but not all of them, so they might not give an answer to your questions because they do not know the answers to all of your questions. Does this make any sense to you? In this case, you have someone that is an expert in MCTS, etc., but this will not always be the case.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-17T09:40:44.640Z
Comments.Text: Sure. No problem
Comments.UserDisplayName: Miguel Saraiva
Tags: <reinforcement-learning><monte-carlo-tree-search><planning>
Title: Several questions related to UCT and MCTS
UserDisplayName: Miguel Saraiva
ThreadText: # Several questions related to UCT and MCTS
  
  (Asked by: username_0 on 2019-07-16)
  
  In [Bandit Based Monte-Carlo Planning](https://link.springer.com/content/pdf/10.1007%2F11871842_29.pdf), the article where UCT is introduced as a planning algorithm, there is an algorithm description in page 285 (4 of the pdf).
  
  Comparing this implementation of UCT (a specific type of MCTS algorithm) to the application normally used in games, there is one major difference. Here, the rewards are calculated for every state, instead of only doing an evaluation at the end of the simulation.
  
  My questions are (they are all related to each other):
  
  * Is this the only big difference? Or in other words, can I do the same implementation as in MCTS for games with the 4 stages: selection, expansion, simulation and backpropagation, where the result of the simulation is the accumulated reward instead of a value between 0 and 1? How would the UCT selection be adjusted in this case?
  * What does the UpdateValue function in line 12 does exactly? In the text it says it is used to adjust the state-action pair value at a given depth, will this be used to the selection? How is this calculated exactly?
  * What is the depth parameter needed for? Is it related with the UpdateValue?
  
  Finally I would like to ask if you know any other papers where a clear implementation of UCT for planning is used with multiple rewards, not only on the end of the simulation.
  
  **Tags:** reinforcement-learning, monte-carlo-tree-search, planning
  > **Comments:**
  > **username_1**: Hi Miguel. Please, next time ask one question per post. In the future, it unlikely that someone will have exactly the same questions, so this post will be useful only for you. Furthermore, you facilitate the life of the answerers: some people might know the answer to some question, but not all of them, so they might not give an answer to your questions because they do not know the answers to all of your questions. Does this make any sense to you? In this case, you have someone that is an expert in MCTS, etc., but this will not always be the case.  
  > **username_0**: Sure. No problem  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-07-16. Score: 2 | Accepted answer
  
  > Is this the only big difference? Or in other words, can I do the same implementation as in a simple MCTS with the 4 stages: selection, expansion, simulation and backpropagation, where the result of the simulation is the accumulated reward instead of a value between 0 and 1? How would the UCT selection be adjusted in this case?
  
  No, this is not the only difference. The important differences between the "original UCT" (as described in the paper you linked), and the "standard UCT" (as typically implemented in game AI research) are nicely summarised in Subsection 2.4 of ["On Monte Carlo Tree Search and Reinforcement Learning"](https://www.jair.org/index.php/jair/article/view/11099/26289). I'd recommend reading the entire paper, or large parts of it, if you have the time, but to summarise the key differences:
  
  * Original stores as many nodes as allowed by memory limitations per iteration, standard only expands one node per iteration.
  * Original uses transpositions, standard implementation does not
  * Original uses discount factor $\gamma$, standard does not (equivalent to picking $\gamma = 1$)
  * Original takes into account at which point in time rewards were observed, standard does not
  
  That last point is the one your question mainly seems to be about. Your idea of **using a standard implementation, but accumulating any intermediate rewards and treating them as one large terminal reward, would not be the same**. Actually taking into account the times at which rewards were observed, as done in the original UCT, can lead to more accurate value estimates in nodes in the middle of the tree (or at least to finding them more quickly). Your idea would be equivalent to learning from Monte-Carlo backups in Reinforcement Learning literature (e.g. Sutton and Barto's book), whereas the original UCT implementation would be more similar to learning from TD($0$) backups.
  
  Note that, when talking about the "standard" implementation of UCT, that's very often in papers about two-player zero-sum games in which all intermediate rewards have a value of $0$, discounting is considered to be unimportant (i.e. $\gamma = 1$), and only terminal game states have a non-zero reward. In this special case, the two ideas do become equivalent.
  
  ---
  
  > What does the UpdateValue function in line 12 does exactly? In the text it says it is used to adjust the state-action pair value at a given depth, will this be used to the selection? How is this calculated exactly?
  
  This would be done in the "standard" way. The visit count is incremented, and $q$ could be added to a sum of all $q$ values ever observed during the search for that state-action pair, such that the average score can be computed as the sum of $q$ scores divided by visit count. This average score is the $\bar{X}$ used in Selection.
  
  ---
  
  > What is the depth parameter needed for? Is it related with the UpdateValue?
  
  I think it's usually not really needed. In the paper they describe that it may be considered to have a depth cut-off. From paper:
  
  > "This can be the reach of a terminal state, **or episodes can be cut at a certain depth (line 8)**."
  
  > **Comments:**
  > **username_0**: if that corresponds to the average score, how can we guarantee it will be bounded between 0 and 1? Even if each individual reward is in between those bounds we cannot say that the sum $R + \gamma R_1 +... $ will be in the same bounds right?  
  > **username_2**: @MiguelSaraiva In practice that may be difficult, yes. In theory, you can always guarantee it if you have a fully defined MDP with $\gamma < 1$ or with a theoretical limit on the episode durations. Then you can always find at least some theoretical upper bound on the magnitude of such a discounted sum, and map it to lie in $[0, 1]$ by dividing all rewards by that upper bound. Those $[0, 1]$ bounds are normally only really important for the theoretical proofs anyway, in practice it's fine if we violate them (though sticking to them might make hyperparam tuning easier).  
  
  ---
  
  