Body: <p>I am a little confused as to how the target values are generated to train the neural network with the Alpha Zero architecture(in specific to a chess game). I understand how the improved policy is generated from the number visits to each child node. In a lot of different sources they say that a training sample is generated by keeping track of (state, policy,_) where _ is the eventual win or loss that occurs in the game. This would mean that if white eventually won the game, every single state that white had in the Monte Carlo Tree would receive a value of +1. Isn't this very counter intuitive since there were clearly suboptimal moves that white chose not to take(for example moving a queen out into direct danger to be captured) but these states will still be marked with a +1. Wouldn't it make more sense to just label the state/moves that directly led to the win(traversing up the tree) instead of marking every considered state based on the outcome of one path.</p>
  <p>Another question I had was instead of training the values of every state on the eventual win or loss +1/-1, would it not make more sense to train it against the final Wins/Visits value that was calculated when conducting simulations on the tree. Wouldn't Wins/Visits be a much more representative for the average value of the state instead of declaring it to be completely a +1 or -1 since there are probably particular subbranches that could result in a win and some that could result in a loss. Sure the target values in the beginning of training would be very inaccurate, but as the network gets more experience with states right before the end of the game, the accuracy of the value predictions would improve upwards the tree.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-06T01:55:18.557Z
Comments.Text: I think you are mistaken in how the target value (or value function) is computed. For example, the statement "if white eventually won the game, every single state that white had in the Monte Carlo Tree would receive a value of +1" is imprecise. Not every state, but only those nodes on that branch is +1.
Comments.UserDisplayName: lpounng
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-06T02:49:04.213Z
Comments.Text: I suggest starting from the [earlier paper of AlphaGo](https://www.researchgate.net/publication/292074166_Mastering_the_game_of_Go_with_deep_neural_networks_and_tree_search)  which contains more details on how the target value etc. are computed. Later papers assume readers are already familiar with the topic and skip much of the details, leading to confusion.
Comments.UserDisplayName: lpounng
Tags: <reinforcement-learning><monte-carlo-tree-search><actor-critic-methods><value-functions><alphazero>
Title: How are Target Values Generated in Alpha Zero Architecture
UserDisplayName: Kiran Manicka
ThreadText: # How are Target Values Generated in Alpha Zero Architecture
  
  (Asked by: username_0 on 2023-07-03)
  
  I am a little confused as to how the target values are generated to train the neural network with the Alpha Zero architecture(in specific to a chess game). I understand how the improved policy is generated from the number visits to each child node. In a lot of different sources they say that a training sample is generated by keeping track of (state, policy,\_) where \_ is the eventual win or loss that occurs in the game. This would mean that if white eventually won the game, every single state that white had in the Monte Carlo Tree would receive a value of +1. Isn't this very counter intuitive since there were clearly suboptimal moves that white chose not to take(for example moving a queen out into direct danger to be captured) but these states will still be marked with a +1. Wouldn't it make more sense to just label the state/moves that directly led to the win(traversing up the tree) instead of marking every considered state based on the outcome of one path.
  
  Another question I had was instead of training the values of every state on the eventual win or loss +1/-1, would it not make more sense to train it against the final Wins/Visits value that was calculated when conducting simulations on the tree. Wouldn't Wins/Visits be a much more representative for the average value of the state instead of declaring it to be completely a +1 or -1 since there are probably particular subbranches that could result in a win and some that could result in a loss. Sure the target values in the beginning of training would be very inaccurate, but as the network gets more experience with states right before the end of the game, the accuracy of the value predictions would improve upwards the tree.
  
  **Tags:** reinforcement-learning, monte-carlo-tree-search, actor-critic-methods, value-functions, alphazero
  
  > **Comments:**
  > **username_1**: I think you are mistaken in how the target value (or value function) is computed. For example, the statement "if white eventually won the game, every single state that white had in the Monte Carlo Tree would receive a value of +1" is imprecise. Not every state, but only those nodes on that branch is +1.  
  > **username_1**: I suggest starting from the [earlier paper of AlphaGo](https://www.researchgate.net/publication/292074166_Mastering_the_game_of_Go_with_deep_neural_networks_and_tree_search)  which contains more details on how the target value etc. are computed. Later papers assume readers are already familiar with the topic and skip much of the details, leading to confusion.  
  
  ---
  
  ## 0 answers
  
  