Answers.Body: <p>Both players are represented by the exact same network with the exact same weights(similar to AplhaGO, AlphaGoZero and AlphaZero). So, they will both behave identical. Because you only have a single network, MuZero can not learn two different policies, but only one.</p>
  <p>You can also think of this in the following way: MuZero actually learn to play only with white(or black, but just one of them) without knowing to play with the other color (at least multiple implementations of the previous algorithms like AlphaGo Zero and AlphaZero that are similar to MuZero are doing exactly this). So, in order to trick it to also be able to play with the other color, when the network need to play with black, you just flip the colors on the table so that black becomes white(and white becomes black) and the network knows what to do. After choosing the move, you flip the whole thing back and that is usually how it is done. So, from the perspective of your network, it will always play white, but because you do the flipping of the colors you can actually put them to play against each other without them knowing that.</p>
  <p>Even without using this trick of flipping the color of the table, by doing the MCTS simulations, you will have for each state the statistics for the actions, and usually as you do more simulations, this statistics show you which actions are the best in each state. And when you train, you try to imitate this. So, your network will learn in each state which actions are the best, and this is the reason why it learns to take the best possible action in each state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-31T21:43:28.517Z
Answers.Text: Okay, but the agent is given information about whose turn it is, so the policy could still end up being turn-dependent. And in some sense it should be: some games like chess are asymmetric, so it doesn't make much sense to learn to play only from one point of view with the color flipping trick.
  I don't understand where the "best" actions are defined as best. In the policy loss they use the cross-entropy between the MCTS policy and the network policy, so maybe it comes from how the predicted values are backed up in the MCTS, but I'm not sure.
Answers.UserDisplayName: Ziofil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-31T22:29:08.253Z
Answers.Text: You are doing max over the actions. So, the network will always choose the best action. Moreover, past papers related to MuZero train the prediction on outcomes of matches, so the agents learn what will actually happen, and they can not make up the result, as they learn from experience. And the experience teaches them that some actions are good and some are bad, and given the way you select the actions using max, you will always choose the best action, given the amount of experience and knowledge the network has. So, the predicted outcome is actually learned and they can not cheat
Answers.UserDisplayName: Otniel-Bogdan Mercea
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-31T22:35:24.180Z
Answers.Text: if one of the networks would learn to play badly, that means that it will learn to choose wrong actions, so that the max action would actually be the worst possible action. But that would mean that the loss between the true outcomes and the predicted outcomes will be big(as you need the outcome in order to choose which actions are the best in MCTS), so the optimizer would try to minimize that, and in the end you will reach the situation where the optimal solution is to predict a result which is as close as possible to reality.
Answers.UserDisplayName: Otniel-Bogdan Mercea
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-31T23:27:29.217Z
Answers.Text: Well, yes I agree that in the MCTS you are doing argmax over the policy, but this alone is not a guarantee to get the best action: it just selects the action with highest "probability". In order for it to be the actual best action, we need the policy to be the probability of best action, and hence the question: how is the policy trained to be such?
Answers.UserDisplayName: Ziofil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-01T11:32:25.320Z
Answers.Text: Because you predict the true value(as said above), in the MCTS after many simulations, the action statistics depend on these predicted values. So if the predicted values are correct(and as said above they are because this is the only way of minimizing the loss), then the statistics of the actions will also reflect the true outcome of taking an action, so the highest probability will reflect the action that is providing the best outcome, because this is how MCTS works. You actually use the predicted values in computing the probability of each action in MCTS.
Answers.UserDisplayName: Otniel-Bogdan Mercea
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-03T10:34:58.133Z
Answers.Text: Perhaps I'm not able to explain clearly what my problem is... One thing is the value, another thing is the policy. They are both part of the cost function, but it's not clear to me how to correctly include the policy: if one side wins and one side loses, why should I reinforce on the moves that lead to a loss? I can still include the prediction of the value, but it doesn't make sense to learn from the policies of both sides, give that one side loses the game.
Answers.UserDisplayName: Ziofil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-31T20:49:09.310Z
Answers.LastActivityDate: 2020-10-31T20:49:09.310Z
Answers.UserDisplayName: Otniel-Bogdan Mercea
Body: <p>I'm coding my own version of MuZero. However, I don't understand how it supposed to learn to play well for both players in a two-player game.</p>
  <p>Take Go for example. If I use a single MCTS to generate an entire game (to be used in the training stage), couldn't MuZero learn to play badly for black in order to become good at predicting a win for white? What is forcing it to play well at every turn?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-31T18:51:07.687Z
Comments.Text: Is is that the policies and values generated by the MCTS and the model are added to the loss for each "player" independently? (although in the paper it doesn't seem that it is like this)
Comments.UserDisplayName: Ziofil
Tags: <reinforcement-learning><self-play><muzero>
Title: How does MuZero learn to play well for both sides of a two-player game?
UserDisplayName: Ziofil
ThreadText: # How does MuZero learn to play well for both sides of a two-player game?
  
  (Asked by: username_0 on 2020-10-31)
  
  I'm coding my own version of MuZero. However, I don't understand how it supposed to learn to play well for both players in a two-player game.
  
  Take Go for example. If I use a single MCTS to generate an entire game (to be used in the training stage), couldn't MuZero learn to play badly for black in order to become good at predicting a win for white? What is forcing it to play well at every turn?
  
  **Tags:** reinforcement-learning, self-play, muzero
  > **Comments:**
  > **username_0**: Is is that the policies and values generated by the MCTS and the model are added to the loss for each "player" independently? (although in the paper it doesn't seem that it is like this)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-10-31. Score: 1
  
  Both players are represented by the exact same network with the exact same weights(similar to AplhaGO, AlphaGoZero and AlphaZero). So, they will both behave identical. Because you only have a single network, MuZero can not learn two different policies, but only one.
  
  You can also think of this in the following way: MuZero actually learn to play only with white(or black, but just one of them) without knowing to play with the other color (at least multiple implementations of the previous algorithms like AlphaGo Zero and AlphaZero that are similar to MuZero are doing exactly this). So, in order to trick it to also be able to play with the other color, when the network need to play with black, you just flip the colors on the table so that black becomes white(and white becomes black) and the network knows what to do. After choosing the move, you flip the whole thing back and that is usually how it is done. So, from the perspective of your network, it will always play white, but because you do the flipping of the colors you can actually put them to play against each other without them knowing that.
  
  Even without using this trick of flipping the color of the table, by doing the MCTS simulations, you will have for each state the statistics for the actions, and usually as you do more simulations, this statistics show you which actions are the best in each state. And when you train, you try to imitate this. So, your network will learn in each state which actions are the best, and this is the reason why it learns to take the best possible action in each state.
  
  > **Comments:**
  > **username_0**: Okay, but the agent is given information about whose turn it is, so the policy could still end up being turn-dependent. And in some sense it should be: some games like chess are asymmetric, so it doesn't make much sense to learn to play only from one point of view with the color flipping trick. I don't understand where the "best" actions are defined as best. In the policy loss they use the cross-entropy between the MCTS policy and the network policy, so maybe it comes from how the predicted values are backed up in the MCTS, but I'm not sure.  
  > **username_1**: You are doing max over the actions. So, the network will always choose the best action. Moreover, past papers related to MuZero train the prediction on outcomes of matches, so the agents learn what will actually happen, and they can not make up the result, as they learn from experience. And the experience teaches them that some actions are good and some are bad, and given the way you select the actions using max, you will always choose the best action, given the amount of experience and knowledge the network has. So, the predicted outcome is actually learned and they can not cheat  
  > **username_1**: if one of the networks would learn to play badly, that means that it will learn to choose wrong actions, so that the max action would actually be the worst possible action. But that would mean that the loss between the true outcomes and the predicted outcomes will be big(as you need the outcome in order to choose which actions are the best in MCTS), so the optimizer would try to minimize that, and in the end you will reach the situation where the optimal solution is to predict a result which is as close as possible to reality.  
  > **username_0**: Well, yes I agree that in the MCTS you are doing argmax over the policy, but this alone is not a guarantee to get the best action: it just selects the action with highest "probability". In order for it to be the actual best action, we need the policy to be the probability of best action, and hence the question: how is the policy trained to be such?  
  > **username_1**: Because you predict the true value(as said above), in the MCTS after many simulations, the action statistics depend on these predicted values. So if the predicted values are correct(and as said above they are because this is the only way of minimizing the loss), then the statistics of the actions will also reflect the true outcome of taking an action, so the highest probability will reflect the action that is providing the best outcome, because this is how MCTS works. You actually use the predicted values in computing the probability of each action in MCTS.  
  > **username_0**: Perhaps I'm not able to explain clearly what my problem is... One thing is the value, another thing is the policy. They are both part of the cost function, but it's not clear to me how to correctly include the policy: if one side wins and one side loses, why should I reinforce on the moves that lead to a loss? I can still include the prediction of the value, but it doesn't make sense to learn from the policies of both sides, give that one side loses the game.  
  
  ---
  
  