Body: <p>If I understood correctly, the AlphaGo Zero network returns two values: a vector of logit probabilities <strong>p</strong> and a value <strong>v</strong>.</p>
  
  <p>My question is: in this vector that it is outputted, do we have a probability for every possible action in the game? If so: does it apply a probability of 0 to actions that are not possible in that particular state? If this is true, how does the network know which actions are valid?</p>
  
  <p>If not: then the network will output vectors of different sizes according to each state. Is this even feasible? And again, how will the network know which actions are valid?</p>
  
  <p>Related questions but none of them covers this question in specific: <a href="https://ai.stackexchange.com/questions/7580/is-the-discount-not-needed-in-a-deterministic-environment-for-reinforcement-lear">1</a>, <a href="https://ai.stackexchange.com/questions/10149/what-is-a-logit-probability">2</a> and <a href="https://ai.stackexchange.com/questions/12759/how-does-alphazero-use-its-value-and-policy-heads-in-conjunction">3</a>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-11T08:14:40.603Z
Comments.Text: Not a duplicate, but more directly answers your question: https://ai.stackexchange.com/questions/2980/how-to-handle-invalid-moves-in-reinforcement-learning I recall answering something that would be a duplicate, but cannot find it. Might be on Data Science instead, perhaps . . . anyway, short anser is yes that the NN has every move encoded, and no the NN it never learns "not to play" invalid moves, instead validity is something that the environment is expected to supply as a callable function that the agent can use to filter choices.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-11T14:15:50.343Z
Comments.Text: Thanks for the link @NeilSlater, it is very helpful! One last question, to see if I understood correctly: so the neural network will return probabilities for all possible actions. After I get the vector from the network, I will have to deal with impossible moves and distribute the probabilities assigned to them to all other valid moves (proportionally). Is that right?
Comments.UserDisplayName: ihavenoidea
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-11T15:01:22.807Z
Comments.Text: That is one way to do it, yes. If you still need details for how to handle it in your case, then keep the question open because I could not find a full duplicate, so perhaps someone will answer here.
Comments.UserDisplayName: Neil Slater
Tags: <deep-learning><policies><deepmind><alphago-zero>
Title: AlphaGo Zero: Does the policy head give a probability for every possible move?
UserDisplayName: ihavenoidea
ThreadText: # AlphaGo Zero: Does the policy head give a probability for every possible move?
  
  (Asked by: username_0 on 2019-11-11)
  
  If I understood correctly, the AlphaGo Zero network returns two values: a vector of logit probabilities **p** and a value **v**.
  
  My question is: in this vector that it is outputted, do we have a probability for every possible action in the game? If so: does it apply a probability of 0 to actions that are not possible in that particular state? If this is true, how does the network know which actions are valid?
  
  If not: then the network will output vectors of different sizes according to each state. Is this even feasible? And again, how will the network know which actions are valid?
  
  Related questions but none of them covers this question in specific: [1](https://ai.stackexchange.com/questions/7580/is-the-discount-not-needed-in-a-deterministic-environment-for-reinforcement-lear), [2](https://ai.stackexchange.com/questions/10149/what-is-a-logit-probability) and [3](https://ai.stackexchange.com/questions/12759/how-does-alphazero-use-its-value-and-policy-heads-in-conjunction).
  
  **Tags:** deep-learning, policies, deepmind, alphago-zero
  
  > **Comments:**
  > **username_1**: Not a duplicate, but more directly answers your question: https://ai.stackexchange.com/questions/2980/how-to-handle-invalid-moves-in-reinforcement-learning I recall answering something that would be a duplicate, but cannot find it. Might be on Data Science instead, perhaps . . . anyway, short anser is yes that the NN has every move encoded, and no the NN it never learns "not to play" invalid moves, instead validity is something that the environment is expected to supply as a callable function that the agent can use to filter choices.  
  > **username_0**: Thanks for the link @username_1, it is very helpful! One last question, to see if I understood correctly: so the neural network will return probabilities for all possible actions. After I get the vector from the network, I will have to deal with impossible moves and distribute the probabilities assigned to them to all other valid moves (proportionally). Is that right?  
  > **username_1**: That is one way to do it, yes. If you still need details for how to handle it in your case, then keep the question open because I could not find a full duplicate, so perhaps someone will answer here.  
  
  ---
  
  ## 0 answers
  
  