Body: <p>Say i want to train a neural network with 10 classes as outputs and use <code>categorical_cross_entropy</code> as a loss function in keras. This will try
  to fit the training data as best as possible irregardless of the outcome (i.e. value). If I want to take value into account, I have to use something like a policy gradient RL algorithm. How do I formulate the loss of policy gradient algorithm in this case ?</p>
  
  <p>The standard categorical cross entropy loss function is as follows where y_ = true value, and y = predicted value:</p>
  
  <pre><code>   loss = -mean( y_ * log(y))
  </code></pre>
  
  <p>I am thinking to just multiply the true value by the reward and still use the <code>categorical cross entropy</code> of keras i.e.</p>
  
  <pre><code>   y_ = y_ * reward
     loss = -mean( y_* log(y) )
  </code></pre>
  
  <p>Is my interpretation correct ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-20T20:54:25.100Z
Comments.Text: Do you actually have a RL problem to solve? It is not clear whether you are trying to train a policy network to solve an environment, or trying to apply concepts from reinforcement learning on something else. If you could [edit] your question to include some context about the problem you are trying to solve, it may clarify what approach would be best.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-20T21:01:01.147Z
Comments.Text: I am using this for playing a game using combined policy+value neural network just like alphago zero. For supervised training from human games, cross entropy loss is sufficient. But for self-play training I want to use the policy gradient algorithm as described in my post, but I am not sure how to compute the loss function for it.
Comments.UserDisplayName: danny
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-20T21:03:58.353Z
Comments.Text: OK that makes sense. What are you intending to use for the "true value" in the context of your question? Are you using MCTS search to establish a refined policy during self-play (like AlphaZero)? Have you read and are trying to replicate the AlphaZero paper? I think that the loss functions are included in that paper . . . at least I have read a few replications which appear to quote the loss functions.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-20T21:09:43.340Z
Comments.Text: The 800 playouts mcts training is too expensive for me, so i am thinking to try out a 1 playout training instead -- i.e. use policy gradient method. They actually did that in their first paper for RL training of the policy network. I think with 800 playouts mcts, fitting the policy network even to the loosing player's moves is Ok to do.
Comments.UserDisplayName: danny
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-20T21:14:56.867Z
Comments.Text: "I think with 800 playouts mcts, fitting the policy network even to the loosing player's moves is Ok to do" Yes I think that is correct - the MCTS search is directly improving the policy, and one of the loss functions is loss between MCTS-discovered improved policy and existing policy. I'm not 100% clear on what you are doing - are you intending to run *any* look-ahead planning in your game, or run something more like Actor-Critic where the agent learns by doing *without* planning (this should still be OK and does have a loss function - in fact a few variations)
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><keras><policy-gradients><reinforce>
Title: Policy gradient loss for neural network training
UserDisplayName: danny
ThreadText: # Policy gradient loss for neural network training
  
  (Asked by: username_0 on 2019-04-20)
  
  Say i want to train a neural network with 10 classes as outputs and use categorical_cross_entropy as a loss function in keras. This will try
  to fit the training data as best as possible irregardless of the outcome (i.e. value). If I want to take value into account, I have to use something like a policy gradient RL algorithm. How do I formulate the loss of policy gradient algorithm in this case ?
  The standard categorical cross entropy loss function is as follows where y_ = true value, and y = predicted value:
     loss = -mean( y_ * log(y))
  
  I am thinking to just multiply the true value by the reward and still use the categorical cross entropy of keras i.e.
     y_ = y_ * reward
     loss = -mean( y_* log(y) )
  
  Is my interpretation correct ?
  
  **Tags:** reinforcement-learning, keras, policy-gradients, reinforce
  > **Comments:**
  > **username_1**: Do you actually have a RL problem to solve? It is not clear whether you are trying to train a policy network to solve an environment, or trying to apply concepts from reinforcement learning on something else. If you could [edit] your question to include some context about the problem you are trying to solve, it may clarify what approach would be best.  
  > **username_0**: I am using this for playing a game using combined policy+value neural network just like alphago zero. For supervised training from human games, cross entropy loss is sufficient. But for self-play training I want to use the policy gradient algorithm as described in my post, but I am not sure how to compute the loss function for it.  
  > **username_1**: OK that makes sense. What are you intending to use for the "true value" in the context of your question? Are you using MCTS search to establish a refined policy during self-play (like AlphaZero)? Have you read and are trying to replicate the AlphaZero paper? I think that the loss functions are included in that paper . . . at least I have read a few replications which appear to quote the loss functions.  
  > **username_0**: The 800 playouts mcts training is too expensive for me, so i am thinking to try out a 1 playout training instead -- i.e. use policy gradient method. They actually did that in their first paper for RL training of the policy network. I think with 800 playouts mcts, fitting the policy network even to the loosing player's moves is Ok to do.  
  > **username_1**: "I think with 800 playouts mcts, fitting the policy network even to the loosing player's moves is Ok to do" Yes I think that is correct - the MCTS search is directly improving the policy, and one of the loss functions is loss between MCTS-discovered improved policy and existing policy. I'm not 100% clear on what you are doing - are you intending to run *any* look-ahead planning in your game, or run something more like Actor-Critic where the agent learns by doing *without* planning (this should still be OK and does have a loss function - in fact a few variations)  
  
  ---
  
  ## 0 answers
  
  