<p>Say i want to train a neural network with 10 classes as outputs and use <code>categorical_cross_entropy</code> as a loss function in keras. This will try
to fit the training data as best as possible irregardless of the outcome (i.e. value). If I want to take value into account, I have to use something like a policy gradient RL algorithm. How do I formulate the loss of policy gradient algorithm in this case ?</p>

<p>The standard categorical cross entropy loss function is as follows where y_ = true value, and y = predicted value:</p>

<pre><code>   loss = -mean( y_ * log(y))
</code></pre>

<p>I am thinking to just multiply the true value by the reward and still use the <code>categorical cross entropy</code> of keras i.e.</p>

<pre><code>   y_ = y_ * reward
   loss = -mean( y_* log(y) )
</code></pre>

<p>Is my interpretation correct ?</p>

CC BY-SA 4.0
2019-04-20T20:54:25.100Z
Do you actually have a RL problem to solve? It is not clear whether you are trying to train a policy network to solve an environment, or trying to apply concepts from reinforcement learning on something else. If you could [edit] your question to include some context about the problem you are trying to solve, it may clarify what approach would be best.
Neil Slater
CC BY-SA 4.0
2019-04-20T21:01:01.147Z
I am using this for playing a game using combined policy+value neural network just like alphago zero. For supervised training from human games, cross entropy loss is sufficient. But for self-play training I want to use the policy gradient algorithm as described in my post, but I am not sure how to compute the loss function for it.
danny
CC BY-SA 4.0
2019-04-20T21:03:58.353Z
OK that makes sense. What are you intending to use for the "true value" in the context of your question? Are you using MCTS search to establish a refined policy during self-play (like AlphaZero)? Have you read and are trying to replicate the AlphaZero paper? I think that the loss functions are included in that paper . . . at least I have read a few replications which appear to quote the loss functions.
Neil Slater
CC BY-SA 4.0
2019-04-20T21:09:43.340Z
The 800 playouts mcts training is too expensive for me, so i am thinking to try out a 1 playout training instead -- i.e. use policy gradient method. They actually did that in their first paper for RL training of the policy network. I think with 800 playouts mcts, fitting the policy network even to the loosing player's moves is Ok to do.
danny
CC BY-SA 4.0
2019-04-20T21:14:56.867Z
"I think with 800 playouts mcts, fitting the policy network even to the loosing player's moves is Ok to do" Yes I think that is correct - the MCTS search is directly improving the policy, and one of the loss functions is loss between MCTS-discovered improved policy and existing policy. I'm not 100% clear on what you are doing - are you intending to run *any* look-ahead planning in your game, or run something more like Actor-Critic where the agent learns by doing *without* planning (this should still be OK and does have a loss function - in fact a few variations)
Neil Slater
<reinforcement-learning><keras><policy-gradients><reinforce>
Policy gradient loss for neural network training
danny
# Policy gradient loss for neural network training

(Asked by: username_0 on 2019-04-20)

Say i want to train a neural network with 10 classes as outputs and use categorical_cross_entropy as a loss function in keras. This will try
to fit the training data as best as possible irregardless of the outcome (i.e. value). If I want to take value into account, I have to use something like a policy gradient RL algorithm. How do I formulate the loss of policy gradient algorithm in this case ?
The standard categorical cross entropy loss function is as follows where y_ = true value, and y = predicted value:
   loss = -mean( y_ * log(y))

I am thinking to just multiply the true value by the reward and still use the categorical cross entropy of keras i.e.
   y_ = y_ * reward
   loss = -mean( y_* log(y) )

Is my interpretation correct ?

**Tags:** reinforcement-learning, keras, policy-gradients, reinforce

> **Comments:**
> **username_1**: Do you actually have a RL problem to solve? It is not clear whether you are trying to train a policy network to solve an environment, or trying to apply concepts from reinforcement learning on something else. If you could [edit] your question to include some context about the problem you are trying to solve, it may clarify what approach would be best.  
> **username_0**: I am using this for playing a game using combined policy+value neural network just like alphago zero. For supervised training from human games, cross entropy loss is sufficient. But for self-play training I want to use the policy gradient algorithm as described in my post, but I am not sure how to compute the loss function for it.  
> **username_1**: OK that makes sense. What are you intending to use for the "true value" in the context of your question? Are you using MCTS search to establish a refined policy during self-play (like AlphaZero)? Have you read and are trying to replicate the AlphaZero paper? I think that the loss functions are included in that paper . . . at least I have read a few replications which appear to quote the loss functions.  
> **username_0**: The 800 playouts mcts training is too expensive for me, so i am thinking to try out a 1 playout training instead -- i.e. use policy gradient method. They actually did that in their first paper for RL training of the policy network. I think with 800 playouts mcts, fitting the policy network even to the loosing player's moves is Ok to do.  
> **username_1**: "I think with 800 playouts mcts, fitting the policy network even to the loosing player's moves is Ok to do" Yes I think that is correct - the MCTS search is directly improving the policy, and one of the loss functions is loss between MCTS-discovered improved policy and existing policy. I'm not 100% clear on what you are doing - are you intending to run *any* look-ahead planning in your game, or run something more like Actor-Critic where the agent learns by doing *without* planning (this should still be OK and does have a loss function - in fact a few variations)  

---

## 0 answers

