Answers.Body: <p>The loss function you are looking for is <a href="https://en.wikipedia.org/wiki/Cross_entropy" rel="nofollow noreferrer">cross entropy loss</a>. The 'label' that you use is the action you took at the time point you are updating for.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-18T22:06:10.680Z
Answers.Text: Okay, thank you. It was not working with my code so I had to build a new, simpler one to test it and it worked. That’s why it took so long. There’s just one more thing... How do I know to do this given the equation in my question? Or is this not what the equation meant?
Answers.UserDisplayName: S2673
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-19T23:07:42.623Z
Answers.Text: Should have clarified in my answer that I was referring to the [cross entropy loss in pytorch](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) — if you read the documentation you see that this loss function is the log of the policy for the action you give it.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-25T01:20:34.503Z
Answers.Text: Thanks. Do you, by any chance, know how this works for a policy gradient with a continuous action space where the neural network outputs a Gaussian distribution? The equation for this update is the same but it is the log of a different output.
Answers.UserDisplayName: S2673
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-07T11:22:28.387Z
Answers.Text: @S2673 The algorithm doesn't change in this situation. Say your NN outputs the mean parameter of the Gaussian, then $\log_\pi(a_t | s_t)$ is just the log of the normal density evaluated at the action you took where the mean parameter in the density is the output of your NN. You are then able to backpropagate through this to update the weights of your network.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-08T03:24:32.683Z
Answers.Text: Thank you. I had figured this out a bit ago.
Answers.UserDisplayName: S2673
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-08T10:34:29.477Z
Answers.Text: @S2673 Apologies, I forgot to reply sooner
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-16T17:23:18.817Z
Answers.LastActivityDate: 2020-09-16T17:23:18.817Z
Answers.UserDisplayName: David
Body: <p>I understand that this is the update for the parameters of a policy in REINFORCE:</p>
  <p><span class="math-container">$$
  \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t},
  $$</span>
  where <span class="math-container">$v_t$</span>  is usually the discounted future reward and  <span class="math-container">$\pi_{\theta}\left(a_{t} \mid s_{t}\right)$</span> is the probability of taken the action that the agent took at time <span class="math-container">$t$</span>. (Tell me if something is wrong here)</p>
  <p>However, I don't understand how to implement this with a neural network.</p>
  <p>Let's say that <code>probs = policy.feedforward(state)</code> returns the probabilities of taking each action, like <code>[0.6, 0.4]</code>. <code>action = choose_action_from(probs)</code> will return the index of the probability chosen. For example, if it chose 0.6, the action would be 0.</p>
  <p>When it is time to update the parameters of the policy network, what should we do? Should we do something like the following?</p>
  <pre><code>gradient = policy.backpropagate(total_discounted_reward*log(probs[action])
  policy.weights += gradient
  </code></pre>
  <p>And I only backpropagate this through one output neuron?</p>
  <p>Which loss function should I use in this case? What would the labels be?</p>
  <p>If you need more explanation, I have this <a href="https://stackoverflow.com/q/63602222/14170431">question</a> on SO.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-16T16:31:21.140Z
Comments.Text: I use a cross entropy loss when I implement this in PyTorch. I tried doing a forward pass of the network evaluated at the current action and taking the log but it didn't seem to work when I implemented it, so I stuck with cross entropy loss.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-16T16:42:30.397Z
Comments.Text: Is this question related to a specific library (eg. pytorch or tensorflow)? Or is this just a question for a more specific but still general pseudocode?
Comments.UserDisplayName: Hai Nguyen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-16T16:51:33.787Z
Comments.Text: @David Ireland Cross entropy with what? What would be the label, y hat, or correct answer in the equation?
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-16T16:53:38.633Z
Comments.Text: @Hai Nguyen It is a lot about the general pseudocode, but if you click on the link you can see I built my own neural network out of just NumPy.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-16T17:05:48.363Z
Comments.Text: @S2673 It is cross-entropy loss using as the label the action you actually took during sampling. Multiplying by the value function is really critical addition though, it can even reverse the sign of all the gradients (which makes sense - if you have chosen an action and it was a bad choice, you want to reduce the chance of taking it again)
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-16T17:17:35.123Z
Comments.Text: @Neil Slater So if you had the probabilities `[0.7,0.3]` and the policy chose `0.7` then you would cross entropy loss with `[0.7,0.3]` and label `[1,0]` then multiply the gradient by the total reward? And is that what happens in REINFORCE or is that something different?
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-16T17:23:05.183Z
Comments.Text: @S2673: Yes, that is essentially what happens in REINFORCE. You can (and often would) choose different multiplier, with baselines, but basic REINFORCE just uses the measured return. A common choice is some estimate of advantage (action value minus state value).
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><backpropagation><policy-gradients><reinforce><cross-entropy>
Title: Which loss function should I use in REINFORCE, and what are the labels?
UserDisplayName: S2673
ThreadText: # Which loss function should I use in REINFORCE, and what are the labels?
  
  (Asked by: username_0 on 2020-09-16)
  
  I understand that this is the update for the parameters of a policy in REINFORCE:
  
  $$
  \Delta \theta\_{t}=\alpha \nabla\_{\theta} \log \pi\_{\theta}\left(a\_{t} \mid s\_{t}\right) v\_{t},
  $$
  where $v\_t$ is usually the discounted future reward and $\pi\_{\theta}\left(a\_{t} \mid s\_{t}\right)$ is the probability of taken the action that the agent took at time $t$. (Tell me if something is wrong here)
  
  However, I don't understand how to implement this with a neural network.
  
  Let's say that `probs = policy.feedforward(state)` returns the probabilities of taking each action, like `[0.6, 0.4]`. `action = choose_action_from(probs)` will return the index of the probability chosen. For example, if it chose 0.6, the action would be 0.
  
  When it is time to update the parameters of the policy network, what should we do? Should we do something like the following?
  
  ```
  gradient = policy.backpropagate(total_discounted_reward*log(probs[action])
  policy.weights += gradient
  
  ```
  
  And I only backpropagate this through one output neuron?
  
  Which loss function should I use in this case? What would the labels be?
  
  If you need more explanation, I have this [question](https://stackoverflow.com/q/63602222/14170431) on SO.
  
  **Tags:** reinforcement-learning, backpropagation, policy-gradients, reinforce, cross-entropy
  
  > **Comments:**
  > **username_1**: I use a cross entropy loss when I implement this in PyTorch. I tried doing a forward pass of the network evaluated at the current action and taking the log but it didn't seem to work when I implemented it, so I stuck with cross entropy loss.  
  > **username_2**: Is this question related to a specific library (eg. pytorch or tensorflow)? Or is this just a question for a more specific but still general pseudocode?  
  > **username_0**: @username_1 Ireland Cross entropy with what? What would be the label, y hat, or correct answer in the equation?  
  > **username_0**: @username_2 It is a lot about the general pseudocode, but if you click on the link you can see I built my own neural network out of just NumPy.  
  > **username_3**: @username_0 It is cross-entropy loss using as the label the action you actually took during sampling. Multiplying by the value function is really critical addition though, it can even reverse the sign of all the gradients (which makes sense - if you have chosen an action and it was a bad choice, you want to reduce the chance of taking it again)  
  > **username_0**: @username_3 So if you had the probabilities `[0.7,0.3]` and the policy chose `0.7` then you would cross entropy loss with `[0.7,0.3]` and label `[1,0]` then multiply the gradient by the total reward? And is that what happens in REINFORCE or is that something different?  
  > **username_3**: @username_0: Yes, that is essentially what happens in REINFORCE. You can (and often would) choose different multiplier, with baselines, but basic REINFORCE just uses the measured return. A common choice is some estimate of advantage (action value minus state value).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-16. Score: 3 | Accepted answer
  
  The loss function you are looking for is [cross entropy loss](https://en.wikipedia.org/wiki/Cross_entropy). The 'label' that you use is the action you took at the time point you are updating for.
  
  > **Comments:**
  > **username_0**: Okay, thank you. It was not working with my code so I had to build a new, simpler one to test it and it worked. That’s why it took so long. There’s just one more thing... How do I know to do this given the equation in my question? Or is this not what the equation meant?  
  > **username_1**: Should have clarified in my answer that I was referring to the [cross entropy loss in pytorch](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) — if you read the documentation you see that this loss function is the log of the policy for the action you give it.  
  > **username_0**: Thanks. Do you, by any chance, know how this works for a policy gradient with a continuous action space where the neural network outputs a Gaussian distribution? The equation for this update is the same but it is the log of a different output.  
  > **username_1**: @username_0 The algorithm doesn't change in this situation. Say your NN outputs the mean parameter of the Gaussian, then $\log_\pi(a_t | s_t)$ is just the log of the normal density evaluated at the action you took where the mean parameter in the density is the output of your NN. You are then able to backpropagate through this to update the weights of your network.  
  > **username_0**: Thank you. I had figured this out a bit ago.  
  > **username_1**: @username_0 Apologies, I forgot to reply sooner  
  
  ---
  
  