Answers.Body: <p>So this question is due to a misunderstanding I had with how the loss function works in A2C. </p>
  
  <p>Usually loss functions are always positives, but in A2C it can be negative as well. I thought that minimizing the loss function means moving it closer to zero, but in fact reducing the loss is indifferent to the sign of the loss value and is about making the loss value smaller.</p>
  
  <p>In other words, the loss for 0.1 is smaller than the loss for 0.9 prediction for a disadvantageous action.</p>
  
  <p>When the advantage is positive, <span class="math-container">$-log (p) * advantage$</span> will be smaller if <span class="math-container">$p$</span> is growing, and when the advantage is negative, <span class="math-container">$-log (p) * advantage$</span> will be smaller if <span class="math-container">$p$</span> is decreasing.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T13:06:35.037Z
Answers.LastActivityDate: 2020-05-01T13:06:35.037Z
Answers.UserDisplayName: asaf92
Body: <p>A2C loss is usually defined as <code>advantage * (-log(actor_predictions)) * target</code> where <code>target</code> is a one-hot vector (with some clipping/noise/etc...) with the selected target. </p>
  
  <p>Does this mean that we get larger losses for smaller mistakes?</p>
  
  <p>If for example the agent has predicted <span class="math-container">$\pi(a|s)=0.9$</span> but the advantage is negative, this would mean a larger mistake than if the agent predicted that <span class="math-container">$\pi(a|s)=0.1$</span>, however, putting the numbers in the formula means a larger loss for the <code>0.1</code> prediction.</p>
  
  <p>Assuming <code>advantage=-1</code>, <code>advantage * (-log(actor_predictions)) * target</code> would mean:</p>
  
  <p><span class="math-container">$$
  -1 * (-log(0.9)) * 1 = log(0.9)=-0.045
  $$</span>
  <span class="math-container">$$
  -1 * (-log(0.1)) * 1 = log(0.1)=-1
  $$</span></p>
  
  <p>Is my understanding correct?</p>
  
Tags: <objective-functions><policy-gradients><advantage-actor-critic>
Title: Is A2C loss function taking smaller steps for larger mistakes?
UserDisplayName: asaf92
ThreadText: # Is A2C loss function taking smaller steps for larger mistakes?
  
  (Asked by: username_0 on 2020-03-24)
  
  A2C loss is usually defined as `advantage * (-log(actor_predictions)) * target` where `target` is a one-hot vector (with some clipping/noise/etc...) with the selected target.
  
  Does this mean that we get larger losses for smaller mistakes?
  
  If for example the agent has predicted $\pi(a|s)=0.9$ but the advantage is negative, this would mean a larger mistake than if the agent predicted that $\pi(a|s)=0.1$, however, putting the numbers in the formula means a larger loss for the `0.1` prediction.
  
  Assuming `advantage=-1`, `advantage * (-log(actor_predictions)) * target` would mean:
  
  $$
  -1 \* (-log(0.9)) \* 1 = log(0.9)=-0.045
  $$
  $$
  -1 \* (-log(0.1)) \* 1 = log(0.1)=-1
  $$
  
  Is my understanding correct?
  
  **Tags:** objective-functions, policy-gradients, advantage-actor-critic
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-05-01. Score: 0 | Accepted answer
  
  So this question is due to a misunderstanding I had with how the loss function works in A2C.
  
  Usually loss functions are always positives, but in A2C it can be negative as well. I thought that minimizing the loss function means moving it closer to zero, but in fact reducing the loss is indifferent to the sign of the loss value and is about making the loss value smaller.
  
  In other words, the loss for 0.1 is smaller than the loss for 0.9 prediction for a disadvantageous action.
  
  When the advantage is positive, $-log (p) \* advantage$ will be smaller if $p$ is growing, and when the advantage is negative, $-log (p) \* advantage$ will be smaller if $p$ is decreasing.
  
  
  ---
  
  