Answers.Body: <p>So it seems I was right. The ValueLoss does not decrease as it IS a 'chasing the tail situation'. If I run the code with ValueLoss = a constant - Value instead of NextValue - Value, I see that the loss decreases.
  A way out of it can be to calculate NextValue not for every run but for some runs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-26T19:38:04.380Z
Answers.Text: Why would you want a "way out of it"? The reason the critic is "chasing its tail" is at least partially because the actor is improving, so the target values are *really* changing. What this does mean is that you should have different expectations when monitoring the critic loss compared to monitoring a supervised learning experiment.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-28T11:18:48.727Z
Answers.Text: Yes you are absolutely right.
Answers.UserDisplayName: Khabbab Zakaria
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-26T18:53:05.667Z
Answers.LastActivityDate: 2022-02-26T18:53:05.667Z
Answers.UserDisplayName: Khabbab Zakaria
Body: <p>I am coding a problem with the Actor-Critic Method. The final loss is a summation of PolicyLoss and ValueLoss. The calculation of the PolicyLoss for each step is given at Equation Number 5 of <a href="https://arxiv.org/pdf/1707.06347.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1707.06347.pdf</a>. And for Valueloss:</p>
  <p><span class="math-container">$V_t = \gamma \times V_{t+1} + r_t$</span></p>
  <p><span class="math-container">$\mbox{Valueloss} = \mbox{L}1\_\mbox{Loss}(Vt, Value)$</span></p>
  <p>I checked that the PolicyLoss depends on this 'Value' as well. And PolicyLoss is decreasing, and so the parts of my AI model depending on the 'Value' are also updating. But I am seeing that the ValueLoss is not decreasing, however, the Total Loss which is the sum(ValueLoss, PolicyLoss) is decreasing.</p>
  <p>I don't understand how to make the ValueLoss decrease, or if it should decrease at all. Because, the equation of ValueLoss suggests that it is like a 'chasing the tail' situation, so it may fluctuate.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-26T10:05:23.813Z
Comments.Text: Please, provide the correct link to the paper you're referring to. I believe that's `https://arxiv.org/abs/1707.06347`.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-26T10:08:40.733Z
Comments.Text: Yes, sorry, I edited now
Comments.UserDisplayName: Khabbab Zakaria
Tags: <python><actor-critic-methods><loss>
Title: Does Value Loss in Actor Critic not decrease at all?
UserDisplayName: Khabbab Zakaria
ThreadText: # Does Value Loss in Actor Critic not decrease at all?
  
  (Asked by: username_0 on 2022-02-26)
  
  I am coding a problem with the Actor-Critic Method. The final loss is a summation of PolicyLoss and ValueLoss. The calculation of the PolicyLoss for each step is given at Equation Number 5 of <https://arxiv.org/pdf/1707.06347.pdf>. And for Valueloss:
  
  $V\_t = \gamma \times V\_{t+1} + r\_t$
  
  $\mbox{Valueloss} = \mbox{L}1\\_\mbox{Loss}(Vt, Value)$
  
  I checked that the PolicyLoss depends on this 'Value' as well. And PolicyLoss is decreasing, and so the parts of my AI model depending on the 'Value' are also updating. But I am seeing that the ValueLoss is not decreasing, however, the Total Loss which is the sum(ValueLoss, PolicyLoss) is decreasing.
  
  I don't understand how to make the ValueLoss decrease, or if it should decrease at all. Because, the equation of ValueLoss suggests that it is like a 'chasing the tail' situation, so it may fluctuate.
  
  **Tags:** python, actor-critic-methods, loss
  
  > **Comments:**
  > **username_1**: Please, provide the correct link to the paper you're referring to. I believe that's `https://arxiv.org/abs/1707.06347`.  
  > **username_0**: Yes, sorry, I edited now  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-02-26. Score: 0
  
  So it seems I was right. The ValueLoss does not decrease as it IS a 'chasing the tail situation'. If I run the code with ValueLoss = a constant - Value instead of NextValue - Value, I see that the loss decreases.
  A way out of it can be to calculate NextValue not for every run but for some runs.
  
  > **Comments:**
  > **username_2**: Why would you want a "way out of it"? The reason the critic is "chasing its tail" is at least partially because the actor is improving, so the target values are *really* changing. What this does mean is that you should have different expectations when monitoring the critic loss compared to monitoring a supervised learning experiment.  
  > **username_0**: Yes you are absolutely right.  
  
  ---
  
  