<p>So it seems I was right. The ValueLoss does not decrease as it IS a 'chasing the tail situation'. If I run the code with ValueLoss = a constant - Value instead of NextValue - Value, I see that the loss decreases.
A way out of it can be to calculate NextValue not for every run but for some runs.</p>

CC BY-SA 4.0
2022-02-26T19:38:04.380Z
Why would you want a "way out of it"? The reason the critic is "chasing its tail" is at least partially because the actor is improving, so the target values are *really* changing. What this does mean is that you should have different expectations when monitoring the critic loss compared to monitoring a supervised learning experiment.
Neil Slater
CC BY-SA 4.0
2022-02-28T11:18:48.727Z
Yes you are absolutely right.
Khabbab Zakaria
CC BY-SA 4.0
2022-02-26T18:53:05.667Z
2022-02-26T18:53:05.667Z
Khabbab Zakaria
<p>I am coding a problem with the Actor-Critic Method. The final loss is a summation of PolicyLoss and ValueLoss. The calculation of the PolicyLoss for each step is given at Equation Number 5 of <a href="https://arxiv.org/pdf/1707.06347.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1707.06347.pdf</a>. And for Valueloss:</p>
<p><span class="math-container">$V_t = \gamma \times V_{t+1} + r_t$</span></p>
<p><span class="math-container">$\mbox{Valueloss} = \mbox{L}1\_\mbox{Loss}(Vt, Value)$</span></p>
<p>I checked that the PolicyLoss depends on this 'Value' as well. And PolicyLoss is decreasing, and so the parts of my AI model depending on the 'Value' are also updating. But I am seeing that the ValueLoss is not decreasing, however, the Total Loss which is the sum(ValueLoss, PolicyLoss) is decreasing.</p>
<p>I don't understand how to make the ValueLoss decrease, or if it should decrease at all. Because, the equation of ValueLoss suggests that it is like a 'chasing the tail' situation, so it may fluctuate.</p>

CC BY-SA 4.0
2022-02-26T10:05:23.813Z
Please, provide the correct link to the paper you're referring to. I believe that's `https://arxiv.org/abs/1707.06347`.
nbro
CC BY-SA 4.0
2022-02-26T10:08:40.733Z
Yes, sorry, I edited now
Khabbab Zakaria
<python><actor-critic-methods><loss>
Does Value Loss in Actor Critic not decrease at all?
Khabbab Zakaria
# Does Value Loss in Actor Critic not decrease at all?

(Asked by: username_0 on 2022-02-26)

I am coding a problem with the Actor-Critic Method. The final loss is a summation of PolicyLoss and ValueLoss. The calculation of the PolicyLoss for each step is given at Equation Number 5 of <https://arxiv.org/pdf/1707.06347.pdf>. And for Valueloss:

$V\_t = \gamma \times V\_{t+1} + r\_t$

$\mbox{Valueloss} = \mbox{L}1\\_\mbox{Loss}(Vt, Value)$

I checked that the PolicyLoss depends on this 'Value' as well. And PolicyLoss is decreasing, and so the parts of my AI model depending on the 'Value' are also updating. But I am seeing that the ValueLoss is not decreasing, however, the Total Loss which is the sum(ValueLoss, PolicyLoss) is decreasing.

I don't understand how to make the ValueLoss decrease, or if it should decrease at all. Because, the equation of ValueLoss suggests that it is like a 'chasing the tail' situation, so it may fluctuate.

**Tags:** python, actor-critic-methods, loss

> **Comments:**
> **username_1**: Please, provide the correct link to the paper you're referring to. I believe that's `https://arxiv.org/abs/1707.06347`.  
> **username_0**: Yes, sorry, I edited now  

---

## 1 answers

### Answer by username_0 on 2022-02-26. Score: 0

So it seems I was right. The ValueLoss does not decrease as it IS a 'chasing the tail situation'. If I run the code with ValueLoss = a constant - Value instead of NextValue - Value, I see that the loss decreases.
A way out of it can be to calculate NextValue not for every run but for some runs.

> **Comments:**
> **username_2**: Why would you want a "way out of it"? The reason the critic is "chasing its tail" is at least partially because the actor is improving, so the target values are *really* changing. What this does mean is that you should have different expectations when monitoring the critic loss compared to monitoring a supervised learning experiment.  
> **username_0**: Yes you are absolutely right.  

---

