Body: <p>I am using stable-baseline3 implementation of the Soft-Actor-Critic (SAC) algorithm. The plotted training curves look promising. However, I am not fully sure how to interpret the actor and critic losses. The entropy coefficient <span class="math-container">$\alpha$</span> is automatically learned during training. As the entropy decreases, the critic loss and actor loss decrease as well.</p>
  <ul>
  <li>How does the entropy coefficient affect the losses?</li>
  <li>Can this be interpreted as the estimations becoming more accurate as the focus is shifted from exploration to exploitation?</li>
  <li>How can negative actor losses be interpreted, what do actor losses tell in general?</li>
  </ul>
  <p><a href="https://i.sstatic.net/0Rt2h.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0Rt2h.png" alt="enter image description here" /></a></p>
  <p>Thanks a lot in advance</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-06T16:29:28.327Z
Comments.Text: I am getting negative actor loss in SAC implementation from stable-baseline3. Anything on this?
Comments.UserDisplayName: devil in the detail
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-06T15:12:38.870Z
Comments.Text: @devilinthedetail that's expected and good, if expected return is positive :)
Comments.UserDisplayName: kmf
Tags: <deep-rl><actor-critic-methods><loss><soft-actor-critic><learning-curve>
Title: How to interpret the training loss curves in Soft-Actor-Critic (SAC)?
UserDisplayName: Manuel
ThreadText: # How to interpret the training loss curves in Soft-Actor-Critic (SAC)?
  
  (Asked by: username_0 on 2021-07-01)
  
  I am using stable-baseline3 implementation of the Soft-Actor-Critic (SAC) algorithm. The plotted training curves look promising. However, I am not fully sure how to interpret the actor and critic losses. The entropy coefficient $\alpha$ is automatically learned during training. As the entropy decreases, the critic loss and actor loss decrease as well.
  
  * How does the entropy coefficient affect the losses?
  * Can this be interpreted as the estimations becoming more accurate as the focus is shifted from exploration to exploitation?
  * How can negative actor losses be interpreted, what do actor losses tell in general?
  
  [![enter image description here](https://i.sstatic.net/0Rt2h.png)](https://i.sstatic.net/0Rt2h.png)
  
  Thanks a lot in advance
  
  **Tags:** deep-rl, actor-critic-methods, loss, soft-actor-critic, learning-curve
  
  > **Comments:**
  > **username_1**: I am getting negative actor loss in SAC implementation from stable-baseline3. Anything on this?  
  > **username_2**: @username_1 that's expected and good, if expected return is positive :)  
  
  ---
  
  ## 0 answers
  
  