Answers.Body: <p>It's difficult to tell how the output of the loss changes over the course of the experiment as you use them as a measure on how well your model performs. Ideally, the loss decreases over time with minimal jittering. A decreasing loss means the model is learning and the hyper parameters fit the dataset.</p>
  <p>If your loss is jittering too much without a clear decreasing trend, it might be that, for example, the learning rate is too high and the model overshoots the minimum. It also might be that there is a flaw in the model.</p>
  <p>To make it short, no, the loss doesn't always move along the same path.</p>
  <p>If you need to test your metric, create tests with small models. Use a small keras model (for example) where you know that there is no flaw inherent with the model and use standard hyper parameters. Then train the model twice. Once with your own metric and once with a standard metric similar to your own provided by a library. The path of the loss should look similar - but it will not be the same.</p>
  <p>I'd do the above approach for every method you implemented yourself to make sure that the method itself is doing what it is intended to do.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-26T07:04:05.473Z
Answers.LastActivityDate: 2022-05-26T07:04:05.473Z
Answers.UserDisplayName: emely_pi
Body: <p>I'm wondering whether problems with a custom PPO implementation (problem couldn't be made into a gym environment) can be debugged considering how the losses change over time.
  In my current experiment, although results do get better in the actual task, the actor(policy) loss, the value function loss and the entropy move in weird ways.
  <a href="https://i.sstatic.net/97VF7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/97VF7.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/JgaBc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JgaBc.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/7NJvD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/7NJvD.png" alt="enter image description here" /></a></p>
  <p>How do these metrics usually change over the course of an experiment? Do they always move along the same paths? Thank you.</p>
  
Tags: <reinforcement-learning><value-functions><proximal-policy-optimization>
Title: PPO custom implementation: do metrics like value loss, actor loss and entropy move a certain way?
UserDisplayName: Antonis Karvelas
ThreadText: # PPO custom implementation: do metrics like value loss, actor loss and entropy move a certain way?
  
  (Asked by: username_0 on 2022-05-25)
  
  I'm wondering whether problems with a custom PPO implementation (problem couldn't be made into a gym environment) can be debugged considering how the losses change over time.
  In my current experiment, although results do get better in the actual task, the actor(policy) loss, the value function loss and the entropy move in weird ways.
  [![enter image description here](https://i.sstatic.net/97VF7.png)](https://i.sstatic.net/97VF7.png)
  
  [![enter image description here](https://i.sstatic.net/JgaBc.png)](https://i.sstatic.net/JgaBc.png)
  
  [![enter image description here](https://i.sstatic.net/7NJvD.png)](https://i.sstatic.net/7NJvD.png)
  
  How do these metrics usually change over the course of an experiment? Do they always move along the same paths? Thank you.
  
  **Tags:** reinforcement-learning, value-functions, proximal-policy-optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-05-26. Score: 1 | Accepted answer
  
  It's difficult to tell how the output of the loss changes over the course of the experiment as you use them as a measure on how well your model performs. Ideally, the loss decreases over time with minimal jittering. A decreasing loss means the model is learning and the hyper parameters fit the dataset.
  
  If your loss is jittering too much without a clear decreasing trend, it might be that, for example, the learning rate is too high and the model overshoots the minimum. It also might be that there is a flaw in the model.
  
  To make it short, no, the loss doesn't always move along the same path.
  
  If you need to test your metric, create tests with small models. Use a small keras model (for example) where you know that there is no flaw inherent with the model and use standard hyper parameters. Then train the model twice. Once with your own metric and once with a standard metric similar to your own provided by a library. The path of the loss should look similar - but it will not be the same.
  
  I'd do the above approach for every method you implemented yourself to make sure that the method itself is doing what it is intended to do.
  
  
  ---
  
  