Body: <p>I have a custom PPO implementation and a problem that has costs rather than rewards, so I basically need to take the negative value for PPO to work. As the values are somewhat large, I've tried various normalization methods. The best one so far is to simply take the log of the cost, as normalizing with a running mean/std (using Welford's algorithm) seems to collapse the value loss to almost zero and learning becomes very unstable. Are there any guidelines for dealing with reward normalization in these cases?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-08T00:26:24.980Z
Comments.Text: have you tried tuning the discount factor? Because that might be more important than normalization. From what I've seen/experienced, value function normalization is typically not too important or necessary
Comments.UserDisplayName: Taw
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-08T07:40:31.763Z
Comments.Text: What if you have values on the scale of billions and trillions?
Comments.UserDisplayName: Antonis Karvelas
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-09T04:32:41.753Z
Comments.Text: I could take cartpole, and make the value function large by making each timestep +1e6 instead of +1. If I then scaled the learning rate by 1e-6, this would be equivalent to solving the regular cartpole with the regular learning rate
Comments.UserDisplayName: Taw
Tags: <reinforcement-learning><rewards><proximal-policy-optimization><reward-normalization>
Title: PPO: how to scale rewards
UserDisplayName: Antonis Karvelas
ThreadText: # PPO: how to scale rewards
  
  (Asked by: username_0 on 2022-06-06)
  
  I have a custom PPO implementation and a problem that has costs rather than rewards, so I basically need to take the negative value for PPO to work. As the values are somewhat large, I've tried various normalization methods. The best one so far is to simply take the log of the cost, as normalizing with a running mean/std (using Welford's algorithm) seems to collapse the value loss to almost zero and learning becomes very unstable. Are there any guidelines for dealing with reward normalization in these cases?
  
  **Tags:** reinforcement-learning, rewards, proximal-policy-optimization, reward-normalization
  
  > **Comments:**
  > **username_1**: have you tried tuning the discount factor? Because that might be more important than normalization. From what I've seen/experienced, value function normalization is typically not too important or necessary  
  > **username_0**: What if you have values on the scale of billions and trillions?  
  > **username_1**: I could take cartpole, and make the value function large by making each timestep +1e6 instead of +1. If I then scaled the learning rate by 1e-6, this would be equivalent to solving the regular cartpole with the regular learning rate  
  
  ---
  
  ## 0 answers
  
  