Answers.Body: <p>I’ll try to find where I found it, but normalizing the rewards has always worked for me. Assuming you have a list of the <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#reward-and-return" rel="nofollow noreferrer">discounted returns</a> for each action, you subtract the whole list by its average value then divide it by its standard deviation. In Python with NumPy, that would look like:</p>
  <pre><code>returns -= np.mean(returns)
  returns /= np.std(returns)
  </code></pre>
  <p>This puts the returns in a small and consistent range that keeps learning similar with different rewards.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-29T11:17:44.567Z
Answers.Text: I don't understand this, how can we get mean and str from the reward that hasn't revealed itself?? we don't have a returns vector.
Answers.UserDisplayName: fardis nadimi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-29T21:13:15.360Z
Answers.Text: I tried appending reward to a list and taking mean and std but loss functions after a couple of iterations were giving non...
Answers.UserDisplayName: fardis nadimi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-30T00:35:22.570Z
Answers.Text: @Fardis Could you please write a bit about how your code works? I want to make sure we are talking about scaling the same data.
Answers.UserDisplayName: S2673
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-30T20:15:20.963Z
Answers.Text: My code is a DDPG and simply we receive an action and for that action, the environment's reward function will produce a reward(it's my own environment). So for step 1, we have 1 reward sample. therefore taking the mean and std of this reward will be misleading.
Answers.UserDisplayName: fardis nadimi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-30T21:20:31.557Z
Answers.Text: @Fardis Thanks for explaining. My solution was meant for when the network is updated using multiple samples at the same time. First, have you made sure your code works by testing in a different environment? If it does then there are two ways I can think of to use my scaling suggestion for your method. My understanding is that DDPG will have a list with rewards stored and even if you only update with one reward at a time, you could subtract it by the mean and divide it by the variance of all of the rewards that have been collected. Another way is to change your reward function to output values
Answers.UserDisplayName: S2673
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-30T21:20:50.920Z
Answers.Text: ...with a mean of 0 and variance of 1 before it is even collected by your network.
Answers.UserDisplayName: S2673
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-28T02:09:01.913Z
Answers.LastActivityDate: 2021-04-28T02:09:01.913Z
Answers.UserDisplayName: S2673
Body: <p>My RL project has all positive continuous rewards for every step and the goal is to have the maximum cumulative reward (episodic reward). The problem is that the rewards are too close and all between 5 and 6, therefore achieving the optimum episodic reward will be harder.</p>
  <p>What scaling methods are recommended? (like min-max scaling or reward ** 3)</p>
  <p>How can I emphasize the episodic reward?</p>
  
Tags: <reinforcement-learning><q-learning><reward-functions><normalisation><standardisation>
Title: How to scale all positive continuous reward?
UserDisplayName: fardis nadimi
ThreadText: # How to scale all positive continuous reward?
  
  (Asked by: username_0 on 2021-04-26)
  
  My RL project has all positive continuous rewards for every step and the goal is to have the maximum cumulative reward (episodic reward). The problem is that the rewards are too close and all between 5 and 6, therefore achieving the optimum episodic reward will be harder.
  
  What scaling methods are recommended? (like min-max scaling or reward \*\* 3)
  
  How can I emphasize the episodic reward?
  
  **Tags:** reinforcement-learning, q-learning, reward-functions, normalisation, standardisation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-28. Score: 0
  
  I’ll try to find where I found it, but normalizing the rewards has always worked for me. Assuming you have a list of the [discounted returns](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#reward-and-return) for each action, you subtract the whole list by its average value then divide it by its standard deviation. In Python with NumPy, that would look like:
  
  ```
  returns -= np.mean(returns)
  returns /= np.std(returns)
  
  ```
  
  This puts the returns in a small and consistent range that keeps learning similar with different rewards.
  
  > **Comments:**
  > **username_0**: I don't understand this, how can we get mean and str from the reward that hasn't revealed itself?? we don't have a returns vector.  
  > **username_0**: I tried appending reward to a list and taking mean and std but loss functions after a couple of iterations were giving non...  
  > **username_1**: @username_0 Could you please write a bit about how your code works? I want to make sure we are talking about scaling the same data.  
  > **username_0**: My code is a DDPG and simply we receive an action and for that action, the environment's reward function will produce a reward(it's my own environment). So for step 1, we have 1 reward sample. therefore taking the mean and std of this reward will be misleading.  
  > **username_1**: @username_0 Thanks for explaining. My solution was meant for when the network is updated using multiple samples at the same time. First, have you made sure your code works by testing in a different environment? If it does then there are two ways I can think of to use my scaling suggestion for your method. My understanding is that DDPG will have a list with rewards stored and even if you only update with one reward at a time, you could subtract it by the mean and divide it by the variance of all of the rewards that have been collected. Another way is to change your reward function to output values  
  > **username_1**: ...with a mean of 0 and variance of 1 before it is even collected by your network.  
  
  ---
  
  