Body: <p>I have two signals that I want to use to model my reward.</p>
  <p>The first one is the <strong>CPU TIME: running mean</strong> from this diagram:</p>
  <p><a href="https://i.sstatic.net/m3Ijz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/m3Ijz.png" alt="enter image description here" /></a></p>
  <p>The second one is the <strong>MAX RESIDUAL</strong> from this diagram:</p>
  <p><a href="https://i.sstatic.net/h8zOg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/h8zOg.png" alt="enter image description here" /></a></p>
  <p>Since they are both equally important, I can weight them together like this:</p>
  <p><span class="math-container">$r = w_\rho \rho + w_\tau \tau$</span></p>
  <p>where <span class="math-container">$r$</span> is the reward function, <span class="math-container">$\tau$</span> is the <strong>CPU TIME: running mean</strong>, and <span class="math-container">$\rho$</span> is the <strong>MAX RESIDUAL</strong>. The problem is, how to set the weights <span class="math-container">$w_\tau,w_\rho$</span> to make the contributions equally important if <span class="math-container">$\rho$</span> and <span class="math-container">$\tau$</span> are on very different scales?</p>
  <p>RL algorithms will learn policies based on increases/decreases of the reward, and if one signal has values that are much smaller than the other, it will influence the reward less, if this is done in a wrong way.</p>
  <p>On the other hand, if the algorithm converges, they must be on different scales, as I want the CPU time to go <em>ideally</em> to <span class="math-container">$0$</span>, and residuals <span class="math-container">$\rho$</span> to be minimized as well.</p>
  <p>Modeling the reward function is a crucial RL step, because it decides what the algorithm will in fact optimize. How are examples like these handled? Are there any best practices for this? Also, what happens when there are <span class="math-container">$n$</span> such signals, that have to be combined with &quot;equally important&quot; weighting into a reward function?</p>
  <p>Basing the weights <span class="math-container">$w$</span> on the <em>current</em> signal values is possible to define their reward, but then the reward contributions won't see <span class="math-container">$\max(\rho), \min(\rho)$</span> and <span class="math-container">$\max(\tau), \min(\tau)$</span> over time.</p>
  <p>So, how do you do feature scaling for reward signals?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-07T12:52:47.707Z
Comments.Text: scale your rewards to be 0 mean and variance 1?
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-07T13:47:40.967Z
Comments.Text: @DavidIreland I tried z-scaling already and it doesn't seem to scale to same min,max intervals for "running" signals... can you suggest an alternative scaling for running signals that does what you suggest? Thanks, btw :)
Comments.UserDisplayName: tmaric
Tags: <reinforcement-learning><deep-rl><reward-design><reward-functions>
Title: How to combine two differently equally important signals into the reward function, that have different scales?
UserDisplayName: tmaric
ThreadText: # How to combine two differently equally important signals into the reward function, that have different scales?
  
  (Asked by: username_0 on 2020-08-07)
  
  I have two signals that I want to use to model my reward.
  
  The first one is the **CPU TIME: running mean** from this diagram:
  
  [![enter image description here](https://i.sstatic.net/m3Ijz.png)](https://i.sstatic.net/m3Ijz.png)
  
  The second one is the **MAX RESIDUAL** from this diagram:
  
  [![enter image description here](https://i.sstatic.net/h8zOg.png)](https://i.sstatic.net/h8zOg.png)
  
  Since they are both equally important, I can weight them together like this:
  
  $r = w\_\rho \rho + w\_\tau \tau$
  
  where $r$ is the reward function, $\tau$ is the **CPU TIME: running mean**, and $\rho$ is the **MAX RESIDUAL**. The problem is, how to set the weights $w\_\tau,w\_\rho$ to make the contributions equally important if $\rho$ and $\tau$ are on very different scales?
  
  RL algorithms will learn policies based on increases/decreases of the reward, and if one signal has values that are much smaller than the other, it will influence the reward less, if this is done in a wrong way.
  
  On the other hand, if the algorithm converges, they must be on different scales, as I want the CPU time to go *ideally* to $0$, and residuals $\rho$ to be minimized as well.
  
  Modeling the reward function is a crucial RL step, because it decides what the algorithm will in fact optimize. How are examples like these handled? Are there any best practices for this? Also, what happens when there are $n$ such signals, that have to be combined with "equally important" weighting into a reward function?
  
  Basing the weights $w$ on the *current* signal values is possible to define their reward, but then the reward contributions won't see $\max(\rho), \min(\rho)$ and $\max(\tau), \min(\tau)$ over time.
  
  So, how do you do feature scaling for reward signals?
  
  **Tags:** reinforcement-learning, deep-rl, reward-design, reward-functions
  > **Comments:**
  > **username_1**: scale your rewards to be 0 mean and variance 1?  
  > **username_0**: @DavidIreland I tried z-scaling already and it doesn't seem to scale to same min,max intervals for "running" signals... can you suggest an alternative scaling for running signals that does what you suggest? Thanks, btw :)  
  
  ---
  
  ## 0 answers
  
  