Answers.Body: <p>There are several ways to normalize various elements of the reward function. One way would be to divide all values by the largest value in the set. This would ensure that all values are between <span class="math-container">$0$</span> and <span class="math-container">$1$</span>. Another way would be to subtract the mean from all values and then divide it by the standard deviation. This would ensure that all values have a mean of <span class="math-container">$0$</span> and a standard deviation of <span class="math-container">$1$</span>.</p>
  <p>A trivial reward function for this scenario would be to simply assign a negative value to <span class="math-container">$R$</span> whenever either <span class="math-container">$A$</span> or <span class="math-container">$B$</span> occurs. However, this may not be the most effective way to penalize the behavior. A more effective approach might be to weigh the penalties differ depending on how severe the violation is. For example, if <span class="math-container">$A$</span> occurs, the penalty could be <span class="math-container">$-1$</span>, but if <span class="math-container">$B$</span> occurs, the penalty could be <span class="math-container">$-10$</span>. This would ensure that more severe violations are penalized more heavily than less severe violations.</p>
  <p>This reward function appears to be penalizing the agent for taking actions that result in the state not being within some boundary, as well as for taking actions that are too far apart from the previous action.</p>
  <p>One way to avoid this issue would be to normalize the values of <span class="math-container">$A$</span> and <span class="math-container">$B$</span> before subtracting them from the current reward. This would ensure that both <span class="math-container">$A$</span> and <span class="math-container">$B$</span> have an equal impact on the reward function, regardless of their range.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-17T09:00:13.987Z
Answers.LastActivityDate: 2022-10-17T09:10:30.480Z
Answers.LastEditDate: 2022-10-17T09:10:30.480Z
Answers.UserDisplayName: Faizy
Body: <p>Suppose I have a reward function <span class="math-container">$R$</span> that I wish to penalize w.r.t two distinct phenomenons <span class="math-container">$A$</span> and <span class="math-container">$B$</span>. <span class="math-container">$A$</span>, for example, could represent the phenomenon of the state not crossing some boundary <span class="math-container">$[s_1,s_2]$</span> and <span class="math-container">$B$</span> can represent the phenomenon that two consecutive actions shouldn't be too far apart <span class="math-container">$|a_t - a_{t+1}| &lt; \epsilon$</span>, for some small <span class="math-container">$\epsilon$</span>. A trivial reward mechanism can be:</p>
  <pre><code>step(prev_a, a,epsilon, min_s, max_s):
      s = env(action)
      if not min_s &lt; s &lt; max_s: 
          r -= s
      if ||a-prev_a|| &gt; epsilon:
          r -= a
  </code></pre>
  <p>As <span class="math-container">$A$</span> and <span class="math-container">$B$</span> are from different worlds (different physical units, if you will), they both have different ranges. For example, a state <span class="math-container">$s$</span> may obtain values that are at most <span class="math-container">$10$</span>, though an action <span class="math-container">$a$</span> may have larger values like <span class="math-container">$100$</span>. Hence, penalizing by subtracting the state or the action from the current reward may lead to the preference of the agent to only make sure the action condition is set, as this one translates to more future rewards.</p>
  <p>How can this issue be addressed? I assume some normalization should be added, though I'm not quite sure how.</p>
  <p>Any ideas?</p>
  
Tags: <reinforcement-learning><deep-rl><rewards><reward-functions><environment>
Title: How to normalizing various elements of the reward function?
UserDisplayName: Hadar Sharvit
ThreadText: # How to normalizing various elements of the reward function?
  
  (Asked by: username_0 on 2022-10-11)
  
  Suppose I have a reward function $R$ that I wish to penalize w.r.t two distinct phenomenons $A$ and $B$. $A$, for example, could represent the phenomenon of the state not crossing some boundary $[s\_1,s\_2]$ and $B$ can represent the phenomenon that two consecutive actions shouldn't be too far apart $|a\_t - a\_{t+1}| < \epsilon$, for some small $\epsilon$. A trivial reward mechanism can be:
  
  ```
  step(prev_a, a,epsilon, min_s, max_s):
      s = env(action)
      if not min_s < s < max_s: 
          r -= s
      if ||a-prev_a|| > epsilon:
          r -= a
  
  ```
  
  As $A$ and $B$ are from different worlds (different physical units, if you will), they both have different ranges. For example, a state $s$ may obtain values that are at most $10$, though an action $a$ may have larger values like $100$. Hence, penalizing by subtracting the state or the action from the current reward may lead to the preference of the agent to only make sure the action condition is set, as this one translates to more future rewards.
  
  How can this issue be addressed? I assume some normalization should be added, though I'm not quite sure how.
  
  Any ideas?
  
  **Tags:** reinforcement-learning, deep-rl, rewards, reward-functions, environment
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-17. Score: 1
  
  There are several ways to normalize various elements of the reward function. One way would be to divide all values by the largest value in the set. This would ensure that all values are between $0$ and $1$. Another way would be to subtract the mean from all values and then divide it by the standard deviation. This would ensure that all values have a mean of $0$ and a standard deviation of $1$.
  
  A trivial reward function for this scenario would be to simply assign a negative value to $R$ whenever either $A$ or $B$ occurs. However, this may not be the most effective way to penalize the behavior. A more effective approach might be to weigh the penalties differ depending on how severe the violation is. For example, if $A$ occurs, the penalty could be $-1$, but if $B$ occurs, the penalty could be $-10$. This would ensure that more severe violations are penalized more heavily than less severe violations.
  
  This reward function appears to be penalizing the agent for taking actions that result in the state not being within some boundary, as well as for taking actions that are too far apart from the previous action.
  
  One way to avoid this issue would be to normalize the values of $A$ and $B$ before subtracting them from the current reward. This would ensure that both $A$ and $B$ have an equal impact on the reward function, regardless of their range.
  
  
  ---
  
  