Answers.Body: <p>In general the reward function can be anything that allows you to solve the task: if the reward is misspecified, the env cannot be solved; this is the <em>reward hypothesis</em> which RL relies on.</p>
  <p>So, consider the reward function to be defined as <span class="math-container">$r(s,a)$</span> or even <span class="math-container">$r(s,a,s')$</span> from a state <span class="math-container">$s$</span>, actions <span class="math-container">$a$</span>, and sometimes including also the next state <span class="math-container">$s'$</span>. It can be as much complex as you want, having various cases, parts, and so on. But consider that if too complex you may find a hard time debugging it, to see if the reward fn suits your task. Also, different rewards (assuming they are correct enabling to solve the problem) may result in slower or faster convergence to optimality.</p>
  <blockquote>
  <p>What I wanted to do is to have different parts depending on the value. Like if my value is in an interval under 15
  secs can I normalize the value? And use another way to calculate reward when it's in an interval above?</p>
  </blockquote>
  <p>In principle you can do that, as I said above. Try to have a reward range that is bounded and possibly limited (e.g. <span class="math-container">$[-1, 1]$</span>), avoiding large values and occasional spikes.</p>
  <p>For example, you mentioned that <span class="math-container">$1/x^2$</span> works well when the delays are quite different. In the case the delays are similar, you can try to transform them (e.g. by <span class="math-container">$\exp$</span> to increase the difference) and then apply the above function. For example, case 1 is <span class="math-container">$1/x^2$</span>, case 2 can be <span class="math-container">$1/\exp(x)$</span>, case 3 something else, etc..</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-30T07:17:47.540Z
Answers.Text: Ok, I think I get it now ! Thanks for your help !
Answers.UserDisplayName: AdrienF
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-25T16:07:49.597Z
Answers.LastActivityDate: 2023-05-25T16:07:49.597Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I'm doing a Q-learning algorithm and I'm designing my reward function. Basically I'm working on optimizing a network while changing some parameters. My metric to measure its optimization is the delay on a flow of data (generated).</p>
  <p>I've discretized my delay values in intervals and now I'm designing the rewards. What I was thinking about is, as I want to prioritize lower delays to use a non linear function such as <span class="math-container">$1/x^2$</span>.
  The range of my delays are usually from <span class="math-container">$10$</span> to <span class="math-container">$40$</span> secs (depending on the perturbations). It works well when the delay is varying a lot, but way less good when the delay isn't varying that much (with low perturbations).</p>
  <p>I was then wondering if there are restrictions on reward functions. What I wanted to do is to have different parts depending on the value. Like if my value is in an interval under <span class="math-container">$15$</span> secs can I normalize the value? And use another way to calculate reward when it's in an interval above?</p>
  <p>I'm new to reinforcement learning so maybe what I said is non sense, but I would gladly hear any advice or idea.</p>
  <p>Adrien</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-25T16:05:27.810Z
Comments.Text: as long as your reward is in $\mathbb{R}$ for all $(s, a, s')$ you can define it however you like.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><machine-learning><reward-functions>
Title: Can a reward function have various cases?
UserDisplayName: AdrienF
ThreadText: # Can a reward function have various cases?
  
  (Asked by: username_0 on 2023-05-25)
  
  I'm doing a Q-learning algorithm and I'm designing my reward function. Basically I'm working on optimizing a network while changing some parameters. My metric to measure its optimization is the delay on a flow of data (generated).
  
  I've discretized my delay values in intervals and now I'm designing the rewards. What I was thinking about is, as I want to prioritize lower delays to use a non linear function such as $1/x^2$.
  The range of my delays are usually from $10$ to $40$ secs (depending on the perturbations). It works well when the delay is varying a lot, but way less good when the delay isn't varying that much (with low perturbations).
  
  I was then wondering if there are restrictions on reward functions. What I wanted to do is to have different parts depending on the value. Like if my value is in an interval under $15$ secs can I normalize the value? And use another way to calculate reward when it's in an interval above?
  
  I'm new to reinforcement learning so maybe what I said is non sense, but I would gladly hear any advice or idea.
  
  Adrien
  
  **Tags:** reinforcement-learning, machine-learning, reward-functions
  > **Comments:**
  > **username_1**: as long as your reward is in $\mathbb{R}$ for all $(s, a, s')$ you can define it however you like.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-05-25. Score: 2 | Accepted answer
  
  In general the reward function can be anything that allows you to solve the task: if the reward is misspecified, the env cannot be solved; this is the *reward hypothesis* which RL relies on.
  
  So, consider the reward function to be defined as $r(s,a)$ or even $r(s,a,s')$ from a state $s$, actions $a$, and sometimes including also the next state $s'$. It can be as much complex as you want, having various cases, parts, and so on. But consider that if too complex you may find a hard time debugging it, to see if the reward fn suits your task. Also, different rewards (assuming they are correct enabling to solve the problem) may result in slower or faster convergence to optimality.
  
  > What I wanted to do is to have different parts depending on the value. Like if my value is in an interval under 15
  > secs can I normalize the value? And use another way to calculate reward when it's in an interval above?
  
  In principle you can do that, as I said above. Try to have a reward range that is bounded and possibly limited (e.g. $[-1, 1]$), avoiding large values and occasional spikes.
  
  For example, you mentioned that $1/x^2$ works well when the delays are quite different. In the case the delays are similar, you can try to transform them (e.g. by $\exp$ to increase the difference) and then apply the above function. For example, case 1 is $1/x^2$, case 2 can be $1/\exp(x)$, case 3 something else, etc..
  
  > **Comments:**
  > **username_0**: Ok, I think I get it now ! Thanks for your help !  
  
  ---
  
  