Answers.Body: <p>Here is how I managed to construct a reward function in <a href="https://github.com/karray/neuroracer" rel="nofollow noreferrer">one</a> of my projects, where I trained an RL model for a self-driving robot that has only a single camera to navigate through a tunnel:</p>
  <p><span class="math-container">$$
  R = \left\{
  \begin{array}{ll}
  d_m - 3 - \left| d_l - d_r \right| &amp; \text{if not terminal state} \\
  -100 &amp; \text{otherwise}
  \end{array} 
  \right.
  $$</span></p>
  <p>where <span class="math-container">$d_m$</span> is the middle distance, <span class="math-container">$d_l$</span> is the left distance and <span class="math-container">$d_r$</span> is the right distance, and <span class="math-container">$d_m, d_l, d_r \in [0, 10]$</span>. To get this information, the agent has a laser sensor. The agent does not directly observe these distances. Instead, it gets a real number as the reward signal that indicates how good the agent is performing an action and tries to map the camera view to it. This function is designed so that the agent should stay in the middle of the tunnel, <span class="math-container">$-\left|d_l - d_r\right|$</span>, and has to avoid head-on collisions, <span class="math-container">$d_m - 3$</span>. Thus, the highest possible reward is <span class="math-container">$R = 10 - 3 - \left| 10 - 10 \right| = 7$</span></p>
  <p><a href="https://i.sstatic.net/S1pYL.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/S1pYL.png" alt="enter image description here" /></a></p>
  <p>Basically, you can use any number of parameters in your reward function as long as it accurately reflects the goal the agent needs to achieve. For instance, I could penalize the agent for frequent steering on straight sections so that it drives smoothly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-17T15:07:08.100Z
Answers.Text: Worth adding the need to weight any separate reward calculations appropriately before combining. When you have multiple possibly conflicting objectives, it is down to you as system designer to set the rules for which is more important, the RL agent cannot balance them for you
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-17T12:57:49.740Z
Answers.LastActivityDate: 2021-06-17T13:07:57.513Z
Answers.LastEditDate: 2021-06-17T13:07:57.513Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>I found this <a href="https://medium.com/@BonsaiAI/deep-reinforcement-learning-models-tips-tricks-for-writing-reward-functions-a84fe525e8e0" rel="nofollow noreferrer">article</a> quite useful on how to shape a reward function in RL. However, the example they gave is quite simple, where the goal is to minimize only two quantities (velocity and distance).</p>
  <p>How would you formulate the reward function if you had, for instance, 4 quantities to optimize?</p>
  <p><a href="https://i.sstatic.net/UsgmG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/UsgmG.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-17T08:28:49.863Z
Comments.Text: At AI Stack Exchange we prefer questions to be self-contained. This one requires going off site to read the article to even understand the basics of what you are asking. Could you please summarise the article in a couple of short paragraphs - what was the environment, how was the reward function shaped (give the equation, you can use LaTex inside `$` like this `$r = x_{t} - x_{t+1}$` which becomes $r = x_{t} - x_{t+1}$), and how did it help?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-17T09:24:01.953Z
Comments.Text: I mentioned the article just as an example, my question is just that I need to formulate a reward function to optimize 4 parameters in the example that I sent they tried to optimize only 2 parameters as shown in the image that I provided now! Thanks for your reply.
Comments.UserDisplayName: BAKYAC
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-17T12:14:38.400Z
Comments.Text: check out this [answer](https://ai.stackexchange.com/questions/18425/how-can-i-implement-the-reward-function-for-an-8-dof-robot-arm-with-trpo/18436#18436)
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-17T23:51:47.677Z
Comments.Text: @BAKYAC Could you please at least provide more details about your concrete problem? You say that you have 4 quantities to optimize. What quantities/variables are these? What problem are you trying to solve with RL? What is the action and state spaces?
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><deep-rl><reward-functions><reward-design><reward-shaping>
Title: How would you shape a reward function if there was four quantities to optimize?
UserDisplayName: BAKYAC
ThreadText: # How would you shape a reward function if there was four quantities to optimize?
  
  (Asked by: username_0 on 2021-06-17)
  
  I found this [article](https://medium.com/@BonsaiAI/deep-reinforcement-learning-models-tips-tricks-for-writing-reward-functions-a84fe525e8e0) quite useful on how to shape a reward function in RL. However, the example they gave is quite simple, where the goal is to minimize only two quantities (velocity and distance).
  
  How would you formulate the reward function if you had, for instance, 4 quantities to optimize?
  
  [![enter image description here](https://i.sstatic.net/UsgmG.png)](https://i.sstatic.net/UsgmG.png)
  
  **Tags:** reinforcement-learning, deep-rl, reward-functions, reward-design, reward-shaping
  > **Comments:**
  > **username_1**: At AI Stack Exchange we prefer questions to be self-contained. This one requires going off site to read the article to even understand the basics of what you are asking. Could you please summarise the article in a couple of short paragraphs - what was the environment, how was the reward function shaped (give the equation, you can use LaTex inside `$` like this `$r = x_{t} - x_{t+1}$` which becomes $r = x_{t} - x_{t+1}$), and how did it help?  
  > **username_0**: I mentioned the article just as an example, my question is just that I need to formulate a reward function to optimize 4 parameters in the example that I sent they tried to optimize only 2 parameters as shown in the image that I provided now! Thanks for your reply.  
  > **username_2**: check out this [answer](https://ai.stackexchange.com/questions/18425/how-can-i-implement-the-reward-function-for-an-8-dof-robot-arm-with-trpo/18436#18436)  
  > **username_3**: @BAKYAC Could you please at least provide more details about your concrete problem? You say that you have 4 quantities to optimize. What quantities/variables are these? What problem are you trying to solve with RL? What is the action and state spaces?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-06-17. Score: 1
  
  Here is how I managed to construct a reward function in [one](https://github.com/karray/neuroracer) of my projects, where I trained an RL model for a self-driving robot that has only a single camera to navigate through a tunnel:
  
  $$
  R = \left\{
  \begin{array}{ll}
  d\_m - 3 - \left| d\_l - d\_r \right| & \text{if not terminal state} \\
  -100 & \text{otherwise}
  \end{array}
  \right.
  $$
  
  where $d\_m$ is the middle distance, $d\_l$ is the left distance and $d\_r$ is the right distance, and $d\_m, d\_l, d\_r \in [0, 10]$. To get this information, the agent has a laser sensor. The agent does not directly observe these distances. Instead, it gets a real number as the reward signal that indicates how good the agent is performing an action and tries to map the camera view to it. This function is designed so that the agent should stay in the middle of the tunnel, $-\left|d\_l - d\_r\right|$, and has to avoid head-on collisions, $d\_m - 3$. Thus, the highest possible reward is $R = 10 - 3 - \left| 10 - 10 \right| = 7$
  
  [![enter image description here](https://i.sstatic.net/S1pYL.png)](https://i.sstatic.net/S1pYL.png)
  
  Basically, you can use any number of parameters in your reward function as long as it accurately reflects the goal the agent needs to achieve. For instance, I could penalize the agent for frequent steering on straight sections so that it drives smoothly.
  
  > **Comments:**
  > **username_1**: Worth adding the need to weight any separate reward calculations appropriately before combining. When you have multiple possibly conflicting objectives, it is down to you as system designer to set the rules for which is more important, the RL agent cannot balance them for you  
  
  ---
  
  