Body: <p>Assume the following scenario:</p>
  <blockquote>
  <p>We have an agent that acts on an environment where the agent should never take an action that results in him leaving the environment. For example, imagine an agent playing tennis and he should never run over the lines that determine the field.
  However, it may occur that the agent has to cross the lines in order to hit the ball (let's stick to the tennis game). In that case he would get a reward for hitting the ball and a penalty for leaving the arena.</p>
  </blockquote>
  <p>I wonder what approach would be better:</p>
  <p>Only give the agent a reward while he is in the arena, so in our example he would not get any reward while outside of the arena.</p>
  <p>Or sum up the penalty and reward, so the agent always gets the reward, regardless if he hurt the constraints or not.</p>
  <p>I thought that giving him the reward, even when outside of the arena, could encourage him to keep hurting the constraints. On the other hand, I fear that not giving the reward, he would miss out on learning how to hit the ball.</p>
  <p>Which approach would be superior?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-17T17:20:32.770Z
Comments.Text: the sign of the reward should tell the agent if it was something wanted or not, and the magnitude how wanted or not it is (then there are more advanced concepts like advantage function that changes this, but still)... the reward function shaping is a very important part as it's the "supervision" induced to the agent, thus if hitting a ball has a reward bigger than the penalty for the line, the agent will learn that it's better to hit the ball (and yes, theoretically you should sum... if you will 5 dollars and you loose 3 dollars, it's the same as winning 2 dollars)
Comments.UserDisplayName: Alberto
Tags: <reinforcement-learning><reward-functions><reward-design>
Title: How to handle penalty and reward occurring simultaneously
UserDisplayName: kklaw
ThreadText: # How to handle penalty and reward occurring simultaneously
  
  (Asked by: username_0 on 2023-07-16)
  
  Assume the following scenario:
  
  > We have an agent that acts on an environment where the agent should never take an action that results in him leaving the environment. For example, imagine an agent playing tennis and he should never run over the lines that determine the field.
  > However, it may occur that the agent has to cross the lines in order to hit the ball (let's stick to the tennis game). In that case he would get a reward for hitting the ball and a penalty for leaving the arena.
  
  I wonder what approach would be better:
  
  Only give the agent a reward while he is in the arena, so in our example he would not get any reward while outside of the arena.
  
  Or sum up the penalty and reward, so the agent always gets the reward, regardless if he hurt the constraints or not.
  
  I thought that giving him the reward, even when outside of the arena, could encourage him to keep hurting the constraints. On the other hand, I fear that not giving the reward, he would miss out on learning how to hit the ball.
  
  Which approach would be superior?
  
  **Tags:** reinforcement-learning, reward-functions, reward-design
  > **Comments:**
  > **username_1**: the sign of the reward should tell the agent if it was something wanted or not, and the magnitude how wanted or not it is (then there are more advanced concepts like advantage function that changes this, but still)... the reward function shaping is a very important part as it's the "supervision" induced to the agent, thus if hitting a ball has a reward bigger than the penalty for the line, the agent will learn that it's better to hit the ball (and yes, theoretically you should sum... if you will 5 dollars and you loose 3 dollars, it's the same as winning 2 dollars)  
  
  ---
  
  ## 0 answers
  
  