Body: <p>Let's say our task is to pick and place a block, like: <a href="https://gym.openai.com/envs/FetchPickAndPlace-v0/" rel="nofollow noreferrer">https://gym.openai.com/envs/FetchPickAndPlace-v0/</a></p>
  
  <p>Reward function 1: -1 for block not placed, 0 for block placed</p>
  
  <p>Reward function 2: 0 for block not placed, +1 for block placed</p>
  
  <p>I noticed training 1 is much faster than 2... I am using the HER implementation from OpenAI. Why is that?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-05T08:07:39.873Z
Comments.Text: Could you please clarify when the rewards are assigned? It makes a big difference to most RL algorithms whether the "block not placed" reward is given on every time step or just at the end as an assssment.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-05T12:07:49.313Z
Comments.Text: And also, it seems that 1st option has a priori denser rewards. I'm not familiar with the HER implementation, apart from the experience replay, does anyone know if it centers the expected reward somehow? (State functions or GAE)
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-07T04:31:21.733Z
Comments.Text: These are given at each time step. The 'density' aka frequency of each type of reward is the same for each reward function.
Comments.UserDisplayName: user3180
Tags: <reinforcement-learning>
Title: Why do these reward functions give different training curves?
UserDisplayName: user3180
ThreadText: # Why do these reward functions give different training curves?
  
  (Asked by: username_0 on 2019-08-05)
  
  Let's say our task is to pick and place a block, like: <https://gym.openai.com/envs/FetchPickAndPlace-v0/>
  
  Reward function 1: -1 for block not placed, 0 for block placed
  
  Reward function 2: 0 for block not placed, +1 for block placed
  
  I noticed training 1 is much faster than 2... I am using the HER implementation from OpenAI. Why is that?
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: Could you please clarify when the rewards are assigned? It makes a big difference to most RL algorithms whether the "block not placed" reward is given on every time step or just at the end as an assssment.  
  > **username_2**: And also, it seems that 1st option has a priori denser rewards. I'm not familiar with the HER implementation, apart from the experience replay, does anyone know if it centers the expected reward somehow? (State functions or GAE)  
  > **username_0**: These are given at each time step. The 'density' aka frequency of each type of reward is the same for each reward function.  
  
  ---
  
  ## 0 answers
  
  