Answers.Body: <p>Page 6 of the paper describes the exact reward functions, and why they were used:</p>
  
  <blockquote>
    <p>Goals: Goals describe the desired position of the object (a box or a
    puck depending on the task) with some fixed tolerance of <span class="math-container">$\epsilon$</span> i.e. <span class="math-container">$G = \mathcal{R}^3$</span>
     and <span class="math-container">$f_g(s) = [|g − s_{object}| ≤ \epsilon]$</span>, where <span class="math-container">$s_{object}$</span> is the position of
    the object in the state s. The mapping from states to goals used in
    HER is simply <span class="math-container">$m(s) = s_{object}$</span>. </p>
    
    <p>Rewards: Unless stated otherwise we use binary and sparse rewards <span class="math-container">$r(s, a, g) = −[f_g(s 0 ) = 0]$</span> where <span class="math-container">$s'$</span> is
    the state after the execution of the action a in the state s. We
    compare sparse and shaped reward functions in Sec. 4.4.</p>
  </blockquote>
  
  <p>So, at least in the base version (which I believe is your <code>fetchreach-v1</code>), the agent receives a reward of -1 for every timestep spent more than <span class="math-container">$\epsilon$</span> from the goal state, and a reward of 0 for every timestep spent within <span class="math-container">$\epsilon$</span> of the goal state. Thus, a score of -5.0 would seem to correspond to the agent moving directly to the goal and staying there, while a score of -50.0 would correspond to the agent failing to reach the goal state entirely.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-30T12:35:41.897Z
Answers.Text: Thank you, this makes a lot of sense. However, how does this qualify as a sparse reward if it is given at every time-step? Shouldn't the sparse (and binary) reward be given at the end of the episode indicating whether or not the task is completed?
Answers.UserDisplayName: rrz0
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-30T13:12:03.560Z
Answers.Text: @Rrz0 The authors have chosen to present this in a slightly confusing way, but maybe it's easier to think about this as an affine transformation of a sparse reward function? Imagine we chose to add 1 to every reward (so 0 reward for non-goal steps, and 1 for being in a goal). The optimal policy would be unchanged, so the agent will learn the same behaviors. However, the output would be a little harder to understand, because it isn't clear that a score of 50 is "good" (50 is a magic number). Personally, I might have just awarded 1/(max_steps) for each step in the right spot.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-30T12:20:10.830Z
Answers.LastActivityDate: 2019-10-30T12:20:10.830Z
Answers.UserDisplayName: John Doucette
Body: <p>In OpenAI Gym &quot;reward&quot; is defined as:</p>
  <blockquote>
  <p><code>reward</code> (float): amount of reward achieved by the previous action. The
  scale varies between environments, but the goal is always to increase
  your total reward.</p>
  </blockquote>
  <p>I am training Hindsight Experience Replay on Fetch robotics environments, where rewards are sparse and binary indicating whether or not the task is completed. The <a href="https://arxiv.org/pdf/1707.01495.pdf" rel="nofollow noreferrer">original paper</a> implementing HER uses <strong>success rate</strong> as a metric in its plots, like so:</p>
  <p><a href="https://i.sstatic.net/ryiyZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ryiyZ.png" alt="enter image description here" /></a></p>
  <p>On page 5 of the original paper, it is stated that the reward is binary and sparse.</p>
  <p>When I print the rewards obtained during a simulation of <code>FetchReach-v1</code> trained with HER, I get the following values. The first column shows the reward and the second column shows the episode length.</p>
  <p><a href="https://i.sstatic.net/t5cCj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/t5cCj.png" alt="enter image description here" /></a></p>
  <p>As can be seen, at every time step, I am getting a reward, sometimes I get a <span class="math-container">$-1$</span> reward at every time step throughout the episode for a total of <span class="math-container">$-50$</span>. The maximum reward I can achieve throughout the episode is <span class="math-container">$0$</span>.</p>
  <p>Therefore my question is: What is the reward obtained at each time-step? What does it represent and how is this different from the success rate?</p>
  
Tags: <reinforcement-learning><terminology><papers><reward-functions><hindsight-experience-replay>
Title: What is the difference between success rate and reward when dealing with binary and sparse rewards?
UserDisplayName: rrz0
ThreadText: # What is the difference between success rate and reward when dealing with binary and sparse rewards?
  
  (Asked by: username_0 on 2019-10-30)
  
  In OpenAI Gym "reward" is defined as:
  
  > `reward` (float): amount of reward achieved by the previous action. The
  > scale varies between environments, but the goal is always to increase
  > your total reward.
  
  I am training Hindsight Experience Replay on Fetch robotics environments, where rewards are sparse and binary indicating whether or not the task is completed. The [original paper](https://arxiv.org/pdf/1707.01495.pdf) implementing HER uses **success rate** as a metric in its plots, like so:
  
  [![enter image description here](https://i.sstatic.net/ryiyZ.png)](https://i.sstatic.net/ryiyZ.png)
  
  On page 5 of the original paper, it is stated that the reward is binary and sparse.
  
  When I print the rewards obtained during a simulation of `FetchReach-v1` trained with HER, I get the following values. The first column shows the reward and the second column shows the episode length.
  
  [![enter image description here](https://i.sstatic.net/t5cCj.png)](https://i.sstatic.net/t5cCj.png)
  
  As can be seen, at every time step, I am getting a reward, sometimes I get a $-1$ reward at every time step throughout the episode for a total of $-50$. The maximum reward I can achieve throughout the episode is $0$.
  
  Therefore my question is: What is the reward obtained at each time-step? What does it represent and how is this different from the success rate?
  
  **Tags:** reinforcement-learning, terminology, papers, reward-functions, hindsight-experience-replay
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-10-30. Score: 1 | Accepted answer
  
  Page 6 of the paper describes the exact reward functions, and why they were used:
  
  > Goals: Goals describe the desired position of the object (a box or a
  > puck depending on the task) with some fixed tolerance of $\epsilon$ i.e. $G = \mathcal{R}^3$
  > and $f\_g(s) = [|g − s\_{object}| ≤ \epsilon]$, where $s\_{object}$ is the position of
  > the object in the state s. The mapping from states to goals used in
  > HER is simply $m(s) = s\_{object}$.
  >
  > Rewards: Unless stated otherwise we use binary and sparse rewards $r(s, a, g) = −[f\_g(s 0 ) = 0]$ where $s'$ is
  > the state after the execution of the action a in the state s. We
  > compare sparse and shaped reward functions in Sec. 4.4.
  
  So, at least in the base version (which I believe is your `fetchreach-v1`), the agent receives a reward of -1 for every timestep spent more than $\epsilon$ from the goal state, and a reward of 0 for every timestep spent within $\epsilon$ of the goal state. Thus, a score of -5.0 would seem to correspond to the agent moving directly to the goal and staying there, while a score of -50.0 would correspond to the agent failing to reach the goal state entirely.
  
  > **Comments:**
  > **username_0**: Thank you, this makes a lot of sense. However, how does this qualify as a sparse reward if it is given at every time-step? Shouldn't the sparse (and binary) reward be given at the end of the episode indicating whether or not the task is completed?  
  > **username_1**: @username_0 The authors have chosen to present this in a slightly confusing way, but maybe it's easier to think about this as an affine transformation of a sparse reward function? Imagine we chose to add 1 to every reward (so 0 reward for non-goal steps, and 1 for being in a goal). The optimal policy would be unchanged, so the agent will learn the same behaviors. However, the output would be a little harder to understand, because it isn't clear that a score of 50 is "good" (50 is a magic number). Personally, I might have just awarded 1/(max_steps) for each step in the right spot.  
  
  ---
  
  