<p>Page 6 of the paper describes the exact reward functions, and why they were used:</p>

<blockquote>
  <p>Goals: Goals describe the desired position of the object (a box or a
  puck depending on the task) with some fixed tolerance of <span class="math-container">$\epsilon$</span> i.e. <span class="math-container">$G = \mathcal{R}^3$</span>
   and <span class="math-container">$f_g(s) = [|g − s_{object}| ≤ \epsilon]$</span>, where <span class="math-container">$s_{object}$</span> is the position of
  the object in the state s. The mapping from states to goals used in
  HER is simply <span class="math-container">$m(s) = s_{object}$</span>. </p>
  
  <p>Rewards: Unless stated otherwise we use binary and sparse rewards <span class="math-container">$r(s, a, g) = −[f_g(s 0 ) = 0]$</span> where <span class="math-container">$s'$</span> is
  the state after the execution of the action a in the state s. We
  compare sparse and shaped reward functions in Sec. 4.4.</p>
</blockquote>

<p>So, at least in the base version (which I believe is your <code>fetchreach-v1</code>), the agent receives a reward of -1 for every timestep spent more than <span class="math-container">$\epsilon$</span> from the goal state, and a reward of 0 for every timestep spent within <span class="math-container">$\epsilon$</span> of the goal state. Thus, a score of -5.0 would seem to correspond to the agent moving directly to the goal and staying there, while a score of -50.0 would correspond to the agent failing to reach the goal state entirely.</p>

CC BY-SA 4.0
2019-10-30T12:35:41.897Z
Thank you, this makes a lot of sense. However, how does this qualify as a sparse reward if it is given at every time-step? Shouldn't the sparse (and binary) reward be given at the end of the episode indicating whether or not the task is completed?
rrz0
CC BY-SA 4.0
2019-10-30T13:12:03.560Z
@Rrz0 The authors have chosen to present this in a slightly confusing way, but maybe it's easier to think about this as an affine transformation of a sparse reward function? Imagine we chose to add 1 to every reward (so 0 reward for non-goal steps, and 1 for being in a goal). The optimal policy would be unchanged, so the agent will learn the same behaviors. However, the output would be a little harder to understand, because it isn't clear that a score of 50 is "good" (50 is a magic number). Personally, I might have just awarded 1/(max_steps) for each step in the right spot.
John Doucette
CC BY-SA 4.0
2019-10-30T12:20:10.830Z
2019-10-30T12:20:10.830Z
John Doucette
<p>In OpenAI Gym &quot;reward&quot; is defined as:</p>
<blockquote>
<p><code>reward</code> (float): amount of reward achieved by the previous action. The
scale varies between environments, but the goal is always to increase
your total reward.</p>
</blockquote>
<p>I am training Hindsight Experience Replay on Fetch robotics environments, where rewards are sparse and binary indicating whether or not the task is completed. The <a href="https://arxiv.org/pdf/1707.01495.pdf" rel="nofollow noreferrer">original paper</a> implementing HER uses <strong>success rate</strong> as a metric in its plots, like so:</p>
<p><a href="https://i.sstatic.net/ryiyZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ryiyZ.png" alt="enter image description here" /></a></p>
<p>On page 5 of the original paper, it is stated that the reward is binary and sparse.</p>
<p>When I print the rewards obtained during a simulation of <code>FetchReach-v1</code> trained with HER, I get the following values. The first column shows the reward and the second column shows the episode length.</p>
<p><a href="https://i.sstatic.net/t5cCj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/t5cCj.png" alt="enter image description here" /></a></p>
<p>As can be seen, at every time step, I am getting a reward, sometimes I get a <span class="math-container">$-1$</span> reward at every time step throughout the episode for a total of <span class="math-container">$-50$</span>. The maximum reward I can achieve throughout the episode is <span class="math-container">$0$</span>.</p>
<p>Therefore my question is: What is the reward obtained at each time-step? What does it represent and how is this different from the success rate?</p>

<reinforcement-learning><terminology><papers><reward-functions><hindsight-experience-replay>
What is the difference between success rate and reward when dealing with binary and sparse rewards?
rrz0
# What is the difference between success rate and reward when dealing with binary and sparse rewards?

(Asked by: username_0 on 2019-10-30)

In OpenAI Gym "reward" is defined as:

> `reward` (float): amount of reward achieved by the previous action. The
> scale varies between environments, but the goal is always to increase
> your total reward.

I am training Hindsight Experience Replay on Fetch robotics environments, where rewards are sparse and binary indicating whether or not the task is completed. The [original paper](https://arxiv.org/pdf/1707.01495.pdf) implementing HER uses **success rate** as a metric in its plots, like so:

[![enter image description here](https://i.sstatic.net/ryiyZ.png)](https://i.sstatic.net/ryiyZ.png)

On page 5 of the original paper, it is stated that the reward is binary and sparse.

When I print the rewards obtained during a simulation of `FetchReach-v1` trained with HER, I get the following values. The first column shows the reward and the second column shows the episode length.

[![enter image description here](https://i.sstatic.net/t5cCj.png)](https://i.sstatic.net/t5cCj.png)

As can be seen, at every time step, I am getting a reward, sometimes I get a $-1$ reward at every time step throughout the episode for a total of $-50$. The maximum reward I can achieve throughout the episode is $0$.

Therefore my question is: What is the reward obtained at each time-step? What does it represent and how is this different from the success rate?

**Tags:** reinforcement-learning, terminology, papers, reward-functions, hindsight-experience-replay


---

## 1 answers

### Answer by username_1 on 2019-10-30. Score: 1 | Accepted answer

Page 6 of the paper describes the exact reward functions, and why they were used:

> Goals: Goals describe the desired position of the object (a box or a
> puck depending on the task) with some fixed tolerance of $\epsilon$ i.e. $G = \mathcal{R}^3$
> and $f\_g(s) = [|g − s\_{object}| ≤ \epsilon]$, where $s\_{object}$ is the position of
> the object in the state s. The mapping from states to goals used in
> HER is simply $m(s) = s\_{object}$.
>
> Rewards: Unless stated otherwise we use binary and sparse rewards $r(s, a, g) = −[f\_g(s 0 ) = 0]$ where $s'$ is
> the state after the execution of the action a in the state s. We
> compare sparse and shaped reward functions in Sec. 4.4.

So, at least in the base version (which I believe is your `fetchreach-v1`), the agent receives a reward of -1 for every timestep spent more than $\epsilon$ from the goal state, and a reward of 0 for every timestep spent within $\epsilon$ of the goal state. Thus, a score of -5.0 would seem to correspond to the agent moving directly to the goal and staying there, while a score of -50.0 would correspond to the agent failing to reach the goal state entirely.

> **Comments:**
> **username_0**: Thank you, this makes a lot of sense. However, how does this qualify as a sparse reward if it is given at every time-step? Shouldn't the sparse (and binary) reward be given at the end of the episode indicating whether or not the task is completed?  
> **username_1**: @Rrz0 The authors have chosen to present this in a slightly confusing way, but maybe it's easier to think about this as an affine transformation of a sparse reward function? Imagine we chose to add 1 to every reward (so 0 reward for non-goal steps, and 1 for being in a goal). The optimal policy would be unchanged, so the agent will learn the same behaviors. However, the output would be a little harder to understand, because it isn't clear that a score of 50 is "good" (50 is a magic number). Personally, I might have just awarded 1/(max_steps) for each step in the right spot.  

---

