Answers.Body: <p>Meanwhile I can answer my own question.
  After figuring it out, what the HER algorithm really needs, I found out, that the right answer is 1: you need a reward function for calculating the new reward and the new <em>done</em> flag.</p>
  <p>In case you are at the first tries and you are trying to implement it using the panda-gym environment, this is what I did:</p>
  <p>Inside the Panda environment there is a <code>compute_reward</code> function, which can be used to calculate the reward. Just pass the <em>actual state</em> and the <em>new goal</em> to that function.</p>
  <p>In code:</p>
  <pre><code>import gymnasium as gyms import panda_gym
  
  ... 
  env = gyms.make('PandaReach-v3')
  
  ...
  
  # The compute_reward functions need an 'info' dictionary as third argument.
  # Just pass an empty dictionary there.
  # 'next_state' should replace with the variable you need. Since in my case
  # I want to calculate the difference between 3 spatial coordinates, so I'm going to take 'next_state' and 'new_goal'. Both are 3D spatial coordinates 
  new_reward = env.unwrapped.compute_reward(next_state, new_goal, {})
  # Or in case you are not using any wrapper new_reward = 
  env.computer_reward(next_state, new_goal, {})
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-19T13:39:58.880Z
Answers.LastActivityDate: 2024-03-19T13:39:58.880Z
Answers.UserDisplayName: Dave
Body: <p>I' m try to implement the HER algorithm from scratch in order to use it in the <a href="https://panda-gym.readthedocs.io/en/latest/usage/advanced_rendering.html" rel="nofollow noreferrer">PandaReach-v3</a> environment.
  I already developed the same algorithm for the bitflip environment and it works as expected.
  So, what's now the problem?
  The problem is the calculation of the new reward <span class="math-container">$r'$</span> given an additional <em>goal</em> as stated in the red box in the following picture:</p>
  <p><a href="https://i.sstatic.net/W2M7q.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/W2M7q.png" alt="enter image description here" /></a></p>
  <p>The problem is, that while in the bitflip environment I just created a trivial function, that basically calculates the difference between the <em>state</em> <span class="math-container">$s_{t}$</span> and the <em>goal</em> <span class="math-container">$g'$</span>, in the case of a more complex problem like the PandaReach I don't kwon how to proceed, since I cannot figure out, how the function for the calculation of the new reward given a  new goal should look like.</p>
  <p>Basically I came up with the following two ideas:</p>
  <ol>
  <li><p>I could implement a function, which calculates the 3D distance between the <em>state</em> and the <em>goal</em>. The function could output a <em>one</em>, if the calculated distance is below a small <span class="math-container">$\epsilon$</span> or <em>zero</em> otherwise;</p>
  </li>
  <li><p>I could use the already implemented <em>step</em> method (from the environment), which delivers in one step the new calculated reward. So no effort from my side. But here the problem is that I need to pass an action to the <em>step</em> method in order to generate a new <em>state</em> and calculate the <em>new reward</em>. And, for a new action, I need a policy. Since HER is a off policy algorithm I have a behavioural policy <span class="math-container">$\pi_{b}$</span> and nothing else. So my concern is whether I can/should/may use the same behavioural policy to sample an action to feed into the <em>step</em> method and get the <em>new reward</em>.</p>
  </li>
  </ol>
  <p>In other words: how do you calculate the new reward <span class="math-container">$r'$</span> in the HER algorithm?</p>
  <p>Many thanks</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-19T11:25:53.527Z
Comments.Text: HER typically assumes you have access to a reward function that can evaluate whether a certain state achieves the goal or not. If you can implement 1. then that should work.
Comments.UserDisplayName: David
Tags: <reward-functions><off-policy-methods><hindsight-experience-replay><sparse-rewards>
Title: A question about the reward calculation in the Hindsight Experience Replay algorithm
UserDisplayName: Dave
ThreadText: # A question about the reward calculation in the Hindsight Experience Replay algorithm
  
  (Asked by: username_0 on 2024-03-18)
  
  I' m try to implement the HER algorithm from scratch in order to use it in the [PandaReach-v3](https://panda-gym.readthedocs.io/en/latest/usage/advanced_rendering.html) environment.
  I already developed the same algorithm for the bitflip environment and it works as expected.
  So, what's now the problem?
  The problem is the calculation of the new reward $r'$ given an additional *goal* as stated in the red box in the following picture:
  
  [![enter image description here](https://i.sstatic.net/W2M7q.png)](https://i.sstatic.net/W2M7q.png)
  
  The problem is, that while in the bitflip environment I just created a trivial function, that basically calculates the difference between the *state* $s\_{t}$ and the *goal* $g'$, in the case of a more complex problem like the PandaReach I don't kwon how to proceed, since I cannot figure out, how the function for the calculation of the new reward given a new goal should look like.
  
  Basically I came up with the following two ideas:
  
  1. I could implement a function, which calculates the 3D distance between the *state* and the *goal*. The function could output a *one*, if the calculated distance is below a small $\epsilon$ or *zero* otherwise;
  2. I could use the already implemented *step* method (from the environment), which delivers in one step the new calculated reward. So no effort from my side. But here the problem is that I need to pass an action to the *step* method in order to generate a new *state* and calculate the *new reward*. And, for a new action, I need a policy. Since HER is a off policy algorithm I have a behavioural policy $\pi\_{b}$ and nothing else. So my concern is whether I can/should/may use the same behavioural policy to sample an action to feed into the *step* method and get the *new reward*.
  
  In other words: how do you calculate the new reward $r'$ in the HER algorithm?
  
  Many thanks
  
  **Tags:** reward-functions, off-policy-methods, hindsight-experience-replay, sparse-rewards
  > **Comments:**
  > **username_1**: HER typically assumes you have access to a reward function that can evaluate whether a certain state achieves the goal or not. If you can implement 1. then that should work.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2024-03-19. Score: 1 | Accepted answer
  
  Meanwhile I can answer my own question.
  After figuring it out, what the HER algorithm really needs, I found out, that the right answer is 1: you need a reward function for calculating the new reward and the new done flag.
  In case you are at the first tries and you are trying to implement it using the panda-gym environment, this is what I did:
  Inside the Panda environment there is a compute_reward function, which can be used to calculate the reward. Just pass the actual state and the new goal to that function.
  In code:
  import gymnasium as gyms import panda_gym
  
  ... 
  env = gyms.make('PandaReach-v3')
  
  ...
  
  # The compute_reward functions need an 'info' dictionary as third argument.
  # Just pass an empty dictionary there.
  # 'next_state' should replace with the variable you need. Since in my case
  # I want to calculate the difference between 3 spatial coordinates, so I'm going to take 'next_state' and 'new_goal'. Both are 3D spatial coordinates 
  new_reward = env.unwrapped.compute_reward(next_state, new_goal, {})
  # Or in case you are not using any wrapper new_reward = 
  env.computer_reward(next_state, new_goal, {})
  
  
  ---
  
  