Answers.Body: <p>Yes, you need to add goal state as an input otherwise you won't know what goal you are trying to pursue. It might take a lot of memory to run HER on ATARI games though, also defining final goal is not straightforward for such environments</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-20T18:41:42.200Z
Answers.Text: Maybe you should clarify why "defining final goal is not straightforward for such environments". What goal are you talking about? Are you talking about the fake goals that are needed to augment the ER?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-13T19:36:57.600Z
Answers.LastActivityDate: 2020-03-13T19:36:57.600Z
Answers.UserDisplayName: Brale
Body: <p>I was reading the following research paper <a href="https://arxiv.org/pdf/1707.01495.pdf" rel="nofollow noreferrer">Hindsight Experience Replay</a>. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the infamous sparse reward problem. It is based on the intuition that human beings constantly try and learn something useful even from their past failed experiences.</p>
  <p>I have almost completely understood the concept. But in the algorithm posited in the paper, I don't really understand how the optimization works. Once the fictitious trajectories are added, we have a state-goal-action dependency. This means our DQN should predict Q-Values based on an input state and the goal we're pursuing (The paper mentions how HER is extremely useful for multi-RL as well).</p>
  <p>Does this mean I need to add another input feature (goal) to my DQN? An input state and an input goal, as two input features to my DQN, which is basically a CNN?</p>
  <p>Because in the optimization step they have mentioned that we need to randomly sample trajectories from the replay buffer and use those for computing the gradients. It wouldn't make sense to compute the Q-Values without the goal now, because then we'd wind up with duplicate values.</p>
  <p>Could someone help me understand how exactly does the optimization take place here?</p>
  <p>I am training Atari's &quot;Montezuma's Revenge&quot; using a double DQN with Hindsight Experience Replay (HER).</p>
  
Tags: <reinforcement-learning><dqn><deep-rl><sparse-rewards><hindsight-experience-replay>
Title: How does the optimization process in hindsight experience replay exactly work?
UserDisplayName: vikram71198
ThreadText: # How does the optimization process in hindsight experience replay exactly work?
  
  (Asked by: username_0 on 2020-03-12)
  
  I was reading the following research paper [Hindsight Experience Replay](https://arxiv.org/pdf/1707.01495.pdf). This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the infamous sparse reward problem. It is based on the intuition that human beings constantly try and learn something useful even from their past failed experiences.
  
  I have almost completely understood the concept. But in the algorithm posited in the paper, I don't really understand how the optimization works. Once the fictitious trajectories are added, we have a state-goal-action dependency. This means our DQN should predict Q-Values based on an input state and the goal we're pursuing (The paper mentions how HER is extremely useful for multi-RL as well).
  
  Does this mean I need to add another input feature (goal) to my DQN? An input state and an input goal, as two input features to my DQN, which is basically a CNN?
  
  Because in the optimization step they have mentioned that we need to randomly sample trajectories from the replay buffer and use those for computing the gradients. It wouldn't make sense to compute the Q-Values without the goal now, because then we'd wind up with duplicate values.
  
  Could someone help me understand how exactly does the optimization take place here?
  
  I am training Atari's "Montezuma's Revenge" using a double DQN with Hindsight Experience Replay (HER).
  
  **Tags:** reinforcement-learning, dqn, deep-rl, sparse-rewards, hindsight-experience-replay
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-13. Score: 0
  
  Yes, you need to add goal state as an input otherwise you won't know what goal you are trying to pursue. It might take a lot of memory to run HER on ATARI games though, also defining final goal is not straightforward for such environments
  
  > **Comments:**
  > **username_2**: Maybe you should clarify why "defining final goal is not straightforward for such environments". What goal are you talking about? Are you talking about the fake goals that are needed to augment the ER?  
  
  ---
  
  