Answers.Body: <p>Ignoring HER for now the <span class="math-container">$Q$</span> and <span class="math-container">$V$</span> functions operate on states and actions which are part of a Markov decision process which we call <span class="math-container">$M_0$</span>.</p>
  <p>Back to HER, the <span class="math-container">$Q$</span> and <span class="math-container">$V$</span> functions now take a goal as an additional parameter called the goal. We will denote individual goals <span class="math-container">$g_n$</span>, the true goal <span class="math-container">$g_0$</span>, and the set of all goals <span class="math-container">$G$</span>. The set of goals is chosen such that every state matches at least one goal. We create a new MDP <span class="math-container">$M_1 = M_0 \times G$</span> (i.e. a larger MDP composed of multiple copies of <span class="math-container">$M_0$</span>, all the states in each being tagged with a goal). The reward is either +1 or 0 depending on whether or not the goal and the component from <span class="math-container">$M_0$</span> match in some predefined sense. In HER trajectories are collected from the subset of <span class="math-container">$M_1$</span> where <span class="math-container">$g = g_0$</span> and added to the replay buffer. When training the <span class="math-container">$Q$</span> and <span class="math-container">$V$</span> functions we don't only use the original trajectories: we create new ones by substituting new values of <span class="math-container">$g$</span>, which we do so strategically so as to include some trajectories with a positive reward.</p>
  <p>Things to note:</p>
  <ol>
  <li>HER doesn't assign rewards to states or trajectories in <span class="math-container">$M_0$</span>: the reward function used is only defined for <span class="math-container">$M_1$</span></li>
  <li>The performance of HER depends on <span class="math-container">$Q$</span> and <span class="math-container">$V$</span> both being models with some ability to extrapolate to unseen data points; such as neural networks, support vector machines, etc. It would not provide any benefit if applied to value tables.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-27T15:55:09.373Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/116726/discussion-on-answer-by-jeremy-list-how-does-hindsight-experience-replay-learn-f).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-19T05:09:33.353Z
Answers.LastActivityDate: 2020-11-27T15:54:58.027Z
Answers.LastEditDate: 2020-11-27T15:54:58.027Z
Answers.UserDisplayName: Jeremy List
Body: <p>I am confused by how HER learns from unsuccessful trajectories. I understand that from failed trajectories it creates 'fake' goals that it can learn from.</p>
  <p>Ignoring HER for now, if in the case where the robotic arm reaches the goal correctly, then the value functions (<span class="math-container">$V$</span>) and action-value functions (<span class="math-container">$Q$</span>) that correspond to the trajectories that get to the goal quicker will increase. These high <span class="math-container">$Q$</span> and <span class="math-container">$V$</span> values are ultimately important for getting the optimal policy.</p>
  <p>However, if you create 'fake' goals from unsuccessful trajectories - that would increase the <span class="math-container">$Q$</span> and <span class="math-container">$V$</span>s of the environment that lead to getting the 'fake' goal. Those new <span class="math-container">$Q$</span> and <span class="math-container">$V$</span>s would be unhelpful and possibly detrimental for the robotic arm to reach the real goal.</p>
  <p>What am I misunderstanding?</p>
  
Tags: <reinforcement-learning><deep-rl><hindsight-experience-replay>
Title: How does Hindsight Experience Replay learn from unsuccessful trajectories?
UserDisplayName: piccolo
ThreadText: # How does Hindsight Experience Replay learn from unsuccessful trajectories?
  
  (Asked by: username_0 on 2018-11-14)
  
  I am confused by how HER learns from unsuccessful trajectories. I understand that from failed trajectories it creates 'fake' goals that it can learn from.
  
  Ignoring HER for now, if in the case where the robotic arm reaches the goal correctly, then the value functions ($V$) and action-value functions ($Q$) that correspond to the trajectories that get to the goal quicker will increase. These high $Q$ and $V$ values are ultimately important for getting the optimal policy.
  
  However, if you create 'fake' goals from unsuccessful trajectories - that would increase the $Q$ and $V$s of the environment that lead to getting the 'fake' goal. Those new $Q$ and $V$s would be unhelpful and possibly detrimental for the robotic arm to reach the real goal.
  
  What am I misunderstanding?
  
  **Tags:** reinforcement-learning, deep-rl, hindsight-experience-replay
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-09-19. Score: 1
  
  Ignoring HER for now the $Q$ and $V$ functions operate on states and actions which are part of a Markov decision process which we call $M\_0$.
  
  Back to HER, the $Q$ and $V$ functions now take a goal as an additional parameter called the goal. We will denote individual goals $g\_n$, the true goal $g\_0$, and the set of all goals $G$. The set of goals is chosen such that every state matches at least one goal. We create a new MDP $M\_1 = M\_0 \times G$ (i.e. a larger MDP composed of multiple copies of $M\_0$, all the states in each being tagged with a goal). The reward is either +1 or 0 depending on whether or not the goal and the component from $M\_0$ match in some predefined sense. In HER trajectories are collected from the subset of $M\_1$ where $g = g\_0$ and added to the replay buffer. When training the $Q$ and $V$ functions we don't only use the original trajectories: we create new ones by substituting new values of $g$, which we do so strategically so as to include some trajectories with a positive reward.
  
  Things to note:
  
  1. HER doesn't assign rewards to states or trajectories in $M\_0$: the reward function used is only defined for $M\_1$
  2. The performance of HER depends on $Q$ and $V$ both being models with some ability to extrapolate to unseen data points; such as neural networks, support vector machines, etc. It would not provide any benefit if applied to value tables.
  
  > **Comments:**
  > **username_2**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/116726/discussion-on-answer-by-jeremy-list-how-does-hindsight-experience-replay-learn-f).  
  
  ---
  
  