Answers.Body: <p>You cannot really do that because you have no way of knowing how good the action really is to make reasonable labels for supervised learning (that's the whole point why we need reinforcement learning). The only way to possibly know that is to make labels based on the return that you got from that action but the return is based on an old trajectory with the old policy. The return for that specific action depends on actions that happened after that action in the trajectory and return for those actions change with time.</p>
  
  <p>To make things clearer, consider a simple case. Let's say you take action <span class="math-container">$a_1$</span> and you end up in state <span class="math-container">$s_1$</span> with reward <span class="math-container">$0$</span>. Then you have two possibilities, you take action <span class="math-container">$a_2$</span> and end up in terminal state <span class="math-container">$s_2$</span> with reward <span class="math-container">$-10$</span> or you take action <span class="math-container">$a_2'$</span> and end up in terminal state <span class="math-container">$s_2'$</span> with reward <span class="math-container">$10$</span>. Let's say you use trajectory <span class="math-container">$a_1 \rightarrow s_1 \rightarrow a_2 \rightarrow s_2$</span> with return <span class="math-container">$-10$</span> to learn about action <span class="math-container">$a_1$</span>. Then your label for that action would probably be that that action is bad, but it actually isn't, if you took action <span class="math-container">$a_2'$</span> after <span class="math-container">$a_1$</span> your return for action <span class="math-container">$a_1$</span> would be <span class="math-container">$10$</span>. So you learned that your action is bad even though it might not be. Now, if later you learn that taking action <span class="math-container">$a_2'$</span> is good to take after <span class="math-container">$a_1$</span> then you would also learn that <span class="math-container">$a_1$</span> might be good but if you keep using that old data with return <span class="math-container">$-10$</span> you will keep learning that <span class="math-container">$a_1$</span> is bad.</p>
  
  <p>You can only use data gathered from the current policy to learn about it because older data might be outdated.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-18T14:46:45.650Z
Answers.Text: Dear @Brale_, thank you very much for the very detailed and helpful answer. It was great to point out, that we actually do not have a label on how good an action really is. The Advantage function seems to be the best estimate of how good an action is?
Answers.UserDisplayName: Ray Walker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-20T16:06:00.343Z
Answers.Text: could we use the immediate reward as a (very) noisy estimate of how good an action really is? Or the return, put all experiences in a replay buffer and sample trajectories from older policies with smaller probability, e.g. exponentially decaying?
Answers.UserDisplayName: Ray Walker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-12T12:27:35.177Z
Answers.LastActivityDate: 2020-04-12T13:34:43.497Z
Answers.LastEditDate: 2020-04-12T13:34:43.497Z
Answers.UserDisplayName: Brale
Body: <p>I believe to understand the reason why <strong>on-policy methods cannot reuse trajectories collected from earlier policies</strong>: the trajectory distribution change with the policy and the policy gradient is derived to be an expectation over these trajectories. </p>
  
  <p>Doesn't the following intuition from the <a href="https://spinningup.openai.com/en/latest/algorithms/vpg.html" rel="nofollow noreferrer">OpenAI Vanilla Policy Gradient description</a> indeed propose that learning from prior experience <strong>should still be possible</strong>?</p>
  
  <blockquote>
    <p>The key idea underlying policy gradients is to push up the probabilities of actions that lead to higher return, and push down the probabilities of actions that lead to lower return.</p>
  </blockquote>
  
  <p>The goal is to change the probabilities of actions. Actions sampled from previous policies are still possible under the current one. </p>
  
  <p>I see that we cannot reuse the previous actions to estimate the policy gradient. But couldn't we <strong>update the policy network with previous trajectories using supervised learning?</strong> The labels for the actions would be between 0 and 1 based on how good an action was. In the simplest case, just 1 for good actions and 0 for bad ones. The loss could be a simple sum of squared differences with a regularization term.</p>
  
  <p>Why is that not used/possible? What am I missing?</p>
  
Tags: <reinforcement-learning><policy-gradients><experience-replay><off-policy-methods><on-policy-methods>
Title: Could we update the policy network with previous trajectories using supervised learning?
UserDisplayName: Ray Walker
ThreadText: # Could we update the policy network with previous trajectories using supervised learning?
  
  (Asked by: username_0 on 2020-04-12)
  
  I believe to understand the reason why **on-policy methods cannot reuse trajectories collected from earlier policies**: the trajectory distribution change with the policy and the policy gradient is derived to be an expectation over these trajectories.
  
  Doesn't the following intuition from the [OpenAI Vanilla Policy Gradient description](https://spinningup.openai.com/en/latest/algorithms/vpg.html) indeed propose that learning from prior experience **should still be possible**?
  
  > The key idea underlying policy gradients is to push up the probabilities of actions that lead to higher return, and push down the probabilities of actions that lead to lower return.
  
  The goal is to change the probabilities of actions. Actions sampled from previous policies are still possible under the current one.
  
  I see that we cannot reuse the previous actions to estimate the policy gradient. But couldn't we **update the policy network with previous trajectories using supervised learning?** The labels for the actions would be between 0 and 1 based on how good an action was. In the simplest case, just 1 for good actions and 0 for bad ones. The loss could be a simple sum of squared differences with a regularization term.
  
  Why is that not used/possible? What am I missing?
  
  **Tags:** reinforcement-learning, policy-gradients, experience-replay, off-policy-methods, on-policy-methods
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-12. Score: 2 | Accepted answer
  
  You cannot really do that because you have no way of knowing how good the action really is to make reasonable labels for supervised learning (that's the whole point why we need reinforcement learning). The only way to possibly know that is to make labels based on the return that you got from that action but the return is based on an old trajectory with the old policy. The return for that specific action depends on actions that happened after that action in the trajectory and return for those actions change with time.
  
  To make things clearer, consider a simple case. Let's say you take action $a\_1$ and you end up in state $s\_1$ with reward $0$. Then you have two possibilities, you take action $a\_2$ and end up in terminal state $s\_2$ with reward $-10$ or you take action $a\_2'$ and end up in terminal state $s\_2'$ with reward $10$. Let's say you use trajectory $a\_1 \rightarrow s\_1 \rightarrow a\_2 \rightarrow s\_2$ with return $-10$ to learn about action $a\_1$. Then your label for that action would probably be that that action is bad, but it actually isn't, if you took action $a\_2'$ after $a\_1$ your return for action $a\_1$ would be $10$. So you learned that your action is bad even though it might not be. Now, if later you learn that taking action $a\_2'$ is good to take after $a\_1$ then you would also learn that $a\_1$ might be good but if you keep using that old data with return $-10$ you will keep learning that $a\_1$ is bad.
  
  You can only use data gathered from the current policy to learn about it because older data might be outdated.
  
  > **Comments:**
  > **username_0**: Dear @Brale_, thank you very much for the very detailed and helpful answer. It was great to point out, that we actually do not have a label on how good an action really is. The Advantage function seems to be the best estimate of how good an action is?  
  > **username_0**: could we use the immediate reward as a (very) noisy estimate of how good an action really is? Or the return, put all experiences in a replay buffer and sample trajectories from older policies with smaller probability, e.g. exponentially decaying?  
  
  ---
  
  