Answers.Body: <p>Let's say your old policy is <span class="math-container">$\pi_b$</span> and your current one is <span class="math-container">$\pi_a$</span>. If you collected trajectory by using policy <span class="math-container">$\pi_b$</span> you would get return <span class="math-container">$G$</span> whose expected value is
  <span class="math-container">\begin{align}
  E_{\pi_b}[G_t|S_t = s] &amp;= E_{\pi_b}[R_{t+1} + G_{t+1}]\\
  &amp;= \sum_a \pi_b(a|s) \sum_{s', r} p(s', r|s, a) [r + E_{\pi_b}[G_{t+1}|S_{t+1} = s']]\\
  \end{align}</span>
  You can see if you write out this recursively that this expectation depends on <span class="math-container">$\pi_b(a|s), \pi_b(a'|s'), \ldots$</span></p>
  
  <p>If you collect trajectory with policy <span class="math-container">$\pi_a$</span> you would get expected return that depends on <span class="math-container">$\pi_a(a|s), \pi_a(a'|s'), \ldots$</span> Since these are two different policies then <span class="math-container">$\pi_b(a|s) \neq \pi_a(a|s)$</span> for some <span class="math-container">$(s, a)$</span>. That would mean that returns have different expected values and are sampled through different distributions. You cannot then use some return <span class="math-container">$G$</span> sampled by following policy <span class="math-container">$\pi_b$</span> to update policy <span class="math-container">$\pi_a$</span> because it's not sampled according to the proper distribution, if we did, we would be updating policy <span class="math-container">$\pi_a$</span> with biased gradient update that does not reflect how policy <span class="math-container">$\pi_a$</span> performed.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-15T19:57:35.183Z
Answers.Text: Or in more plain, non-mathematical terms; if you're traing on old examples stored in some replay buffer, which were created by another / an older policy, you're **by definition** doing off-policy learning!
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-15T19:58:06.513Z
Answers.Text: Ok. I am working with the Max Lapan's book, i.e. *Deep Reinforcement Learning Hands-on* on the subject of A2C. I tried to build my own implementation of the Atari Pong game using that algorithm and based on Max code, but my code is not converging at all. I'm struggling on that converging issue for a week now.  I think there's a very subtle error in the code, but I can't say what it is. Here is the question: https://ai.stackexchange.com/q/21187/2444. Can you help me with that?
Answers.UserDisplayName: jgauth
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-15T20:06:31.593Z
Answers.Text: Both code are running in a different Colab Env. You can probably see a very significant difference in the results ... right now.
Answers.UserDisplayName: jgauth
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-15T22:05:56.987Z
Answers.Text: I'm not sure if I will have time to look into code and I'm also not too familiar with Pytorch. If the other code is similar and it works try to change those lines that are different one portion at the time to try to spot the error. So my advice would be to try to isolate portions of the code and make sure they work and that they provide same behavior in both cases. To do that easier, you can also set the random seed so that you always initialize with the same parameters in both cases.
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-15T23:00:58.960Z
Answers.Text: Can you develop what you mean by "you can also set the random seed so that you always initialize with the same parameters in both cases" ?
Answers.UserDisplayName: jgauth
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-15T23:55:26.027Z
Answers.Text: for example in numpy you can call `numpy.random.seed` and your random numbers will always follow same pattern every time you run the program.
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-15T19:32:00.710Z
Answers.LastActivityDate: 2020-05-15T19:32:00.710Z
Answers.UserDisplayName: Brale
Body: <blockquote>
  <p>One of the approaches to improving the stability of the Policy
  Gradient family of methods is to use multiple environments in
  parallel. The reason behind this is the fundamental problem we
  discussed in Chapter 6, Deep Q-Network, when we talked about the
  correlation between samples, which breaks the independent and
  identically distributed (i.i.d) assumption, which is critical for
  Stochastic Gradient Descent (SDG) optimization. The negative
  consequence of such correlation is very high variance in gradients,
  which means that our training batch contains very similar examples,
  all of them pushing our network in the same direction. However, this
  may be totally the wrong direction in the global sense, as all those
  examples could be from one single lucky or unlucky episode. With our
  Deep Q-Network (DQN), we solved the issue by storing a large amount of
  previous states in the replay buffer and sampling our training batch
  from this buffer. If the buffer is large enough, the random sample
  from it is much better representation of the states distribution at
  large. Unfortunately, this solution won't work for PG methods, at most
  of them are on-policy, which means that we have to train on samples
  generated by our current policy, so, remembering old transitions is
  not possible anymore.</p>
  </blockquote>
  <p>The above excerpt is from Maxim Lapan in the book <em>Deep Reinforcement Learning Hands-on</em> page 284.</p>
  <p>How does being on-policy prevent us from using the replay buffer with the policy gradients? Can you explain to me mathematically why we can't use replay buffer with A3C for instance?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-15T16:53:20.270Z
Comments.Text: I answered similar question [here](https://ai.stackexchange.com/questions/20189/could-we-update-the-policy-network-with-previous-trajectories-using-supervised-l/20197#20197) you can take a look
Comments.UserDisplayName: Brale
Tags: <reinforcement-learning><policy-gradients><actor-critic-methods><experience-replay><a3c>
Title: How does being on-policy prevent us from using the replay buffer with the policy gradients?
UserDisplayName: jgauth
ThreadText: # How does being on-policy prevent us from using the replay buffer with the policy gradients?
  
  (Asked by: username_0 on 2020-05-12)
  
  > One of the approaches to improving the stability of the Policy
  > Gradient family of methods is to use multiple environments in
  > parallel. The reason behind this is the fundamental problem we
  > discussed in Chapter 6, Deep Q-Network, when we talked about the
  > correlation between samples, which breaks the independent and
  > identically distributed (i.i.d) assumption, which is critical for
  > Stochastic Gradient Descent (SDG) optimization. The negative
  > consequence of such correlation is very high variance in gradients,
  > which means that our training batch contains very similar examples,
  > all of them pushing our network in the same direction. However, this
  > may be totally the wrong direction in the global sense, as all those
  > examples could be from one single lucky or unlucky episode. With our
  > Deep Q-Network (DQN), we solved the issue by storing a large amount of
  > previous states in the replay buffer and sampling our training batch
  > from this buffer. If the buffer is large enough, the random sample
  > from it is much better representation of the states distribution at
  > large. Unfortunately, this solution won't work for PG methods, at most
  > of them are on-policy, which means that we have to train on samples
  > generated by our current policy, so, remembering old transitions is
  > not possible anymore.
  
  The above excerpt is from Maxim Lapan in the book *Deep Reinforcement Learning Hands-on* page 284.
  
  How does being on-policy prevent us from using the replay buffer with the policy gradients? Can you explain to me mathematically why we can't use replay buffer with A3C for instance?
  
  **Tags:** reinforcement-learning, policy-gradients, actor-critic-methods, experience-replay, a3c
  > **Comments:**
  > **username_1**: I answered similar question [here](https://ai.stackexchange.com/questions/20189/could-we-update-the-policy-network-with-previous-trajectories-using-supervised-l/20197#20197) you can take a look  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-15. Score: 5 | Accepted answer
  
  Let's say your old policy is $\pi\_b$ and your current one is $\pi\_a$. If you collected trajectory by using policy $\pi\_b$ you would get return $G$ whose expected value is
  \begin{align}
  E\_{\pi\_b}[G\_t|S\_t = s] &= E\_{\pi\_b}[R\_{t+1} + G\_{t+1}]\\
  &= \sum\_a \pi\_b(a|s) \sum\_{s', r} p(s', r|s, a) [r + E\_{\pi\_b}[G\_{t+1}|S\_{t+1} = s']]\\
  \end{align}
  You can see if you write out this recursively that this expectation depends on $\pi\_b(a|s), \pi\_b(a'|s'), \ldots$
  
  If you collect trajectory with policy $\pi\_a$ you would get expected return that depends on $\pi\_a(a|s), \pi\_a(a'|s'), \ldots$ Since these are two different policies then $\pi\_b(a|s) \neq \pi\_a(a|s)$ for some $(s, a)$. That would mean that returns have different expected values and are sampled through different distributions. You cannot then use some return $G$ sampled by following policy $\pi\_b$ to update policy $\pi\_a$ because it's not sampled according to the proper distribution, if we did, we would be updating policy $\pi\_a$ with biased gradient update that does not reflect how policy $\pi\_a$ performed.
  
  > **Comments:**
  > **username_2**: Or in more plain, non-mathematical terms; if you're traing on old examples stored in some replay buffer, which were created by another / an older policy, you're **by definition** doing off-policy learning!  
  > **username_0**: Ok. I am working with the Max Lapan's book, i.e. *Deep Reinforcement Learning Hands-on* on the subject of A2C. I tried to build my own implementation of the Atari Pong game using that algorithm and based on Max code, but my code is not converging at all. I'm struggling on that converging issue for a week now.  I think there's a very subtle error in the code, but I can't say what it is. Here is the question: https://ai.stackexchange.com/q/21187/2444. Can you help me with that?  
  > **username_0**: Both code are running in a different Colab Env. You can probably see a very significant difference in the results ... right now.  
  > **username_1**: I'm not sure if I will have time to look into code and I'm also not too familiar with Pytorch. If the other code is similar and it works try to change those lines that are different one portion at the time to try to spot the error. So my advice would be to try to isolate portions of the code and make sure they work and that they provide same behavior in both cases. To do that easier, you can also set the random seed so that you always initialize with the same parameters in both cases.  
  > **username_0**: Can you develop what you mean by "you can also set the random seed so that you always initialize with the same parameters in both cases" ?  
  > **username_1**: for example in numpy you can call `numpy.random.seed` and your random numbers will always follow same pattern every time you run the program.  
  
  ---
  
  