Answers.Body: <p><a href="https://stats.stackexchange.com/q/184657/82135">This post</a> contains many answers that describe the difference between on-policy vs. off-policy.</p>
  <p>Your book may be referring to how the current (DQN-based) state-of-the-art (SOTA) algorithms, such as <a href="https://openreview.net/pdf?id=H1Dy---0Z" rel="nofollow noreferrer">Ape-X</a>, <a href="https://openreview.net/pdf?id=r1lyTjAqYX" rel="nofollow noreferrer">R2D2</a>, <a href="https://deepmind.com/blog/article/Agent57-Outperforming-the-human-Atari-benchmark" rel="nofollow noreferrer">Agent57</a> are technically &quot;off-policy&quot;, since they use a (very large!) replay buffer, often filled in a distributed manner. This has a number of benefits, such as reusing experience and not forgetting important experiences.</p>
  <p>Another benefit is that you can collect a lot of experience distributedly. Since RL is typically not bottlenecked by the computation for training but rather from collecting experiences, the distributed replay buffer in Ape-X can enable much faster training, in terms of seconds but not sample complexity.</p>
  <p>However, it's important to emphasize that these replay-buffer approaches are <strong>almost on-policy</strong>, in the sense that the replay buffer is constantly updated with new experiences. So, the policy in the replay buffer is &quot;not too different&quot; from your current policy (just a few gradient steps away). Most importantly, this allows the policy to learn from its own mistakes if it makes any...</p>
  <p>Off-policy learning, in general, can also refer to <em>batch RL</em> (a.k.a. <em>offline RL</em>), where you're provided a dataset of experiences from another behavior policy, and your goal is to improve over it. Notably, you don't get to rollout your current policy in any way! In this case, algorithms that worked well with a replay-buffer (like DQN, SAC) fail miserably, since they over-estimate the value of actions when they extrapolate outside the &quot;support&quot; of the dataset. See <a href="https://arxiv.org/pdf/1812.02900.pdf" rel="nofollow noreferrer">the BCQ paper</a> which illustrates how a lot of &quot;off-policy&quot; algorithms like DQN fail when the &quot;distance between the two policies is large&quot;. For this task, SOTA is a form of <em>weighted behavioral cloning</em> called <em>Critic Regularized Regression (CRR)</em>.</p>
  <p>It's also worth noting that <a href="https://en.wikipedia.org/wiki/Importance_sampling" rel="nofollow noreferrer">importance sampling</a> can correct off-policy gradients to be on-policy; but the farther away your target policy is, the larger the variance. This is especially deadly for long horizon tasks (often called <em>curse of horizon</em>).</p>
  <p>To summarize, using replay-buffer (which makes the algorithm off-policy), especially a distributed one, can offer a lot of benefits over pure on-policy algorithms. However, this is a very special class of off-policy algorithms, where the behavioral policy is close to your policy.</p>
  <p>But in general, off-policy is a lot harder than on-policy; you'll suffer from extrapolation bias if you use DQN-based approaches, and exponential variance blow-up if you use importance sampling to correct for it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T06:48:21.693Z
Answers.Text: thanks for the response. But your answer goes way above my head :-(
Answers.UserDisplayName: Exploring
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T06:55:38.477Z
Answers.Text: which part do you want clarification on? do you understand on-policy vs. off-policy after reading @mugoh's post?
Answers.UserDisplayName: kaiwenw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T10:42:51.853Z
Answers.Text: Maybe you should cite a paper that supports this "in this case, a lot of the traditional "off-policy" algos like DQN, SAC fail miserably, and the SOTA is a form of weighted behavioral cloning called Critic Regularized Regression (CRR)", for those not familiar with the SOTA. It's also not clear what you mean by the last sentence and in particular "but in general on-policy is much easier than off-policy!" (easier in terms of what?). Also, not sure why you say that DQN-based algorithms are "almost on-policy". That seems like saying that there are almost no off-policy algorithms.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T10:46:13.763Z
Answers.Text: I would actually say that off-policy algorithms can become on-policy, i.e. they generalize on-policy ones, but they are not "almost on-policy", if you want to be precise. Well, at least, I don't fully get what you mean by that.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T12:20:16.480Z
Answers.Text: This answer contains useful information, but I think it is pitched incorrectly to level of knowledge displayed in the question.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T12:24:35.847Z
Answers.Text: @nbro - being "almost on policy" means that the distribution of action choice between behaviour policy and target policy is close - e.g. a low KL divergence (although that is probably not the right measure for RL policy comparisons). If there was a high divergence in off-policy RL, then the learning rate per time step experienced can slow down dramatically, as the amount of data received that is relevant to the target policy will be low.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T12:54:27.153Z
Answers.Text: @NeilSlater Yes, sure you can measure the distance between two probability distributions, so that makes sense to say "almost on-policy" provided that is really true in the cases the OP is mentioning. When you say "high-variance", do you mean variance in the type of experience received? If the behaviour policy is very different from the target policy, I guess that having low variance of the experience received with the behaviour policy may not necessarily be a good thing. In any case, that statement is very compact and it's not fully/precisely clear what you mean by that.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T13:20:18.610Z
Answers.Text: @nbro: The variance in estimated returns increases, the more you have to correct for difference between on-policy and off-policy distributions.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T19:08:39.350Z
Answers.Text: @nbro you're right that off-policy generalizes on-policy, but as the two policies become more and more different, the MSE in the estimate will become very large
Answers.UserDisplayName: kaiwenw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T19:11:19.547Z
Answers.Text: @kaiwenw yes, that's what Neil was saying above, and I see your point now. Can you please edit your answer to include these details? I think they are useful, then I could clean up all these comments (that may just be noisy).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T05:23:44.453Z
Answers.LastActivityDate: 2020-11-28T02:01:13.920Z
Answers.LastEditDate: 2020-11-28T02:01:13.920Z
Answers.UserDisplayName: kaiwenw
Body: <p>I am self-studying about Reinforcement Learning using different online resources. I now have a basic understanding of how RL works.</p>
  <p>I saw this in <a href="https://artint.info/2e/html/ArtInt2e.Ch12.S7.html" rel="noreferrer">a book</a>:</p>
  <blockquote>
  <p>Q-learning is an off-policy learner. An off-policy learner learns the value of an optimal policy independently of the agent’s actions, as long as it explores enough.</p>
  </blockquote>
  <blockquote>
  <p>An on-policy learner learns the value of the policy being carried out by the agent, including the exploration steps.</p>
  </blockquote>
  <p>However, I am not quite understanding the difference. Secondly, I came across that off-policy learner works better than the on-policy agent. I don't understand why that would be i.e. why off-policy would be better than the on-policy.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-26T04:42:46.173Z
Comments.Text: Here's a very related question https://stats.stackexchange.com/q/184657
Comments.UserDisplayName: mugoh
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-26T09:49:49.627Z
Comments.Text: You say "Secondly, I came across that off-policy learner works better than the on-policy agent. ". Where did you hear/read this?
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><comparison><q-learning><off-policy-methods><on-policy-methods>
Title: Why does off-policy learning outperform on-policy learning?
UserDisplayName: Exploring
ThreadText: # Why does off-policy learning outperform on-policy learning?
  
  (Asked by: username_0 on 2020-11-26)
  
  I am self-studying about Reinforcement Learning using different online resources. I now have a basic understanding of how RL works.
  
  I saw this in [a book](https://artint.info/2e/html/ArtInt2e.Ch12.S7.html):
  
  > Q-learning is an off-policy learner. An off-policy learner learns the value of an optimal policy independently of the agent’s actions, as long as it explores enough.
  
  > An on-policy learner learns the value of the policy being carried out by the agent, including the exploration steps.
  
  However, I am not quite understanding the difference. Secondly, I came across that off-policy learner works better than the on-policy agent. I don't understand why that would be i.e. why off-policy would be better than the on-policy.
  
  **Tags:** reinforcement-learning, comparison, q-learning, off-policy-methods, on-policy-methods
  > **Comments:**
  > **username_1**: Here's a very related question https://stats.stackexchange.com/q/184657  
  > **username_2**: You say "Secondly, I came across that off-policy learner works better than the on-policy agent. ". Where did you hear/read this?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-11-26. Score: 6
  
  [This post](https://stats.stackexchange.com/q/184657/82135) contains many answers that describe the difference between on-policy vs. off-policy.
  
  Your book may be referring to how the current (DQN-based) state-of-the-art (SOTA) algorithms, such as [Ape-X](https://openreview.net/pdf?id=H1Dy---0Z), [R2D2](https://openreview.net/pdf?id=r1lyTjAqYX), [Agent57](https://deepmind.com/blog/article/Agent57-Outperforming-the-human-Atari-benchmark) are technically "off-policy", since they use a (very large!) replay buffer, often filled in a distributed manner. This has a number of benefits, such as reusing experience and not forgetting important experiences.
  
  Another benefit is that you can collect a lot of experience distributedly. Since RL is typically not bottlenecked by the computation for training but rather from collecting experiences, the distributed replay buffer in Ape-X can enable much faster training, in terms of seconds but not sample complexity.
  
  However, it's important to emphasize that these replay-buffer approaches are **almost on-policy**, in the sense that the replay buffer is constantly updated with new experiences. So, the policy in the replay buffer is "not too different" from your current policy (just a few gradient steps away). Most importantly, this allows the policy to learn from its own mistakes if it makes any...
  
  Off-policy learning, in general, can also refer to *batch RL* (a.k.a. *offline RL*), where you're provided a dataset of experiences from another behavior policy, and your goal is to improve over it. Notably, you don't get to rollout your current policy in any way! In this case, algorithms that worked well with a replay-buffer (like DQN, SAC) fail miserably, since they over-estimate the value of actions when they extrapolate outside the "support" of the dataset. See [the BCQ paper](https://arxiv.org/pdf/1812.02900.pdf) which illustrates how a lot of "off-policy" algorithms like DQN fail when the "distance between the two policies is large". For this task, SOTA is a form of *weighted behavioral cloning* called *Critic Regularized Regression (CRR)*.
  
  It's also worth noting that [importance sampling](https://en.wikipedia.org/wiki/Importance_sampling) can correct off-policy gradients to be on-policy; but the farther away your target policy is, the larger the variance. This is especially deadly for long horizon tasks (often called *curse of horizon*).
  
  To summarize, using replay-buffer (which makes the algorithm off-policy), especially a distributed one, can offer a lot of benefits over pure on-policy algorithms. However, this is a very special class of off-policy algorithms, where the behavioral policy is close to your policy.
  
  But in general, off-policy is a lot harder than on-policy; you'll suffer from extrapolation bias if you use DQN-based approaches, and exponential variance blow-up if you use importance sampling to correct for it.
  
  > **Comments:**
  > **username_0**: thanks for the response. But your answer goes way above my head :-(  
  > **username_3**: which part do you want clarification on? do you understand on-policy vs. off-policy after reading @mugoh's post?  
  > **username_2**: Maybe you should cite a paper that supports this "in this case, a lot of the traditional "off-policy" algos like DQN, SAC fail miserably, and the SOTA is a form of weighted behavioral cloning called Critic Regularized Regression (CRR)", for those not familiar with the SOTA. It's also not clear what you mean by the last sentence and in particular "but in general on-policy is much easier than off-policy!" (easier in terms of what?). Also, not sure why you say that DQN-based algorithms are "almost on-policy". That seems like saying that there are almost no off-policy algorithms.  
  > **username_2**: I would actually say that off-policy algorithms can become on-policy, i.e. they generalize on-policy ones, but they are not "almost on-policy", if you want to be precise. Well, at least, I don't fully get what you mean by that.  
  > **username_4**: This answer contains useful information, but I think it is pitched incorrectly to level of knowledge displayed in the question.  
  > **username_4**: @nbro - being "almost on policy" means that the distribution of action choice between behaviour policy and target policy is close - e.g. a low KL divergence (although that is probably not the right measure for RL policy comparisons). If there was a high divergence in off-policy RL, then the learning rate per time step experienced can slow down dramatically, as the amount of data received that is relevant to the target policy will be low.  
  > **username_2**: @NeilSlater Yes, sure you can measure the distance between two probability distributions, so that makes sense to say "almost on-policy" provided that is really true in the cases the OP is mentioning. When you say "high-variance", do you mean variance in the type of experience received? If the behaviour policy is very different from the target policy, I guess that having low variance of the experience received with the behaviour policy may not necessarily be a good thing. In any case, that statement is very compact and it's not fully/precisely clear what you mean by that.  
  > **username_4**: @nbro: The variance in estimated returns increases, the more you have to correct for difference between on-policy and off-policy distributions.  
  > **username_3**: @nbro you're right that off-policy generalizes on-policy, but as the two policies become more and more different, the MSE in the estimate will become very large  
  > **username_2**: @kaiwenw yes, that's what Neil was saying above, and I see your point now. Can you please edit your answer to include these details? I think they are useful, then I could clean up all these comments (that may just be noisy).  
  
  ---
  
  