Answers.Body: <p>In the DRL nanodegree in Udacity, the instructor says it is possible to combine on- and off-policy learning and suggests the following paper where this has been done: <a href="https://arxiv.org/pdf/1611.02247.pdf" rel="nofollow noreferrer">Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic</a> (ICLR 2017). 
  Citing the paper:</p>
  
  <blockquote>
    <p>The core idea is to use the first-order Taylor expansion of the critic
    as a control variate, resulting in an analytical gradient term through
    the critic and a Monte Carlo policy gradient term consisting of the
    residuals in advantage approximations. The method helps unify policy
    gradient and actor-critic methods: it can be seen as using the
    off-policy critic to reduce variance in policy gradient or using
    on-policy Monte Carlo returns to correct for bias in the critic
    gradient.</p>
  </blockquote>
  
  <p>The authors provide an open source implementation of it in <a href="https://github.com/shaneshixiang/rllabplusplus" rel="nofollow noreferrer">https://github.com/shaneshixiang/rllabplusplus</a></p>
  
  <p>There is a follow-up paper by the same authors also addressing this problem: Interpolated Policy Gradient: <a href="https://papers.nips.cc/paper/6974-interpolated-policy-gradient-merging-on-policy-and-off-policy-gradient-estimation-for-deep-reinforcement-learning.pdf" rel="nofollow noreferrer">Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning</a> (NIPS 2017).</p>
  
  <p>The Related Work section in both papers might be also worth looking at.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-14T18:59:21.580Z
Answers.Text: Dear @user5093249, thank you very much for your detailed and definitely interesting answer. I'll definitely have a look at the papers. I'm especially interested in combining existing algorithms (like TD3 and PPO) instead of developing new ones that combine on- and off-policy approaches.
Answers.UserDisplayName: Ray Walker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-14T19:56:04.030Z
Answers.Text: @RayWalker your approach seems to be in line with what the authors suggest. In fact, in the first paper, they state that "since Q-Prop uses both on-policy policy updates and off-policy critic learning, it can take advantage of prior work along both lines of research". They implemented it on top of TRPO-GAE for example.
Answers.UserDisplayName: user5093249
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-15T18:40:37.197Z
Answers.Text: Ok great, thank you very much for the additional information. I've now had a look at both abstracts and accepted your answer. It very well answers the main question in the topic.  And the Q-Prop paper just jumped to the top of my reading list : )
Answers.UserDisplayName: Ray Walker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-14T13:41:02.223Z
Answers.LastActivityDate: 2020-05-14T13:41:02.223Z
Answers.UserDisplayName: user5093249
Body: <p><strong>On-Policy Algorithms</strong> like PPO directly maximize the performance objective or an approximation of it. They tend to be quite stable and reliable but are often sample inefficient. <strong>Off-Policy Algorithms</strong> like TD3 improve the sample inefficiency by reusing data collected with previous policies, but they tend to be less stable. (Source: <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html" rel="nofollow noreferrer">Kinds of RL Algorithms - Spinning up - OpenAI</a>)</p>
  
  <p>Looking at learning curves comparing SOTA algorithms, we see that off-policy algorithms quickly improve performance at the training's beginning. Here <a href="https://spinningup.openai.com/en/latest/spinningup/bench.html#walker2d-tensorflow-versions" rel="nofollow noreferrer">an example</a>:
  <a href="https://i.sstatic.net/kLViU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/kLViU.png" alt="3M timestep benchmark of different algorithms for Walker2d-v3"></a></p>
  
  <p>Can we start training off-policy and after some time use the learned and quickly improved policy to init the policy network of an on-policy algorithm? </p>
  
Tags: <reinforcement-learning><deep-rl><open-ai><off-policy-methods><on-policy-methods>
Title: Can we combine Off-Policy with On-Policy Algorithms?
UserDisplayName: Ray Walker
ThreadText: # Can we combine Off-Policy with On-Policy Algorithms?
  
  (Asked by: username_0 on 2020-05-07)
  
  **On-Policy Algorithms** like PPO directly maximize the performance objective or an approximation of it. They tend to be quite stable and reliable but are often sample inefficient. **Off-Policy Algorithms** like TD3 improve the sample inefficiency by reusing data collected with previous policies, but they tend to be less stable. (Source: [Kinds of RL Algorithms - Spinning up - OpenAI](https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html))
  
  Looking at learning curves comparing SOTA algorithms, we see that off-policy algorithms quickly improve performance at the training's beginning. Here [an example](https://spinningup.openai.com/en/latest/spinningup/bench.html#walker2d-tensorflow-versions):
  [![3M timestep benchmark of different algorithms for Walker2d-v3](https://i.sstatic.net/kLViU.png)](https://i.sstatic.net/kLViU.png)
  
  Can we start training off-policy and after some time use the learned and quickly improved policy to init the policy network of an on-policy algorithm?
  
  **Tags:** reinforcement-learning, deep-rl, open-ai, off-policy-methods, on-policy-methods
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-14. Score: 1 | Accepted answer
  
  In the DRL nanodegree in Udacity, the instructor says it is possible to combine on- and off-policy learning and suggests the following paper where this has been done: [Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic](https://arxiv.org/pdf/1611.02247.pdf) (ICLR 2017).
  Citing the paper:
  
  > The core idea is to use the first-order Taylor expansion of the critic
  > as a control variate, resulting in an analytical gradient term through
  > the critic and a Monte Carlo policy gradient term consisting of the
  > residuals in advantage approximations. The method helps unify policy
  > gradient and actor-critic methods: it can be seen as using the
  > off-policy critic to reduce variance in policy gradient or using
  > on-policy Monte Carlo returns to correct for bias in the critic
  > gradient.
  
  The authors provide an open source implementation of it in <https://github.com/shaneshixiang/rllabplusplus>
  
  There is a follow-up paper by the same authors also addressing this problem: Interpolated Policy Gradient: [Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning](https://papers.nips.cc/paper/6974-interpolated-policy-gradient-merging-on-policy-and-off-policy-gradient-estimation-for-deep-reinforcement-learning.pdf) (NIPS 2017).
  
  The Related Work section in both papers might be also worth looking at.
  
  > **Comments:**
  > **username_0**: Dear @user5093249, thank you very much for your detailed and definitely interesting answer. I'll definitely have a look at the papers. I'm especially interested in combining existing algorithms (like TD3 and PPO) instead of developing new ones that combine on- and off-policy approaches.  
  > **username_1**: @RayWalker your approach seems to be in line with what the authors suggest. In fact, in the first paper, they state that "since Q-Prop uses both on-policy policy updates and off-policy critic learning, it can take advantage of prior work along both lines of research". They implemented it on top of TRPO-GAE for example.  
  > **username_0**: Ok great, thank you very much for the additional information. I've now had a look at both abstracts and accepted your answer. It very well answers the main question in the topic.  And the Q-Prop paper just jumped to the top of my reading list : )  
  
  ---
  
  