Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning
Off-policy model-free deep reinforcement learning methods using previously collected data can improve sample efficiency over on-policy policy gradient techniques. On the other hand, on-policy algorithms are often more stable and easier to use. This paper examines, both theoretically and empirically, approaches to merging on-and off-policy updates for deep reinforcement learning. Theoretical results show that off-policy updates with a value function estimator can be interpolated with on-policy policy gradient updates whilst still satisfying performance bounds. Our analysis uses control variate methods to produce a family of policy gradient algorithms, with several recently proposed algorithms being special cases of this family. We then provide an empirical comparison of these techniques with the remaining algorithmic details fixed, and show how different mixing of off-policy gradient estimates with on-policy samples contribute to improvements in empirical performance. The final algorithm provides a generalization and unification of existing deep policy gradient techniques, has theoretical guarantees on the bias introduced by off-policy updates, and improves on the state-of-the-art model-free deep RL methods on a number of OpenAI Gym continuous control benchmarks.
* summary: This paper studies how to merge on-policy and off-policy gradient algorithms. The authors propose a new policy gradient framework that unifies many previous on-policy and off-policy gradient methods. Many previous policy gradient algorithms can not only be re-derived and but also get improved by the introduction of control variate. Even though this framework introduces bias to gradient updates but, they show in theoretical results that this bias can be bounded. Experiments are very well done and provide enough insights to understand the proposed framework. 
* In overall, the paper is very well written with excellent related work and discussions. The main contents are well presented as well, the theoretical results are very intuitive and make sense. 
* I think this is very strong paper. I have only few comments as following:
- I think that even if one is familiar with TRPO, he might still not be able to know how to transfer the technique of TRPO to approximately implement Algorithm 2. It would be nicer if the authors elaborate on this step.
- As seen in Eq. (6) and Theorem 1, the bias introduced by the difference between \beta and \pi is non-trivial. Therefore, I find that IPG does not really always outperform Q-Prop, sometimes they are comparable (when \beta is not equal to \pi). According to my understanding, v from 0 to 1 means a shifting gradually from on-policy to off-policy, therefore one could compensate the bias by gathering more data/larger batch-size. Therefore, it might be more interesting to see if Table 2 is expanded with more experiment setting: v increases gradually from 0 to 1 (with and without control variate), and also with the change of data-size/batch-size to see if bias can be compensated somehow.
- line 151, should it be $v=0$?