Deep reinforcement learning has achieved remarkable successes in solving various challenging artificial intelligence tasks. A variety of different algorithms have been introduced and improved towards human-level performance. Although technical advances have been developed for each individual algorithms, there has been strong evidence showing that further substantial improvements can be achieved by properly combining multiple approaches with difference biases and variances. In this work, we propose to use the James-Stein (JS) shrinkage estimator to combine on-policy policy gradient estimators which have low bias but high variance, with low-variance high-bias gradient estimates such as those constructed based on model-based methods or temporally smoothed averaging of historical gradients. Empirical results show that our simple shrinkage approach is very effective in practice and substantially improve the sample efficiency of the state-of-the-art on-policy methods on various continuous control tasks.
This paper presents two algorithms that improve on PPO by using James-Stein (JS) shrinkage estimator. The first algorithm, PPO-MBS, combines low bias of on-policy methods with low variance of model-based RL algorithms. The second, PPO-STS, uses JS to create a statistical momentum and reduce variance of the PPO algorithm. Both algorithms are evaluated on Mujoco environments aiming to show improvements in average cumulative reward, reduced bias and variance.
The paper’s topic is highly relevant, as the authors point out, the current state of the art in RL is largely divided between model-based and model-free methods. This paper aims at bridging the gap, and taking advantage of both sides. The writing is clear and concise, with all the math properly introduced. The proposed methods are interesting and novel. As far as I am aware, this is solid and unexplored approach with potentially significant impact.
There are two concerns with this paper. First, the evaluation results are promising, yet not fully convincing. It appears that 5 million steps is not sufficient for the policy convergence for any of the tasks (average reward keeps increasing). At the same time the variance gap (Figure 1) is reducing. 
- As the training continues, does the variance for PPO-MBS become larger?
- Similar question for the average reward - the advantage of the PPO-MBS vs. comparison methods seems to be reducing. What happens with the additional training?
- How do the trajectories and behavior of the Walker2D (and other) look with PPO-MBS vs. others - is the higher average reward indicative of the qualitatively better behavior?
- Why do Walker2D and Hopper increase \alpha over time, while Swimmer and Reacher lean more towards mode model based policy over time?
- Is there something significant about the structure of the problems?
- Similar questions arise from the evaluation of the PPO-STS - it appears that the training is even less complete in this case, rendering the conclusions about the quality of the learned policy at convergence invalid.
- Why are Humanoid and Ant not evaluated on PPO-MBS? The authors should extend the training to convergence on all problems, and present the results including movies of example trajectories of all environments for both algorithms in the supplementary material.
Second, the paper introduces two competing methods.  While, the authors compare them directly,  they do not discuss how the two methods relate to each other. In the Reacher and Hopper task seems like all three methods perform about the same, and in the Hopper PPO-STS even performs worse than the baseline. This makes it difficult to assess the significance of the exposition. When should one be used, and when the other one is better?
The authors should consider either a) splitting the paper into two focusing each paper on a single algorithm with more in-depth evaluations and discussions, or b) combining the two algorithms into ones. In any case, further analysis that illuminates when the methods should be used, and how they improve the training are needed.