Combining Model-based and Model-free RL via Multi-step Control Variates
Model-free deep reinforcement learning algorithms are able to successfully solve a wide range of continuous control tasks, but typically require many on-policy samples to achieve good performance. Model-based RL algorithms are sample-efficient on the other hand, while learning accurate global models of complex dynamic environments has turned out to be tricky in practice, which leads to the unsatisfactory performance of the learned policies. In this work, we combine the sample-efficiency of model-based algorithms and the accuracy of model-free algorithms. We leverage multi-step neural network based predictive models by embedding real trajectories into imaginary rollouts of the model, and use the imaginary cumulative rewards as control variates for model-free algorithms. In this way, we achieved the strengths of both sides and derived an estimator which is not only sample-efficient, but also unbiased and of very low variance. We present our evaluation on the MuJoCo and OpenAI Gym benchmarks.
 The main idea of the paper is to improve off-policy policy gradient estimates using control variates based on multi-step rollouts, and reduce the variance of those control variates using the reparameterization trick. This is laid out primarily in Equations 1-5, and seems like a nice idea, although I must admit I had some trouble following the maths in Equation 5. They include results showing that their method has better sample efficiency than TRPO (which their method also uses under the hood to update value function parameters). My main issue with this paper is that the empirical section is a bit weak, for instance only one run seems to be shown for both methods, there is no mention of hyper-parameter selection, and the measure used for generating Table 1 seems pretty arbitrary to me (how were those thresholds chosen?). In addition, one thing I would have liked to get out of this paper is a better understanding of how much each component helps. This could have been done via empirical work, for instance: - Explore the effect of the planning horizon, and implicitly compare to SVG(1), which as the authors point out is the same as their method with a horizon of 1. - Show the effect of the reparameterization trick on estimator variance. - Compare the bias and variance of TRPO estimates vs the proposed method.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Combining Model-based and Model-free RL via Multi-step Control Variates
Model-free deep reinforcement learning algorithms are able to successfully solve a wide range of continuous control tasks, but typically require many on-policy samples to achieve good performance. Model-based RL algorithms are sample-efficient on the other hand, while learning accurate global models of complex dynamic environments has turned out to be tricky in practice, which leads to the unsatisfactory performance of the learned policies. In this work, we combine the sample-efficiency of model-based algorithms and the accuracy of model-free algorithms. We leverage multi-step neural network based predictive models by embedding real trajectories into imaginary rollouts of the model, and use the imaginary cumulative rewards as control variates for model-free algorithms. In this way, we achieved the strengths of both sides and derived an estimator which is not only sample-efficient, but also unbiased and of very low variance. We present our evaluation on the MuJoCo and OpenAI Gym benchmarks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The main idea of the paper is to improve off-policy policy gradient estimates using control variates based on multi-step rollouts, and reduce the variance of those control variates using the reparameterization trick. This is laid out primarily in Equations 1-5, and seems like a nice idea, although I must admit I had some trouble following the maths in Equation 5. They include results showing that their method has better sample efficiency than TRPO (which their method also uses under the hood to update value function parameters). My main issue with this paper is that the empirical section is a bit weak, for instance only one run seems to be shown for both methods, there is no mention of hyper-parameter selection, and the measure used for generating Table 1 seems pretty arbitrary to me (how were those thresholds chosen?). In addition, one thing I would have liked to get out of this paper is a better understanding of how much each component helps. This could have been done via empirical work, for instance: - Explore the effect of the planning horizon, and implicitly compare to SVG(1), which as the authors point out is the same as their method with a horizon of 1. - Show the effect of the reparameterization trick on estimator variance. - Compare the bias and variance of TRPO estimates vs the proposed method.
assistant
 The main idea of the paper is to improve off-policy policy gradient estimates using control variates based on multi-step rollouts, and reduce the variance of those control variates using the reparameterization trick. This is laid out primarily in Equations 1-5, and seems like a nice idea, although I must admit I had some trouble following the maths in Equation 5. They include results showing that their method has better sample efficiency than TRPO (which their method also uses under the hood to update value function parameters). My main issue with this paper is that the empirical section is a bit weak, for instance only one run seems to be shown for both methods, there is no mention of hyper-parameter selection, and the measure used for generating Table 1 seems pretty arbitrary to me (how were those thresholds chosen?). In addition, one thing I would have liked to get out of this paper is a better understanding of how much each component helps. This could have been done via empirical work, for instance: - Explore the effect of the planning horizon, and implicitly compare to SVG(1), which as the authors point out is the same as their method with a horizon of 1. - Show the effect of the reparameterization trick on estimator variance. - Compare the bias and variance of TRPO estimates vs the proposed method.