Model-based reinforcement learning (MBRL) with model-predictive control or online planning has shown great potential for locomotion control tasks in both sample efficiency and asymptotic performance. Despite the successes, the existing planning methods search from candidate sequences randomly generated in the action space, which is inefficient in complex high-dimensional environments. In this paper, we propose a novel MBRL algorithm, model-based policy planning (POPLIN), that combines policy networks with online planning. More specifically, we formulate action planning at each time-step as an optimization problem using neural networks. We experiment with both optimization w.r.t. the action sequences initialized from the policy network, and also online optimization directly w.r.t. the parameters of the policy network. We show that in the MuJoCo benchmarking environments, POPLIN is about 3x more sample efficient than the previously stateof-the-art algorithms, such as PETS, TD3 and SAC. To explain the effectiveness of our algorithm, we show that the optimization surface in parameter space is smoother than in action space. Further more, we found the distilled policy network can be effectively applied without the expansive model predictive control during test time for some environments such as Cheetah. Code is released 1 .
Summary
This work provides a novel model-based reinforcement learning algorithm for continuous domains (Mujoco) dubbed POPLIN. The presented algorithm is similar in vein to the state-of-the-art PETS algorithm, a planning algorithm that uses state-unconditioned action proposal distributions to identify good action sequences with CEM in the planning routine. The important difference compared to PETS is the incorporation of a parametric state-conditioned policy (trained on real data) in the planning routine to obtain better action-sequences (CEM is used to learn the "offset" from the parametric policy). The paper presents two different algorithmic ablations where CEM either operates in action space or parameter space (POPLIN-A and POPLIN-P respectively), in combination with different objectives to learn the parametric policy. The method is evaluated on 12 continuous benchmarks and compared against state-of-the-art model-based and model-free algorithms, indicating dominance of the newly proposed method.
Quality
The quality of the paper is high. This is an experimental study and the number of benchmarks and baselines is far above average compared to other papers in that field. One minor point is that averaging experiments over 4 seeds only is usually not optimal in these environments, but in light of the sheer amount of baselines and benchmarks excusable. While the experimental results are impressive, the authors mention that asymptotic performance in Walker2d and Humanoid might not match the asymptotic performance of model-free baselines. This could be stated more clearly. Also, there are no Humanoid experiments in the paper despite mentioned in the text (2nd paragraph in Section 5.1)?
Clarity
The clarity of the paper can be in parts improved upon. For example, how does the "policy control" ablation (mentioned in Section 4.3) work precisely, i.e. the interplay between executing the parametric policy in the real world and harnessing the environment model? I assume the policy distillation techniques in Section 4.4 are different alternatives for the second-to-last lines in the pseudocodes from the appendix? Which one is the default used in Section 5.1? On a minor note, above Equation (7), a target network is mentioned---where does the target network occur in Equation (7)? There are some plots that do not mention the name of the environment, e.g. in Figure (4), but also some in the appendix. Furthermore, it could be stated more clearly that the reward function is assumed to be known. If the authors improve the clarity of their paper significantly, I am willing to increase my score further (presupposing that no severe issues arise in the discussion phase).
Originality
Adding a parametric policy to PETS is not the most original idea, but clearly a gap in the current literature.
Significance
The experiments and the empirical results make the paper quite significant.
Update
The authors improved the clarity of the paper. I therefore increase to 8. Section 4.4 paragraph "Setting parameter average (AVG)" can still be improved---does this go together with POPLIN-P-Uni from Section 4.2?