SUPERVISED POLICY UPDATE FOR DEEP REINFORCE- MENT LEARNING
We propose a new sample-efficient methodology, called Supervised Policy Update (SPU), for deep reinforcement learning. Starting with data generated by the current policy, SPU formulates and solves a constrained optimization problem in the non-parameterized proximal policy space. Using supervised regression, it then converts the optimal non-parameterized policy to a parameterized policy, from which it draws new samples. The methodology is general in that it applies to both discrete and continuous action spaces, and can handle a wide variety of proximity constraints for the non-parameterized optimization problem. We show how the Natural Policy Gradient and Trust Region Policy Optimization (NPG/TRPO) problems, and the Proximal Policy Optimization (PPO) problem can be addressed by this methodology. The SPU implementation is much simpler than TRPO. In terms of sample efficiency, our extensive experiments show SPU outperforms TRPO in Mujoco simulated robotic tasks and outperforms PPO in Atari video game tasks.
The authors formulate policy optimization as a two step iterative procedure: 1) solving a constrained optimization problem in the non-parameterized policy space, 2) using supervised regression to project this onto a parameterized policy. This approach generally applies to both continuous and discrete action spaces and can handle a variety of constraints. Their primary claims is that this approach improves sample-efficiency over TRPO on Mujoco tasks and over PPO on Atari games.
The method proposed in the paper has strong similarities with existing methods, but lacks comparisons with these approaches. The authors have not clearly demonstrated that SPU provides novel insights beyond the existing literature. I'm happy to change my score if the authors can convince me otherwise.
Main comments:
The focus of the paper is sample-efficiency, but the intro restricts to the on-policy setting. The authors should justify this choice. It is well known that off-policy algorithms (e.g., SAC for continuous control and Implicit Quantile Networks for Atari) are much more sample-efficient.
In Sec 4, what is the advantage of breaking the problem up into these 3 steps versus directly trying to solve (9),(10)? In fact, if we convert (10) into a penalty and take the derivative, we arrive at nearly the same gradient as (17). As this is central to the SPU framework, this needs to be justified.
MPO (Abdolmaleki et al. 2018) is very closely related to SPU. It is unclear if SPU provides any additional insights or benefits over MPO. This needs to be discussed and compared.
The experimental section could be strengthened by:
* Given the similarity to SPU, comparisons to MPO and GAC should be made, or clear justification for why they are not comparable must be given.
* Why is the comparison on Mujoco to TRPO in the main text and the comparison to PPO relegated to the appendix? It would make more sense to compare to PPO, so the authors need to justify this decision.
* The results on Mujoco are quite poor compared to state-of-the-art methods (e.g., SAC). The authors should justify why we should care about their results.
Comments:
In Sec 2, the authors should be careful about the discounting. For example, they are almost surely not having A_{it} approximate \gamma^t A^{\pi_{\theta_k}}, rather A^{\pi_{\theta_k}}.
In Sec 2, the KL is denoted as KL(\pi || pi_k), but in the text is described as the KL from \pi_k to \pi (reversed). From the equations, it appears that is an error, and it should read KL from \pi to \pi_k.
In Sec 3, the description of NPG/TRPO is not accurate. The main goal of NPG/TRPO work was to establish monotonic improvement.
In Sec 3, computational speed is cited as a major deficit of GAC, especially the solving linear systems with the Hessian (wrt to the actions). This seems rather surprising. Inverting a 1000x1000 matrix on a modern computer takes <1 second, so it doesn't seem like this should be the limiting step for any of the problems encountered.
The KL penalty version of PPO seems closely related to SPU. Can the authors mention differences with that version of PPO in the related work?
In Sec 4, step iii is described as supervised learning. Can the authors elaborate on why? I would typically think of the other direction as supervised learning as that leads to MLE.
In Sec 5.1, what is the justification/reasoning for setting \tilde{\lambda_{s_i}} = \lambda and introducing the indicator functions?
Sec 5.2 is not evaluated and Sec 5.3 produces inferior results, so it may make sense to move these to the appendix. Otherwise, the authors should explain situations where we would expect these to be useful or provide some additional insight. It also should be noted that the proximity constraints in TRPO/PPO follow from a theoretical argument and are not arbitrary choices.
Sec 5.3 seems to deviate quite a bit from the SPU framework. In addition to the differences pointed out in the text, the "supervised" loss changes too. Can the authors justify/explain the reasoning for these changes?
=====
I appreciate the authors' efforts to improve the paper. However, there is still substantial room for improvement in writing clarity. For example, the authors optimize the reverse KL from typical supervised learning, which makes even the title of the method confusing. The method that was experimentally evaluated can be derived more simply without the two-step procedure by directly taking the gradient and add the heuristically motivated per state indicator. This in itself is interesting and the authors demonstrate that it works well experimentally. I think the paper would be substantially more useful to the community if the authors focused on that contribution alone. As it stands now, I find the paper difficult to read because most of the theoretical results have no bearing on the method.