GUIDE ACTOR-CRITIC FOR CONTINUOUS CONTROL
Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-critic methods only use values or gradients of the critic to update the policy parameter. In this paper, we propose a novel actor-critic method called the guide actor-critic (GAC). GAC firstly learns a guide actor that locally maximizes the critic and then it updates the policy parameter based on the guide actor by supervised learning. Our main theoretical contributions are two folds. First, we show that GAC updates the guide actor by performing second-order optimization in the action space where the curvature matrix is based on the Hessians of the critic. Second, we show that the deterministic policy gradient method is a special case of GAC when the Hessians are ignored. Through experiments, we show that our method is a promising reinforcement learning method for continuous controls.
The paper presents a clever trick for updating the actor in an actor-critic setting: computing a guide actor that diverges from the actor to improve critic value, then updating the actor parameters towards the guide actor. This can be done since, when the parametrized actor is Gaussian and the critic value can be well-approximated as quadratic in the action, the guide actor can be optimized in closed form.
The paper is mostly clear and well-presented, except for two issues: 1) there is virtually nothing novel presented in the first half of the paper (before Section 3.3); and 2) the actual learning step is only presented on page 6, making it hard to understand the motivation behind the guide actor until very late through the paper.
The presented method itself seems to be an important contribution, even if the results are not overwhelmingly positive. It'd be interesting to see a more elaborate analysis of why it works well in some domains but not in others. More trials are also needed to alleviate any suspicion of lucky trials.
There are some other issues with the presentation of the method, but these don't affect the merit of the method:
1. Returns are defined from an initial distribution that is stationary for the policy. While this makes sense in well-mixing domains, the experiment domains are not well-mixing for most policies during training, for example a fallen humanoid will not get up on its own, and must be reset.
2. The definition of beta(a|s) as a mixture of past actors is inconsistent with the sampling method, which seems to be a mixture of past trajectories.
3. In the first paragraph of Section 3.3: "[...] the quality of a guide actor mostly depends on the accuracy of Taylor's approximation." What else does it depend on? Then: "[...] the action a_0 should be in a local vicinity of a."; and "[...] the action a_0 should be similar to actions sampled from pi_theta(a|s)." What do you mean "should"? In order for the Taylor approximation to be good?
4. The line before (19) is confusing, since (19) is exact and not an approximation. For the approximation (20), it isn't clear if this is a good approximation. Why/when is the 2nd term in (19) small?
5. The parametrization nu of \hat{Q} is never specified in Section 3.6. This is important in order to evaluate the complexities involved in computing its Hessian.