Guided Policy Search via Approximate Mirror Descent
Guided policy search algorithms can be used to optimize complex nonlinear policies, such as deep neural networks, without directly computing policy gradients in the high-dimensional parameter space. Instead, these methods use supervised learning to train the policy to mimic a "teacher" algorithm, such as a trajectory optimizer or a trajectory-centric reinforcement learning method. Guided policy search methods provide asymptotic local convergence guarantees by construction, but it is not clear how much the policy improves within a small, finite number of iterations. We show that guided policy search algorithms can be interpreted as an approximate variant of mirror descent, where the projection onto the constraint manifold is not exact. We derive a new guided policy search algorithm that is simpler and provides appealing improvement and convergence guarantees in simplified convex and linear settings, and show that in the more general nonlinear setting, the error in the projection step can be bounded. We provide empirical results on several simulated robotic navigation and manipulation tasks that show that our method is stable and achieves similar or better performance when compared to prior guided policy search methods, with a simpler formulation and fewer hyperparameters.
The authors propose a new, simplified approach to guided policy search. Guided policy search consists in first improving a set of local policies starting from different initial states, second using supervised learning to "compile" them into a global policy, and then iterating those two steps towards convergence of local and global policies. In the previous method, a constraint was added through a surrogate cost function so that the local policies and global policies converge to each other. In this new approach, the surrogate cost function is removed and the process is simpler, based on minimizing the KL divergence between local and global policies. This simplification was made possible by recognizing the similarity between guided policy search and mirror descent. A very preliminary experimental study shows some advantages of the new approach.
Despite my initial lack of knowledge about guided policy search methods, which makes the paper hard to read for me, I believe this is a strong paper presenting state-of-the-art research. However, there are a few issues that I'm listing hereafter by order of decreasing importance.  My main point is with the way Algorithm 2 is written. In contrast to Algorithm 1, the C-step of Algorithm 2 requires to already have a global policy \pi_\theta so as to compute the KL divergence. Nothing is said about providing an initial policy \pi_\theta_0. Did I miss something here? Or shouldn't you split step 2 into a step 2 and a step 4 where you perform the re-optimization of local policies, as in Algo 1?  If I'm right, aren't there some constraints on the way \pi_\theta is initialized? In any case, this should be clarified.  The experimental evaluation is rather weak: - the description of the experiments is sketchy, one cannot reproduce them out of what is described (which simulator?, position of the holes?, size of the arm?, etc.) - the dimensions for x_t and u_t are not given - results are averaged over 3 runs only. Statistical significance is not checked. Worse than that, I doubt the validity of the success rates: how can you get a success rate of 62.50% over three runs with 4 holes? It means you succeeded 7,5 times (!). Something is not specified here (are there several instances of peg insertion for the same hole over one run?). - computational cost is not mentioned: it is high, to explain why averaging over 3 runs only? A comparison of computational cost and sample efficiency with respect to previous guided policy search methods or simpler baselines such as TRPO looks mandatory - the authors put scalability forward, but their policy is a 2 layers network with 40 neurons each, which is rather small. This should be discussed.  On one side, the paper is difficult and long. A good deal of the technical part of the paper is rejected into the supplementary material, and some sections of the paper can hardly be read without reading this supplementary part. I must say I didn't check many technical details in the methods nor the proofs in the supplementary material.  On the other side, I didn't like much Section 4.2, which tries to justify a rather elaborated policy for step size selection, but the message is ruined by footnote 3, which states that finally using a simple rule of thumb is enough.  So I would suggest the authors to remove Section 4.2 and use the resulting free space to clarify the experimental section and the methods.  Also, wouldn't Section 3.2 find a better place into Section 4?  It is never specified what is the domain of x_t and u_t. Are they in R^n? In a closed domain of R^n?  More local points:  Lines 55-58, this should be written more clearly. The first sentence states that policy gradient methods can be applied to huge policies (citing recent deep methods), and the second states that they are limited to small policies (citing older work). You may rather insist on number of episodes for training, as done later on.  Maybe you should not use casual writing such as "we'll, we've, we don't"... in scientific papers.  I just found one typo: In Algo 1, and each p(u_t|x_t) should be each p_i(u_t|x_t)