Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics
We present a policy search method that uses iteratively refitted local linear models to optimize trajectory distributions for large, continuous problems. These trajectory distributions can be used within the framework of guided policy search to learn policies with an arbitrary parameterization. Our method fits time-varying linear dynamics models to speed up learning, but does not rely on learning a global model, which can be difficult when the dynamics are complex and discontinuous. We show that this hybrid approach requires many fewer samples than model-free methods, and can handle complex, nonsmooth dynamics that can pose a challenge for model-based techniques. We present experiments showing that our method can be used to learn complex neural network policies that successfully execute simulated robotic manipulation tasks in partially observed environments with numerous contact discontinuities and underactuation.
Summary:The authors propose the combined use of trajectory optimization and policy learning for systems with unknown dynamics approximated by linear local models. The proposed trajectory optimization algorithm is based on DDP/iLQG methods. The local linear models are assumed Gaussian and obtained from samples returned by the previous policy solution. The paper introduces the iLQG-based method such that the approximated local models can be used. Optimization with approximated models is made possible by limiting the changes of the modified iLQG solution using KL-divergence as a constraint. The paper also presents the use of their trajectory optimization method within the framework of Guided Policy Search previously proposed in [11] such that richer parameterized policies can be learned by iteratively swapping between the trajectory optimization and the policy learning steps. Results are evaluated in simulation.Quality:The paper is of high quality. It is clearly written. Sometimes the paper relies too much on prose (see comments below) and support for a few statements are not clearly found. The paper is of high interest for the robotics and robot learning community especially due to the "unknown dynamics" feature, however, large part of the merit is attributed to the fact that it builds upon the the contributions of [11, 12, 13]. The evaluation of the method is of high quality and extensive and several relevant state-of-the-art methods are selected for comparison. The simulated tasks are interesting and show in a clear way the differences in performance. Clarity and Questions:Certain parts of the paper are not clear and although he supporting information may be somewhere in the paper it is not always easy to find.. Please, define the use of "unknown dynamics" and "hybrid" right at the introduction. Unknown dynamics inevitably induces the "model-free" image, but the paper does not seem to consider itself as a model-free method. At the same time, the idea of creating a model from samples is usually regarded as model-based (such as PILCO). I personally would not introduce the new term "unknown dynamics" but rather use something like "local Gaussian models" instead.. In line 52, "parameterized policy never needs to be executed on the real system" needs clarification. What I understand from this  sentence is that you would use the Gaussian controller during the whole learning process, and only at the end learn the parameterized policy based on what the Gaussian controller found (?). In that case it seems that you are not using the GPS framework as the parameterized policy is not being influenced by the trajectory optimization and vice-versa.. Line 59, the conclusion "...transforms a discontinuous deterministic problem into a smooth stochastic one" is very strong but the intuition that lead to it is not convincing. Is there a better way to explain it? A related request: since simulations with contact are difficult to model and local models do not capture discontinuity the paper would become much stronger with experimental results on a tasks with real physical contact or impact. . Although the GMM for background dynamics distribution seems to be very beneficial (in lines 400-401 it even seems to be intrinsic part of the method), it is only in Section 3.2 that GMMs are firstly mentioned. Until this point the whole method is explained as if there is no GMM. This can be a little confusing as at this point I do not know if GMM is part of the method or GMM is an extra feature that improves the method. As it seems to be always a good thing, why not present the method as unknown dynamics approximated by a mixture of Gaussians?. How does the system behave under saturation? Any real physical robot will have actuation limits. Does it compromise the induction of Gaussian trajectory distributions if the commands from the controller are saturated?. The fact that the iLQG with a true model performs worse than your method (which builds on iLQG but uses approximated linear models) is counter-intuitive. How it is possible?Originality and Significance:The paper is incremental in relation to the works of [11,12,13] although the contribution it adds is significant. The idea of being able to optimize trajectories and design a controller for an unknown system dynamics is of high interest for the robotics community, which was not previously addressed by [11,12,13].