Probabilistic Differential Dynamic Programming
We present a data-driven, probabilistic trajectory optimization framework for systems with unknown dynamics, called Probabilistic Differential Dynamic Programming (PDDP). PDDP takes into account uncertainty explicitly for dynamics models using Gaussian processes (GPs). Based on the second-order local approximation of the value function, PDDP performs Dynamic Programming around a nominal trajectory in Gaussian belief spaces. Different from typical gradientbased policy search methods, PDDP does not require a policy parameterization and learns a locally optimal, time-varying control policy. We demonstrate the effectiveness and efficiency of the proposed algorithm using two nontrivial tasks. Compared with the classical DDP and a state-of-the-art GP-based policy search method, PDDP offers a superior combination of data-efficiency, learning speed, and applicability.
This paper proposes a DDP formulation for models represented as Gaussian processes. The derivations of the required equations that connects the Gaussian Process model with the DDP framework are provided.The method allows the solution of local optimal trajectory problems for systems whose dynamics are not known in advance.As such this work shares several characteristics with PILCO. The simulated results show that solutions are found much faster than PILCO. Quality and clarityAlthough the concept of the paper is strong, the paper needs more work to be done, particularly regarding the presentation of the results.Presentation of the results is currently too shallow, especially the discussion, which is basically non-existent for figures 1 and 2.Claims such as "safe exploration" lacks support and the computational complexity conclusion is questionable.The writing could be refined in general; I am not referring to specific sentences, but it feels this work is lacking some additional number of polishing iterations on the writing.Originality and SignificanceThe fact that DDP is being proposed as the optimization method has an impact for the DDP/iLQG practitioners and also for the model-based optimization community. Unarguably the presented work shares several similarities with PILCO both in the philosophy and also the methodology. However, while the work of PILCO basically kept the choices of the optimizer open, this work suggest DDP and develops the full framework thus providing a self-contained solution.This work can be much improved if extensive and careful analyses between PDDP and PILCO are provided.Questions . Line 33: "... Compared to global optimal control approaches, the local optimal DDP shows superior scalability to high-dimensional problems..." is an incomplete statement. The big trade-off is that DDP solution is only local and prone to provide a poor a local optima. I agree that for high-dimensional problems a local solution is usually the only option. If there was no trade-off there would be no reason for the full dynamic programming.. Line 212: The sentence " ... where the variance of control distribution and the off-diagonal entries of the state covariance matrix can be neglected because of no impact on control performances." is not obvious especially regarding the control distribution. Noise in control can have a large impact on how the states integrate over time. It may also influence the final cost. Are you assuming a deterministic system?. Line: 294. The conclusion on the computational complexity is precipitated. Yes, although the complexity of the learning policy does not relate directly to the number of states, for the usual case of fully actuated systems, the number of control inputs is proportinal to the number of states which makes the complexity of the algorithm O3.. The formulation of the "safe exploration" feature is not clear in the text. Although Figure 3(a) is a cartoon that shows the intuition, where is this intuition implemented in the algorithm? . Line 322: "To keep the system stable" seems to be a bad sentence, unless you are talking about unstable system such as an inverted pendulum. Do you mean "final zero velocity"?. The variance in (a) seems simply to small, almost a deterministic system. As the GP is learning the system from samples one can infer that the variance during the initial iterations are large and decrease with the number iterations. It would be interesting to see such a plot.. I do not see the point of proposing another method (GPDDP) that  does not bring any benefit. What is the motivation for that? If what you want to show is the effect of the propagation of uncertainty then you can just say that without the need of introducing the GPDDP as a new second method.. The paper must also compare the traditional DDP results when dynamics are fully known  in Figs 1 and 2.Minor questions:. Legends for Figure 1 are too small. The y axes of (a) and (c) should be on the same range.. Why not showing the costs for PILCO on figures 1(b) and 2(b)?