Variational Policy Search via Trajectory Optimization
In order to learn effective control policies for dynamical systems, policy search methods must be able to discover successful executions of the desired task. While random exploration can work well in simple domains, complex and highdimensional tasks present a serious challenge, particularly when combined with high-dimensional policies that make parameter-space exploration infeasible. We present a method that uses trajectory optimization as a powerful exploration strategy that guides the policy search. A variational decomposition of a maximum likelihood policy objective allows us to use standard trajectory optimization algorithms such as differential dynamic programming, interleaved with standard supervised learning for the policy itself. We demonstrate that the resulting algorithm can outperform prior methods on two challenging locomotion tasks.
The paper presents a new model-based policy search 
      algorithm that is based on the variational decomposition of policy search. 
      I order to estimate the variational distribution, a linearized model is 
      used to minimize the KL-term involved in determining the variational 
      distribution. The resulting algorithm outperforms existing methods which 
      is shown on challenging benchmark tasks. I personally think the 
      approach might be promising and i also like the experiments, but I think 
      there are crucial comparisons missing. First of all, the presented method 
      is a model-based method, and, hence the authors should also compare 
      against other model-based approaches. For example, the pilco framework [1] 
      learns a GP model for long-term prediction with moment matching (which is 
      actually a very similar approximation as the one performed by ddp). The 
      policy is then updated by Bfgs. This algorithm is state of the art in 
      model-based policy search and it is not clear to me why the presented 
      algorithm should perform better than pilco. Furthermore, the authors claim 
      that the ddp solution has several disadvantages to the learned policy. 
      While this might be well the case, the authors should also show that in 
      their evaluations. Maybe the generalization of the learned policy is 
      better but the ddp solution has higher quality for a single task. That 
      would be good to know. The comparison to the variant of power is also 
      rather unfair as power does not use a model. However, as the model is 
      assumed to be known, the model can be used to generate a vast amount of 
      samples. In the limit of an infinite amount of samples, power should 
      produce the same results as the ddp solution?. It would be interesting to 
      know how many samples are needed to match the performance of the ddp 
      solution. In theory, the presented method probably simplifies the 
      computational demands in comparison to the samplebased version of power, 
      but, in the limit of infinite samples, the result should be the same. 
      Hence, the authors need to evaluate the computational benefits of the 
      presented method. Furthermore, the relationship to existing state of the 
      art stochastic optimal control (soc) algorithms, such as soc by 
      approximate inference [2] or policy improvements by path integrals [3] 
      should at least be discussed. Minor issues: - the choice of 
      alpha_k seems to be very hacky to me. See [4] for a more sophisticated 
      approach for choosing the temperature of the exponential transformation. - 
      I do not really get why the exploration strategy of power is "random" 
      while the exploration strategy of ddp is guided. Both use exactly the same 
      exploration policy. While power uses samples to obtain the variational 
      distribution, ddp uses approximation to determine a closed form solution 
      for q. In the limit of infinite samples, power should even produce the 
      better results as no approximations are used. - how does your method 
      perform if the model is not known, but needs to be learned from data, 
      e.g., by using a Gaussian Process? [1] Deisenroth et. al. : PILCO: 
      A model-based and data-efficient approach to policy search [2] Rawik et. 
      al.: Stochastic optimal control by approximate inference [3] Theoudorou 
      et. al.: A Generalized Path Integral Control Approach to Reinforcement 
      Learning [4] Kupscik et. al.: Data-Efficient Generalization of Robot 
      Skills with Contextual Policy Search