Sample Efficient Path Integral Control under Uncertainty
We present a data-driven optimal control framework that is derived using the path integral (PI) control approach. We find iterative control laws analytically without a priori policy parameterization based on probabilistic representation of the learned dynamics model. The proposed algorithm operates in a forward-backward manner which differentiate it from other PI-related methods that perform forward sampling to find optimal controls. Our method uses significantly less samples to find analytic control laws compared to other approaches within the PI control family that rely on extensive sampling from given dynamics models or trials on physical systems in a model-free fashion. In addition, the learned controllers can be generalized to new tasks without re-sampling based on the compositionality theory for the linearly-solvable optimal control framework. We provide experimental results on three different tasks and comparisons with state-of-the-art model-based methods to demonstrate the efficiency and generalizability of the proposed framework.
Summary  The paper presents a new method for path integral control. The proposed method leverages that the system control matrix G is often known, and the uncontrolled dynamics (or dynamics under a reference controller) can be learnedusing a Gaussian Process. The constraint on the reward function takes a more general shape than in previous PI approaches which means among others that noise and controls can act in different subspaces. The authors also showhow their framework can be used for generalizing from known to new tasks, and evaluate the method on three simulatedrobotics problems, where their method compares favourably to SOTA reinforcement learning and control methods.    Quality   Generally, the derivations seem correct and principled (although I'm unsure about the task generalization, see below).Related work is sufficiently discussed, the authors point out the similarities and differences to related work andcompare to state of the art methods. The experiments are convincing, altough some details are missing (see below).  The task generalization seems odd: (13) states that the exponent of the reward functions will be linearly combinedto yield the exponent of the reward function of the new task. However, it's not clear to me how that results in the exponent of the value function being the same kind of linear combination (14). In any case, this should be explained,but I think that actually there might be an error here ((14) also states psi is the exponent of the cost function,which holds only for the last time step as shown in the equation around line 247).As defined by the authors, the task generalization is specific to the case that the task is defined by a target, although it seems that othertask variations would be possible as long as a suitable task similarity kernel can be defined.  The experiments considers relevant and realistic tasks, and compare to SOTA methods. The technical details are a bit sparsein the experimental section: the paper should mention what the dynamics and reward functions are, and how long sampled trajectoriesare, in order for the experiment to be reproducible. (Possibly in the appendix). One baseline is iterative PI control,is this one of the methods in table 1? It would be insightful to see how it compares on these qualitative aspects. The comparison plots in Figures 1 and 2 should show error bars. The reported score is the exponent of the value fc, psi. Is this the psi as calculated by the robot? What if the robot is overly optimistic in its calculation of psi? It wouldbe more objective to report the average (or cumulative (discounted)) reward (that is also what the algorithm sets out to optimize).      Clarity  Generally, the paper is well-written and explains the proposed methods rather well. The comparison to other methods, e.g. in Table 1, helps understanding the relationship to the SOTA. There are a couple of grammatical and formatting errors, see below under Minor points. One confusing point is that the GP has different regression targets in (2) (line 094) and line 208,which doesn't include the reference controls. This should be made consistent or explained. The authors should explain how the hyperparameters of the GP are set. It's unclear what's meant by "the posterior distribution can be obtained by constraining the joint distribution" -- is conditioning meant?Algorithm 1 is confusing - here, it looks as if an open-loop control sequence is optimized while the rest of the paper discusses learning a feed-back controller. If it's really just an open-loop sequence u_1 ... u_T that's returned in line 13 of the algorith, how can the algorithm deal with the noise? This shouldbe clarified or corrected.      Originality and Significance    Although the main ingredients of the proposed algorithm have been used in earlier algorithms (propagation through an learnedstochastic forward model as in [15], path integral control), the algorithm combines these in a novel way. As far as I know, this is an original formulation that seems to attain very good performance. By evaluating on realistic problems against strongbenchmarks, the proposed algorithm can be concluded to be a significant improvement over prior work.    Comments on the rebuttal:Thanks for clarifying (14). Still, it seems there is a typo and Psi should be Phi (or Psi_t+dt should be included in the middle part). To me, it's therefore unfortunately still unclear what's meant here. I changed my confidence& score accordingly. Thanks for clarifying the open-loop part, I would really change the notation here to avoid confusion. I'm also still confused about what meant by the psi in figure (1), as I understand it this is the exponent of the final costs, but average cost would be more standard. Maybe the average cost could additionally be reported in the supplementary material?  Minor points    * Although generally well-written, the paper has some grammar issues. I recommend letting someone proofread it  (line 040 "has been existed", line 143 (while -> in contrast ?), line 231 - comma instead of full stop or reformulate,   line 345 is based on ... model -> are based on ... models )   * the authors should avoid creating empty sections (e.g. between sec 2 and sec 2.1)  * there are some formatting issues (margins in line 128, table 1, brackets not scaled correctly in (8))  * would be clearer if the authors state how to obtain \tilde{G} from G* line 192 table.1. -> Table 1.  * notation: in (13) x_t is written boldface on the right-hand side but italic on the left-hand side.  * line 339: Pilco requires an optimizer for policy evaluation -> is policy improvement meant?  * line 332 "6 actuators on each joint" -> is "one actuator per joint" meant?