NEURAL PROBABILISTIC MOTOR PRIMITIVES FOR HUMANOID CONTROL
We focus on the problem of learning a single motor module that can flexibly express a range of behaviors for the control of high-dimensional physically simulated humanoids. To do this, we propose a motor architecture that has the general structure of an inverse model with a latent-variable bottleneck. We show that it is possible to train this model entirely offline to compress thousands of expert policies and learn a motor primitive embedding space. The trained neural probabilistic motor primitive system can perform one-shot imitation of whole-body humanoid behaviors, robustly mimicking unseen trajectories. Additionally, we demonstrate that it is also straightforward to train controllers to reuse the learned motor primitive space to solve tasks, and the resulting movements are relatively naturalistic. To support the training of our model, we compare two approaches for offline policy cloning, including an experience efficient method which we call linear feedback policy cloning. We encourage readers to view a supplementary video summarizing our results.
This paper considers the problem of transferring motor skills from multiple experts to a student policy. To this end, the paper proposes two approaches: (1) an approach for policy cloning that learns to mimic the (local) linear feedback behavior of an expert (where the expert takes the form of a neural network), and (2) an approach that learns to compress a large number of experts via a latent space model. The approaches are applied to the problem of one-shot imitation from motion capture data (using the CMU motion capture database). The paper also considers an extension of the proposed approach to the problem of high-level planning; this is done by treating the learned latent space as a new action space and training a high-level policy that operates in this space. 
Strengths:
S1. The supplementary video was clear and helpful in understanding the setup.
S2. The paper is written in a generally readable fashion.
S3. The related work section does a thorough job of describing the context of the work.  
However, I have some significant concerns with the paper. These are described below. 
Significant concerns:
C1. My biggest concern is that the paper does not make a strong case for the benefits of LPFC over simpler strategies. The results in Figure 3 demonstrate that a linear feedback policy computed along the expert's nominal trajectory performs as well as (and occasionally even better than) LPFC. This is quite concerning.
C2. Moreover, as the authors themselves admit, "while LPFC did not work quite as well in the full-scale model as cloning from noisy rollouts, we believe it holds promise insofar as it may be useful in rollout-limited settings...". However, the paper does not present any theoretical/experimental evidence that would suggest this.
C3. Another concern has to do with the two-step procedure for LPFC (Section 2.2), where the first step is to learn an expert policy (in the form of a neural network) and the second step is to perform behavior cloning by finding a policy that tries to match the local behavior of the expert (i.e., finding a policy that attempts to produce similar actions as the expert policy linearized about the nominal trajectory). This two-step procedure seems unnecessary; the paper does not make a case for why the expert policies are not chosen as linear feedback controllers (along nominal trajectories) in the first place.
C4. The linearization of the expert policy produced in (1) may not lead to a stabilizing feedback controller and could easily destabilize the system. It is easy to imagine cases where the expert neural network policy maintains trajectories of the system in a tube around the nominal trajectory, but whose linearization does not lead to a stabilizing feedback controller. Do you see this in practice? If not, is there any intuition for why this doesn't occur? If this doesn't occur in practice, this would suggest that the expert policies are not highly nonlinear in the neighborhood of states under consideration (in which case, why learn neural network experts in the first place instead of directly learning a linear feedback controller as the expert policy as suggested in C3?)
C5. I would have liked to have seen more implementation details in Section 3. In particular, how exactly was the linear feedback policy along the expert's nominal trajectory computed? Is this the same as (2)? Or did you estimate a linear dynamical model (along the expert's nominal trajectory) and then compute an LQR controller? More details on the architecture used for the behavioral cloning baseline would also have been helpful (was this a MLP? How many layers?)
Minor comments:
- There are some periods missing at the end of equations (eqs. (1), (2), (6), (8), (9)).