GENERATIVE PREDECESSOR MODELS FOR SAMPLE- EFFICIENT IMITATION LEARNING
We propose Generative Predecessor Models for Imitation Learning (GPRIL), a novel imitation learning algorithm that matches the state-action distribution to the distribution observed in expert demonstrations, using generative models to reason probabilistically about alternative histories of demonstrated states. We show that this approach allows an agent to learn robust policies using only a small number of expert demonstrations and self-supervised interactions with the environment. We derive this approach from first principles and compare it empirically to a stateof-the-art imitation learning method, showing that it outperforms or matches its performance on two simulated robot manipulation tasks and demonstrate significantly higher sample efficiency by applying the algorithm on a real robot.
The submission builds up on recent advances in neural density estimation to develop a new algorithm for imitation learning based on a probabilistic model for predecessor state dynamics. In particular, the method trains masked autoregressive flows as a probabilistic model for state action pairs conditioned on future states. This model is used to estimate the gradient of the stationary distribution of a policies visited states. Finally, the proposed objective uses this estimate and the gradient of the log likelihood of expert actions under the policy to maximise the similarity of the expert’s and agent’s stationary state-action distributions. 
The proposed method outperforms existing imitation learning approach (GAIL & BC) on 2 simulation-based manipulation tasks. It performs particularly well in terms of sample efficiency. 
The magnitude of difference between the sample efficiencies of GAIL and the proposed approach seems quite surprising and it would be beneficial if the authors could explicitly state if the measured number of samples include the ones used for training of the probabilistic model as well as the policy (apologies if I have missed a section fulfilling this purpose).
While the improvements on the presented experiments are clear, the experimental section represents a small shortcoming of the submitted paper. The 2 experiments (clip and peg insertion) are quite similar in type and to not take into account other common domains e.g. locomotion tasks from the original GAIL paper. Furthermore, an additional comparison to SAIL would be recommended since the approaches are closely related as the authors acknowledge. The provided comparison with different types of available expert data is quite interesting and could possibly be extended to test other state-of-the-art methods (action-free versions of GAIL, AIRL,etc.).
Nonetheless, the paper overall presents a strong submission based on novelty & relevance of the proposed method and is recommended for publication. 
Minor issues:
- Related work: improve transitions between the section about trajectory tracking and BC.
- Ablation studies with less flexible probabilistic models would strengthen the experiment section further. 
- Add derivation from Eq. 3 to 4 and 5 to appendix to render the paper more self-contained and easier to access.
- A release of the code base would further strengthen the contributions of the submission.
General recommendation:
- The authors are encouraged to further investigate off-policy corrections for improved convergence.