Consider an imitation learning problem that the imitator and the expert have different dynamics models. Most of the current imitation learning methods fail because they focus on imitating actions. We propose a novel state alignment based imitation learning method to train the imitator to follow the state sequences in expert demonstrations as much as possible. The state alignment comes from both local and global perspectives and we combine them into a reinforcement learning framework by a regularized policy update objective. We show the superiority of our method on standard imitation learning settings and imitation learning settings where the expert and imitator have different dynamics models.
Summary of claims:
The paper proposes an imitation learning method that aims to align state distributions rather than state-action distributions to account for cases where the imitator dynamics differ from expert dynamics. They achieve this by two objectives: one local, the other global. The local objective aligns the next state to be close to the expert's next state in each transition by first training a VAE on the expert demonstrations, and using the trained VAE in conjunction with a pretrained inverse dynamics model to compute the action that the imitator needs to imitate. The global objective tries to do a global alignment of states encountered in the imitator and expert trajectories, by minimizing the Wasserstein distance between the two trajectory distributions. The paper claims that using these two objectives results in a method that outperforms existing inverse reinforcement learning and behavior cloning approaches in settings where the imitator and expert dynamics differ.
Decision:
I recommend the paper to be rejected. I have three main reasons for my decision (with more details in the next section):
1. The paper is very poorly written : A lot of details are missing in the paper, notation is not standardized,  related work is just a list of previous papers without any context on how the proposed method is related, previous methods are referred to without any citations, and quite a few blanket statements which are not substantiated.
2. Incomplete approach description: Quite a few components of the approach are not explained (or even discussed), no intuition provided for the choices made in the approach, the concept of different dynamics is not formalized, some technical inconsistencies in the algorithm, no formal problem statement (which would really help in standardizing notation), and most claims made about the approach are not justified or substantiated
3. Poor experiments: Experiments are not well chosen to reflect the premise and claims of the paper, little to no details given for how the baseline approaches were trained, no details on policy parameterization, and missing comparison with baseline approaches in some experiments
Comments:
(1) Problem setup: 
(a)Problem setup is very vague and not formalized. 
(b) Differing dynamics could mean several things: different agent dynamics (like different action spaces; different actuators; etc.), different environment dynamics (different moving obstacles in the world;) etc. 
(c) Basically, different dynamics can mean a lot more than what was accounted for in the paper
(2) Blanket Statements: 
(a) The authors keep saying that their framework is more flexible without any justification as to why, 
(b) "simply train an inverse dynamics model"- training an inverse dynamics model can be very hard especially when environment dynamics are stochastic/when the inverse model is multimodal, 
(c) "constraint becomes loosened" - this statement doesn't make any sense without more explanation,
(d) Several other blanket statements about existing approaches
(3) Notation : 
(a) Notation was never standardized in the paper, 
(b) what is \phi? what is the input-output of \phi? 
(c) What is \theta_old? What is \sigma? 
(d) There are a lot of things that needed explanation, especially in the algorithm
(4) Related Work : 
(a) The related work section is just a dump of citations without giving any context for where the proposed work lies in the spectrum of these works. How does it compare? Why is it better/worse? 
(b) Missing related work that was publshed in ICML 2019 that has a very similar approach in matching state distributions ("Provably efficient Imitation Learning from Observations Alone" or FAIL) and works very well.
(5) Motivation : 
(a) The approach, in general, needs better motivation. The toy example in the introduction was good but the experiments did not reflect the complexity of that example. 
(b) Try to have a running example in the paper that will help you motivate the approach better.
(6) Background : 
(a) The background section is very minimal and lacks any details necessary. 
(b) I had to read the beta-VAE paper to understand what it does. 
(c) The section also lacks any minimal background in IL/RL and the notation could also have been standardized in this section
(7) Figures: 
(a) All the figures in this paper could use a lot of improvement in terms of descriptive captions, more informative legends, bigger fonts, descriptive text, and more figures as well
(8) Algorithm : 
(a) The algorithm was not referenced anywhere in the text, 
(b) No definition for \tau, details on pretraining inverse dynamics model lacking in both text and algorithm, 
(c) *Policy prior is used to pretrain policy before the VAE was trained!* which doesn't make any sense since the policy prior is obtained using the VAE, 
(d) the equation at the end of Sec 4.3 has r(s, a) whereas reward is defined as r(s_t, s_{t+1}) but they are not equivalent when dynamics are stochastic
(9) Experiments: 
(a) What is AIRL? No reference was given. 
(b) Why is keeping the variance of the policy constant reasonable? How do you come up with the value? 
(c) How do you pretrain the VAE, invserse dynamics model? 
(d) The setup of making the ant's legs smaller or body heavier seems very artificial. I am sure its easy to come up with more realistic setups in navigation domains, for example. Try to use more realistic experiments in the future. 
(e) For results in Fig 3, is AIRL, GAIL also pretrained with VAE or by BC? Seems like SAIL was pretrained but the others weren't since SAIL starts off with a high score at the start. 
(f) For Sec 5.1.2, comparison with baseline approaches are missing. Legends for the plots are terribly small.
Conceptual questions:
1. How do you account for cases where due to differing dynamics, states reached by expert in demonstrations are unreachable by the imitator? 
2. How do you account for cases where the environment dynamics changes between expert and imitator?
3. Why does using Wasserstein distance make sense? Why not other f-divergences? Also, matching global distributions can be very misleading if you have states that are visited multiple times in the same trajectory. FAIL recommends matching state distribution at each time-step instead and is much more stable
4. How is this different from GAIL where we match state visitation distribution (instead of state-action visitation distribution?)
Things to improve:
1. Writing needs to be improved a lot
2. Better experiments - more realistic domains
3. Approach needs to be explained more formally