Imitation learning from demonstrations usually relies on learning a policy from trajectories of optimal states and actions. However, in real life expert demonstrations, often the action information is missing and only state trajectories are available. We present a model-based imitation learning method that can learn environment-specific optimal actions only from expert state trajectories. Our proposed method starts with a model-free reinforcement learning algorithm with a heuristic reward signal to sample environment dynamics, which is then used to train the state-transition probability. Subsequently, we learn the optimal actions from expert state trajectories by supervised learning, while back-propagating the error gradients through the modeled environment dynamics. Experimental evaluations show that our proposed method successfully achieves performance similar to (state, action) trajectory-based traditional imitation learning methods even in the absence of action information, with much fewer iterations compared to conventional model-free reinforcement learning methods. We also demonstrate that our method can learn to act from only video demonstrations of expert agent for simple games and can learn to achieve desired performance in less number of iterations.
Model-Based Imitation Learning from State Trajectories
SIGNIFICANCE AND ORIGINALITY:
The authors propose a model-based method for accelerating the learning of a policy
by observing only the state transitions of an expert trace.
This is an important problem in many fields such as robotics where
finding a feasible policy is hard using pure RL methods.
The authors propose a unique two step method to find a high-quality model-based policy.
First: To create the environment model for the model-based learner, 
 they need a source of state transitions with actions ( St, At,xa St+1 ).
To generate these samples, they first employ a model-free algorithm.
The model-free algorithm is trained to try to duplicate the expert state at each trajectory.
In continuous domains, the state is not unique … so they build a soft next state predictor
that gives a probability over next states favoring those demonstrated by the expert.
Since the transitions were generated by the agent acting in the environment,
these transitions have both states and actions ( St, At, St+1 ).
These are added to a pool.
The authors argue that the policy found by this model-free learner is
not highly accurate or guaranteed to converge, but presumably is good at
generating transitions relevant to the expert’s policy.
(Perhaps slowly reducing the \sigma in the reward would improve accuracy?)
I guess if expert trace data is sparse, the model-free learner can generate a lot 
of transitions which enable it to create accurate dynamics models which in turn
allow it to extract more information out of sparse expert traces?
Second: They then train a model based agent using the collected transitions ( St, At, St+1 ).
They formulate the problem as a maximum likelihood problem with two terms: 
an action dynamics model which is learned from local exploration using the learner’s own actions and outcomes
and expert policy model in terms of the actions learned above 
that maximizes the probability of the observed expert’s trajectory.
This is a nice clean formulation that integrates the two processes.
I thought the comparison to an encoder - decoder network was interesting.
The authors do a good job of positioning the work in the context of recent work in IML.
It looks like the authors extract position information from flappy bird frames, 
so the algorithm is only using images for obstacle reasoning?
QUALITY
The propose model is described fairly completely and evaluated on 
a “reaching" problem and the "flappy bird” game domain.
The evaluation framework is described in enough detail to replicate the results.
Interestingly, the assisted method starts off much higher in the “reacher” task.
Presumably this task is easy to observe the correct actions.
The flappy bird test shows off the difference between unassisted learning (DQN),
model free learning with the heuristic reward (DQN+reward prediction) 
and model based learning. 
Interestingly, DQN + heuristic reward approaches expert performance
while behavioral cloning never achieves expert performance level even though it has actions.
Why does the model-based method only run to 600 steps and stopped before convergence??
Does it not converge to expert level?? If so, this would be useful to know.
There are minor grammatical mistakes that can be corrected.
After equation 5, the authors suggest categorical loss for discrete problems, 
but cross-entropy loss might work better. Maybe this is what they meant.
CLARITY
The overall approach and algorithms are described fairly clearly. Some minor typos here and there.
Algorithm 1 does not make clear the relationship between the model learned in step 2 and the algorithms in steps 4 to 6.
I would reverse the order of a few things to align with a right to left ordering principle. 
In Figure 1, put the model free transition generator on the left and the model-based sample consumer on the right.
In Figure 3, put the “reacher” test on the left and the “flappy bird” on the right.
PROS AND CONS
Interesting idea for learning quickly from small numbers of samples of expert state trajectories. 
Not clear that method converges on all problems. 
Not clear that the method is able to extract the state from video — authors had to extract position manually
(this point is more about their deep architecture than the imitation framework they describe -
though perhaps a key argument for the authors is the ability to work with small numbers of 
expert samples and still be able to train deep methods ) ??
POST REVIEW SUBMISSION:
The authors make a number of clarifying comments to improve the text and add the reference suggested by another reviewer.