Reinforcement learning typically requires carefully designed reward functions in order to learn the desired behavior. We present a novel reward estimation method that is based on a finite sample of optimal state trajectories from expert demonstrations and can be used for guiding an agent to mimic the expert behavior. The optimal state trajectories are used to learn a generative or predictive model of the "good" states distribution. The reward signal is computed by a function of the difference between the actual next state acquired by the agent and the predicted next state given by the learned generative or predictive model. With this inferred reward function, we perform standard reinforcement learning in the inner loop to guide the agent to learn the given task. Experimental evaluations across a range of tasks demonstrate that the proposed method produces superior performance compared to standard reinforcement learning with both complete or sparse hand engineered rewards. Furthermore, we show that our method successfully enables an agent to learn good actions directly from expert player video of games such as the Super Mario Bros and Flappy Bird.
To speed up RL algorithms, the authors propose a simple method based on utilizing expert demonstrations. The proposed method consists in explicitly learning a prediction function that maps each time-step into a state. This function is learned from expert demonstrations. The cost of visiting a state is then defined as the distance between that state and the predicted state according to the learned function. This reward is then used in standard RL algorithms to learn to stick close to the expert's demonstrations. An on-loop variante of this method consists of learning a function that maps each state into a next state according to the expert, instead of the off-loop function that maps time-steps into states.
While the experiments clearly show the advantage of this method, this is hardly surprising or novel. The concept of encoding the demonstration explicitly in the form of a reward has been around for over a decade. This is the most basic form of teaching by demonstration. Previous works had used other models for generalizing demonstrations (GMMs, GPs, Kernel methods, neural nets etc..). This paper uses a three layered fully connected auto-encoder (which is not that deep of a model, btw) for the same purpose. The idea of using this model as a reward instead of directly cloning the demonstrations is pretty straightforward. 
Other comments:
- Most IRL methods would work just fine by defining rewards on states only and ignoring actions all together. If you know the transition function, you can choose actions that lead to highly rewarding states, so you don't need to know the expert's executed actions.
- "We assume that maximizing likelihood of next step prediction in equation 1 will be globally optimized in RL". Could you elaborate more on this assumption? Your model finds rewards based on local state features, where a greedy (one-step planning) policy would reproduce the expert's demonstrations (if the system is deterministic). It does not compare the global performance of the expert to alternative policies (as is typically done in IRL).
- Related to the previous point: a reward function that makes every step of the expert optimal may not be always exist. The expert may choose to go to terrible states with the hope of getting to a highly rewarding state in the future. Therefore, the objective functions set in this paper may not be the right ones, unless your state description contains features related to future states so that you can incorporate future rewards in the current state (like in the reacher task, where a single image contains all the information about the problem). What you need is actually features that can capture the value function (like in DQN) and not just the immediate reward (as is done in IRL methods). 
- What if in two different trajectories, the expert chooses opposite actions for the same state appearing in both trajectories? For example, there are two shortest paths to a goal, one starts with going left and another starts with going right. If you try to generate a state that minimizes the sum of distances to the two states (left and right ones), then you may choose to remain in the middle, which is suboptimal. You wouldn't have this issue with regular IRL techniques, because you can explain both behaviors with future rewards instead of trying to explain every action of the expert using only local state description.