We study the problem of Reinforcement learning from demonstrations (RLfD), where the learner is provided with both some expert demonstrations and reinforcement signals from the environment. One approach leverages demonstration data in a supervised manner, which is simple and direct, but can only provide supervision signal over those states seen in the demonstrations. Another approach uses demonstration data for reward shaping. By contrast, the latter approach can provide guidance on how to take actions, even for those states are not seen in the demonstrations. Specifically, such reward shaping approach trains an agent not only to imitate demonstrated actions when it encounters demonstrated states, but also to reach demonstrates states, when it confronts states that are not observed in the demonstration data. However, existing algorithms in the latter one adopt shaping reward which is not directly dependent on current policy, limiting the algorithms to treat demonstrated states the same as other states, and fail to directly exploit supervision signal in demonstration data. In this paper, we propose a novel objective function with policy-dependent shaping reward, so as to get the best of both worlds. We present a convergence proof for policy iteration of the proposed objective, under the tabular setting. Then we develop a new practical algorithm, termed as Demonstration Actor Critic (DAC). Experiments on a range of popular benchmark sparse-reward tasks shows that our DAC method obtains a significant performance gain over five strong and off-the-shelf baselines.
This paper presents a method for doing RL from demonstrations in continuous control tasks. It combines both an augmented reward for minimizing the KL between the policy and the expert actions as well as directly minimizing that KL in the policy. Results on 5 sparse reward mujoco tasks show that it out-performs other related methods.
The motivation for the paper is difficult to follow. They claim that using demonstration data in a supervised manner "cannot generalize supervision signal over those states unseen in the demonstrations," but most of these approaches are using neural networks and definitely are generalizing those signals to other states. Whether they're generalizing accurately or not is a different question. In contrast, they say that reward shaping approaches do not suffer that problem because they evaluate trajectories rather than states, but there will still be a problem of generalizing to new trajectories. 
The abstract is even more confusing as it tries to jump straight into the issues with these approaches without any explanation. I don't think there's enough space in the abstract to go into that level of detail.
The description of DQfD and DDPGfD in the related work is not accurate. They're described as "treating demonstration data as self-generated data," but in fact they both add supervised losses to more closely match the demonstrated data. https://ieeexplore.ieee.org/document/8794074 is another method built on DDPG that has both a critic and actor loss like yours and would make a useful comparison.
The related work section should also discuss and compare/contrast to GAIL, I was surprised that wasn't in there, especially since you also use a discriminator to differentiate expert and agent actions. 
The end of the related work section is not very clear, you say these methods are problematic because "the adopted shaping reward yields no direct dependence on the current policy" but there's no explanation or motivation for why that would be a problem. 
Assumption 1 seems like a very strong assumption that would not be true for many human experts. 
For the experiments, I wonder about the impact of only using sparse reward tasks. Converting the tasks to sparse reward in this way makes them partially observable, and then potentially the expert demonstrations are required to overcome that partial observability. How do the methods compare on the unmodified tasks? There was nothing specific in your algorithm that meant it should specifically address sparse reward tasks. What about tasks that are naturally sparse reward? 
Overall, the algorithm is interesting and the results are nice. The motivation and related work need to be made clearer to situate this work with the other related works. And the experiments should go beyond these tasks that have been modified to have sparse rewards. 
The revised version of the paper addresses many of my concerns about the motivation, related works, and comparisons with GAIL, so I'm updating my score to Weak Accept.