DISCRIMINATOR-ACTOR-CRITIC: ADDRESSING SAMPLE INEFFICIENCY AND REWARD BIAS IN ADVERSARIAL IMITATION LEARNING
Algorithms for imitation learning based on adversarial optimization, such as generative adversarial imitation learning (GAIL) and adversarial inverse reinforcement learning (AIRL), can effectively mimic demonstrated behaviours by employing both reward and reinforcement learning (RL). However, applications of such algorithms are challenged by the inherent instability and poor sample efficiency of on-policy RL. In particular, the inadequate handling of absorbing states in canonical implementations of RL environments causes an implicit bias in reward functions used by these algorithms. While these biases might work well for some environments, they lead to sub-optimal behaviors in others. Moreover, despite the ability of these algorithms to learn from a few demonstrations, they require a prohibitively large number of the environment interactions for many real-world applications. To address these issues, we first propose to extend the environment MDP with absorbing states which leads to task-independent, and more importantly, unbiased rewards. Secondly, we introduce an off-policy learning algorithm, which we refer to as Discriminator-Actor-Critic. We demonstrate the effectiveness of proper handling of absorbing states, while empirically improving the sample efficiency by an average factor of 10. Our implementation is available online 1 .
The paper suggests to use TD3 to compute an off-policy update instead of the TRPO/PPO updates in GAIL/AIRL in order to increase sample efficiency.
The paper further discusses the problem of implicit step penalties and survival bias caused by absorbing states, when using the upper-bounded/lower-bounded reward functions log(D) and -(1-log(D)) respectively. To tackle these problem, the paper proposes to explicit add a unique absorbing state at the end of each trajectory, such that its rewards can be learned as well.
Pro:
The paper is well written and clearly presented. 
Using a more sample efficient RL method for the policy update is sensible and turned out effective in the experiments.
Properly handling simulator resets in MDPs is a well known problem in reinforcement learning that I think is insufficiently discussed in the context of IRL.
Cons:
The contributions seem rather small.
a) Replacing the policy update is trivial, since the rl methods are used as black-box modules for the discussed AIL methods. 
b) Using importance weighting to reuse old trajectories for the discriminator update hardly counts as a contribution either--especially when the importance weights are simply omitted in practice. I also think that the reported problems due to the high variance have not been sufficiently investigated. There should be a better solution than just pretending that the replay buffer corresponds to roll-outs of the current policy. Would it maybe help to use self-normalized importance weights? The paper does also not analyze how such assumption/approximation affects the theoretical guarantees.
c) The problem with absorbing states is in my opinion the most interesting contribution of the paper. However, the discussion is rather shallow and I do not think that the illustrative example is very convincing. Section 4.1.1. argues that for the given policy roll-out, the discriminator reward puts more reward on the policy trajectory than the expert trajectory. However, it is neither surprising nor problematic that the discriminator reward does not produce the desired behavior during learning. By assigning more cumulative reward for s2_a1->s1 than for s2_a2->g, the policy would (after a few more updates) choose the latter action much less frequently than with probability 0.5 and the corresponding reward would grow towards infinity until at some point Q(s2,a2) > Q(s2,a1)--when the policy would match the expert exactly. The illustrative example also uses more policy-labeled transitions than agent-labeled ones for learning the classifier, which may also be problematic. The paper further argues that a strictly positive reward function always rewards a policy for avoiding absorbing states, which I think is not true in general. A strictly positive reward function can still produce arbitrary large reward for any action that reaches an absorbing state. Hence, the immediate reward for choosing such action can be made larger than the discounted future reward when not ending the episode (for any gamma < 1). Even for state-only reward functions the problem does not persist when reseting the environment after reaching the absorbing state such that the training trajectories contain states that are only reached if the simulator gets reset. Hence, I am not convinced that adding a special absorbing state to the trajectory is necessary if the simulation reset is correctly implemented. This may be different for resets due to time limits that can not be predicted by the last state-action tuple. However, issues relating to time limits are not addressed in the paper. I also think that it is strange that the direct way of computing the return for the terminal state is much less stable than recursively computing it and think that the paper should include a convincing explanation.
---------------
Update 21.11.2018
I think my initial assessment was too positive. During the rebuttal, I noticed that the discussion of reward bias was not only shallow but also wrong in some aspects and very misleading, because problems arising from hacky implementations of some RL toolboxes were discussed as theoretical shortcoming of AIL algorithms. Hence, I think the initial submission should be clearly rejected. However, the authors submitted a revised version that presents the root of the observed problem much more accurately. I think that the revised version is substantially better than the original submission. However, I think that my initial rating is still valid (better: became valid), because the main issues that I raised for the initial submission still apply to the current revision, namely:
- The technical contributions are minor.
- The theoretical discussion (in particular regarding absorbing states) is quite shallow.
The merits of the paper are:
- Good results due to off-policy learning
- Raising awareness and providing a fix for a common pitfall 
I think that the problems arising from incorrectly treated absorbing states needs to be discussed more profoundly. 
Some suggestions: 
Section 3.1
"As we discuss in detail in Section 4.2 [...]"
I think this should refer to section 4.1. Also the discussion should in section 4.1 should be a bit more detailed. How do common implementations implicitly assign zero rewards? Which implementations are affected? Which papers published inferior results due to this bug? I think it is also important to note, that absorbing states are hidden from the algorithm and that the reward function is thus only applied to non-absorbing states.
"We will demonstrate empirically in Section 4.1 [...]"
The demonstration is currently missing. I think it would be nice to illustrate the problem on a simple example. The original example might actually work, as shown by the code example of the rebuttal, however the explanation was not convincing. Maybe it would be easier to argue with a simpler algorithm (e.g MaxEnt-IRL, potentially projecting the rewards to positive values)?
Section 3.1 seems to focus too much on resets that are caused by time limits. Such resets are inherently different from terminal states such as falling down in locomotion tasks, because they can not be modelled with the given MDP formulation unless time is considered part of the state. Indeed, I think that for infinite horizon MDPs without time-awareness, time limits can not be modelled using absorbing states (I think the RL book misses to mention that time needs to be part of the state such that the policy remains Markovian, which is a bit misleading). Instead those resets are often handled by returning an estimate of the future return (bootstrapping). This treatment of time limits is already part of the TD3 implementation and as far as I understood not the focus of the paper. Instead section 3.1. should focus on resets caused by task failure/completion, which can actually be modelled with absorbing states, because the agent will always transition to the absorbing state when a terminal state is reached which is in line with Markovian dynamics.
Section 4.2 should also add a few more details. Did I understand correctly, that when computing the return R_T the sum is indeed finite and stopped after a fixed horizon? If yes, this should be reflected in the equation, and the horizon should be mentioned in the paper. The paper should also better explain how  the proposed fix enables the algorithm to learn the reward of the absorbing state. For example, section 4.2. does not even mention that the state s_a was added as part of the solution. 
-------------
Update 22.11.2018
By highlighting the difference between termination due to time-limits and termination due to task completion, and by better describing how the proposed fix addresses the problem of reward bias that is present in common AIL implementations, the newest revision further improves the submission. 
I think that the submission can get accepted and I adapted my rating accordingly.
Minor:
Conclusion should also squeeze in somehow that the reward biases are caused by the implementations.
Typo in 4.2: "Thus, when sample[sic] from the replay buffer AIL algorithms will be able to see absorbing states there[sic]
were previous hidden, [...]"