ADVERSARIAL IMITATION VIA VARIATIONAL INVERSE REINFORCEMENT LEARNING
We consider a problem of learning the reward and policy from expert examples under unknown dynamics. Our proposed method builds on the framework of generative adversarial networks and introduces the empowerment-regularized maximum-entropy inverse reinforcement learning to learn near-optimal rewards and policies. Empowerment-based regularization prevents the policy from overfitting to expert demonstrations, which advantageously leads to more generalized behaviors that result in learning near-optimal rewards. Our method simultaneously learns empowerment through variational information maximization along with the reward and policy under the adversarial learning formulation. We evaluate our approach on various high-dimensional complex control tasks. We also test our learned rewards in challenging transfer learning problems where training and testing environments are made to be different from each other in terms of dynamics or structure. The results show that our proposed method not only learns nearoptimal rewards and policies that are matching expert behavior but also performs significantly better than state-of-the-art inverse reinforcement learning algorithms.
The paper proposes a method for inverse reinforcement learning based on AIRL. It's main contribution is that the shaping function is not learned while training the discriminator, but separately as an approximation of the empowerment (maximum mutual information). This shaping term aims to learn disentangled rewards without being restricted to learning state-only reward functions, which is a major restriction of AIRL.
The main weakness of the paper is, that it does not justify or motivate the main deviations compared to AIRL. The new objective for updating the policy is especially problematic because it does no longer correspond to the RL objective but includes an additional term that biases the policy towards actions that increase its empowerment. Although both terms of the update can be derived independently from an IRL and Empowerment perspective respectively, optimizing the sum was not derived from a common problem formulation. By combining these objectives, the learned reward function may lead to policies that fail to match the expert demonstration without such bias. This does not imply that the approach is not sound per se, however, simply presenting such update without any discussion is insufficient--especially given that it constitutes the main novelty of the approach. I think the paper would be much stronger if the update was derived from an empowerment-regularized IRL formulation. And even then, the implications of such bias/regularization would need to be properly discussed and evaluated, in particular with respect to the trade-off lambda, which--again--is hardly mentioned in the submission. I'm also not sure if the story of the paper works out; when we simply want to use empowerment as shaping term, why not use two separate policies for computing the empowerment and reward function respectively. Is the bias in the policy update maybe more important than the shaping term in the discriminator update for learning disentangled rewards?
Keeping these issues aside, I actually like the paper. It tackles the main drawback of AIRL and the idea seems quite nice. Having a reward function that does not actively induce actions that can be explained by empowerment, may not always be appropriate, but often enough it may be a sensible approach to get more generalizable reward functions. The paper is also well written with few typos. The parts that are discussed are clear and the experimental results seem fine as well (although more experiments on the reward transfer would be nice).
Minor notes:
I think there is a sign error in the policy update
Typo in the theorem, grantee should be guarantee
Question:
Please confirm that the reward transfer was learned with a standard RL formulation. Does the learned policy change, when we use the empowerment objective as well?
Update (22.11)
I think that the revised version is much better than the original submission because it now correctly attributes the improved generalization to an inductive bias in the policy update.  However, the submission still seems borderline to me. 
- The proposed method uses the empowerment both for regularization as well as for reward shaping, but it is not clear whether the latter improves generalization. If the reward shaping was not necessary, it would be cleaner to use empowerment only for regularization. If the reward shaping is beneficial, this should be shown in an ablative experiment.
- The benefit of using empowerment (whether for reward shaping or for regularization) should be discussed. Empowerment for generalization is currently hardly motivated.
- The derivation could be a bit more rigorous.
As the presentation is now much more sound, I slightly increased my rating.