Compatible Reward Inverse Reinforcement Learning
Inverse Reinforcement Learning (IRL) is an effective approach to recover a reward function that explains the behavior of an expert by observing a set of demonstrations. This paper is about a novel model-free IRL approach that, differently from most of the existing IRL algorithms, does not require to specify a function space where to search for the expert's reward function. Leveraging on the fact that the policy gradient needs to be zero for any optimal policy, the algorithm generates a set of basis functions that span the subspace of reward functions that make the policy gradient vanish. Within this subspace, using a second-order criterion, we search for the reward function that penalizes the most a deviation from the expert's policy. After introducing our approach for finite domains, we extend it to continuous ones. The proposed approach is empirically compared to other IRL methods both in the (finite) Taxi domain and in the (continuous) Linear Quadratic Gaussian (LQG) and Car on the Hill environments.
This paper proposes an approach for behavioral cloning that constructs a function space for a particular parametric policy model based on the null space of the policy gradient. 
I think a running example (e.g., for discrete MDP) would help explain the approach. I found myself flipping back and forth from the Algorithm (page 6) to the description of each step. I have some lingering confusion about using Eq. (4) to estimate discounted vectors d(s,a) since (4) only computed d(s). I assume a similar estimator is employed for d(s,a).
My main concern with the paper’s approach is that of overfitting and lack of transfer. The main motivation of IRL is that the estimated reward function can be employed in other MDPs (without rewards) that have similar feature representations. Seeking to construct a sufficiently rich feature representation that can drive the policy gradient to zero suggests that there is overfitting to the training data at the expense of generalization abilities. I am skeptical then that reasonable generalization can be expected to modifications of the original MDP (e.g., if the dynamics changes or set of available actions changes) or to other MDPs. What am I missing that allows this approach to get beyond behavioral cloning?
My understanding of the Taxi experiment in particular is that data is generated according to the policy model under line 295 and that the proposed method uses this policy model, while comparison methods are based on different policy models (e.g., BC instead estimates a Boltzmann policy). This seems like an extremely unfair advantage to the proposed approach that could account for all of the advantages demonstrated over BC. What happens when likelihood is maximized for BC under this actual model? Experiments with data generated out of model and generalization to making predictions beyond the source MDP are needed to improve the paper.
In summary, though the method proposed is interesting, additional experiments that mimic “real-world” settings are needed before publication to investigate the benefits of the approach suitability to noisy data and for generalization purposes.