A significant challenge for the practical application of reinforcement learning to real world problems is the need to specify an oracle reward function that correctly defines a task. Inverse reinforcement learning (IRL) seeks to avoid this challenge by instead inferring a reward function from expert behavior. While appealing, it can be impractically expensive to collect datasets of demonstrations that cover the variation common in the real world (e.g. opening any type of door). Thus in practice, IRL must commonly be performed with only a limited set of demonstrations where it can be exceedingly difficult to unambiguously recover a reward function. In this work, we exploit the insight that demonstrations from other tasks can be used to constrain the set of possible reward functions by learning a "prior" that is specifically optimized for the ability to infer expressive reward functions from limited numbers of demonstrations. We demonstrate that our method can efficiently recover rewards from images for novel tasks and provide intuition as to how our approach is analogous to learning a prior.
This paper aims to address the problem of lacking sufficient demonstrations in inverse reinforcement learning (IRL) problems. They propose to take a meta learning approach, in which a set of i.i.d. IRL tasks are provided to the learner and the learner aims to learn a strategy to quickly recover a good reward function for a new task that is assumed to be sampled from the same task distribution. Particularly, they adopt the gradient-based meta learning algorithm, MAML, and the maximal entropy (MaxEnt) IRL framework, and derive the required meta gradient expression for parameter update. The proposed algorithm is evaluated on a synthetic grid-world problem, SpriteWorld. The experimental results suggest the proposed algorithm can learn to mimic the optimal policy under the true reward function that is unknown to the learner. 
Strengths: 
1) The use of meta learning to improve sample efficiency of IRL is a good idea.
2) The combination of MAML and MaxEnt IRL is new to my knowledge. 
3) Providing the gradient expression is useful, which is the main technical contribution of this paper. (But it needs to be corrected; see below.)
4) The paper is well motivated and clearly written "in a high level" (see below). 
Weakness: 
1) The derivation of (5) assumes the problem is tabular, and the State-Visitations-Policy procedure assumes the dynamics/transition of the MDP is known. These two assumption are rather strong and therefore should be made explicitly in the problem definition in Section 3.
2)  Equation (8) is WRONG. The direction of the derivation takes is correct, but the final expression is incorrect. This is mostly because of the careless use of notation in derivation on p 15 in the appendix (the last equation), in which the subscript i is missed for the second term. The correct expression of (8) should have a rightmost term in the form  (\partial_\theta r_\theta) D  (\partial_\theta r_\theta)^T, where D is a diagonal matrix that contains \partial_{r_i} (\E_{\tau} [ \mu_\tau])_i and i is in 1,...,|S||A|. 
3) Comparison with imitation learning and missing details of the experiments. 
a) The paper assumes the expert is produced by the MaxEnt model. In the experiments, it is unclear whether this is true or not, as the information about the demonstration and the true reward is not provided. 
b) While the experimental results suggest the algorithm can recover the similar performance to the optimal policy of the true reward function, whether this observation can generalize outside the current synthetic environment is unclear to me. In imitation learning, it is known that the expert policy is often sub-optimal, and therefore the goal in imitation learning is mostly only to achieve expert-level performance. Given this, the way this paper evaluate the performance is misleading and improper to me, which leads to an overstatement of the benefits of the algorithm. 
c) It would be interesting to compare the current approach with, e.g., the policy-based supervised learning approach to imitation learning (i.e. behavior cloning). 
4) The rigorousness in technicality needs to be improved. While the paper is well structured, the writing at the mathematical level is careless, which leads to ambiguities and mistakes (though one might be able to work out the right formula after going through the details of the entire paper). Below I list a few points. 
    a) The meta-training set {T_i; i=1,...,N} and the meta-test set {T_j; i=1,...,M} seems to overload the notation. I suppose this is unintentional but it may appear that the two sets share the first T_1,.., T_M tasks, e.g., when N>=M, instead of being disjoint. 
    b) The set over which the summation is performed in (4) is unclear; alpha in (4) is not defined, though I guess it's a positive step size.
    c) On p4, "we can view this problem as aiming to learn a prior over the intentions of human demonstrators" is an overstatement to me. At best, this algorithm learns a prior over rewards for solving maximal entropy IRL, not intention. And the experiment results do not corroborate  the statement about "human" intention.
    d) On p4,  "since the space of relevant reward functions is much smaller than the space of all possible rewards deﬁnable on the raw observations" needs to be justified. This may not be true in general, e.g., learning the set of relevant functions may require a larger space than learning the reward functions.
    e) The authors call \mu_\tau the "state" visitation, but this is rather confusing, as it is the visiting frequency of state and action (which is only made clear late in the appendix). 
    f) On p5, it writes "... taking a small number of gradient steps on a few demonstrations from given task leads" But the proposed algorithm actually only takes "one" gradient step in training. 
    g) The convention of derivatives used in the appendix is the transpose of the one used in the main paper.
Minor points: 
1) typo in (2) 
2) p_\phi is not defined, L_{IRL} is not defined, though the definition of both can be guessed.
3) T^{tr} seems to be typo in (11)
4) A short derivation of (2) in the Appendix would be helpful.