We consider the Inverse Reinforcement Learning problem in Contextual Markov Decision Processes. In this setting, the reward, which is unknown to the agent, is a function of a static parameter referred to as the context. There is also an "expert" who knows this mapping and acts according to the optimal policy for each context. The goal of the agent is to learn the expert's mapping by observing demonstrations. We define an optimization problem for finding this mapping and show that when it is linear, the problem is convex. We present and analyze the sample complexity of three algorithms for solving this problem: the mirrored descent algorithm, evolution strategies, and the ellipsoid method. We also extend the first two methods to work with general reward functions, e.g., deep neural networks, but without the theoretical guarantees. Finally, we compare the different techniques empirically in driving simulation and a medical treatment regime.
This paper introduces a formulation for the contextual inverse reinforcement learning (COIRL) problem and proposed three algorithms for solving the proposed problem. Theoretical analysis of scalability and sample complexity are conducted for cases where both the feature function and the context-to-reward mapping function are linear. Experiments were conducted in both a simulated driving domain and a medical treatment domain to compare the three proposed algorithms empirically. Empirical results for using a deep network as the contextual mapping function is also provided.
As a special case of IRL for POMDPs, the contextual IRL problem (with latent contexts) is an interesting research topic that is of interest to researchers working on generalizing IRL to a greater range of real-world applications. This paper was written with clarity and detailed descriptions of experiments. The authors presented their algorithms with thorough theoretical analysis. However, further ablation study and proofs are needed to demonstrate the proposed problem formulation and algorithms outperform existing IRL frameworks.
The authors motivate the problem of COIRL from the formulation of contextual MDP by Hallak et al. (2015). However, the referenced work of Hallak et al. (2015) (published as a preprint on arXiv) does not provide a sufficient argument/evidence for why or in what cases this particular formulation, especially when the context is observed (as assumed by this paper), is better than alternative ones, such as directly modeling the context as part of the state. When a discrete number of contexts exist (as in the simulated driving domain: average vehicle v.s. ambulance), the proposed problem can be reduced to solving a set of normal IRL problems, i.e. learning a reward function under each observed context. For continuous context variables (such as age and weight for patients in the medical treatment domain), an intuitive solution is to model them as part of the state/feature and run normal IRL algorithms. However, the authors did not analyze or show in either of their experiments how existing IRL algorithms compare with the proposed algorithms. Without further explanation, the authors claim, in the fourth paragraph of section 1, that “..Apprenticeship Learning algorithms that try to mimic the expert cannot be used..”, yet it is not trivial to understand why the proposed formulation and algorithms would outperform existing methods/baselines. 
Lastly, it would be more insightful to the readers if the authors can provide some discussion on how they would extend their algorithms to the more interesting/practical case where the context is not directly observed and analyze how latent contexts would affect the performance/complexity of their proposed algorithms.
Overall, this is a well-written paper on an exciting research topic but lacks sufficient analysis and experimental results to support the significance of the intended contributions.
Reference (from the paper): 
Hallak, A., Di Castro, D., & Mannor, S. (2015). Contextual Markov decision processes. arXiv preprint arXiv:1502.02259.
==================================
Edit after rebuttal period: 
- The updated version of the paper includes preliminary results of comparisons with alternative methods and shows that COIRL does have better scalability for modeling problems with large number of observable contexts. It would be great if the authors can show how the baseline performs in the two original experiments as well to understand the improvements gained from adopting COIRL for these problems.
- As stated before, I think COIRL is an exciting research topic on its own. While many recent works on IRL focuses on generalizing to complex problem with deep neural nets as function approximation tools, it is important to analyze the underlying problems from bottom up (in smaller domains and with assumptions that may not always hold) in order to better understand the limits of different solutions. I'd certainly love to see more follow-up works on this topic and how it can be extended to more complex problems.
- Hence, after carefully reading the authors responses and the updated version of the paper, I changed my decision to 'Weak Accept'.