While most approaches to the problem of Inverse Reinforcement Learning (IRL) focus on estimating a reward function that best explains an expert agent's policy or demonstrated behavior on a control task, it is often the case that such behavior is more succinctly represented by a simple reward combined with a set of hard constraints. In this setting, the agent is attempting to maximize cumulative rewards subject to these given constraints on their behavior. We reformulate the problem of IRL on Markov Decision Processes (MDPs) such that, given a nominal model of the environment and a nominal reward function, we seek to estimate state, action, and feature constraints in the environment that motivate an agent's behavior. Our approach is based on the Maximum Entropy IRL framework, which allows us to reason about the likelihood of an expert agent's demonstrations given our knowledge of an MDP. Using our method, we can infer which constraints can be added to the MDP to most increase the likelihood of observing these demonstrations. We present an algorithm which iteratively infers the Maximum Likelihood Constraint to best explain observed behavior, and we evaluate its efficacy using both simulated behavior and recorded data of humans navigating around an obstacle.
In this work, a novel inverse constraint learning method is proposed, where the goal is to find out the constraints over state-action pairs for given demonstration and MDP **including a reward function** (so different from inverse cost learning). The novelty of this work comes from introducing maximum entropy inverse reinforcement learning (MaxEntIRL) framework to previous works [1, 2], and this work mainly focused on the tabular setting. The objective of this work is to solve the optimization in (8), which tries to find out the constraint that maximizes the probability of trajectories that cannot be generated if that constraint is applied. (Such an objective minimizes the normalization constant in (5) and results in maximization of the demonstration likelihood under the constraint.) To solve this objective, the proposed algorithm first computes the feature occupancy (Algorithm 1), and then use those feature occupancy with greedy iterative constraint inference (Algorithm 2 that motivated by maximum coverage problem) to get constraints. Two experiments in the GridWorld show that the proposed method effectively works. 
I think this work is quite fundamental, impactful to be accepted at the conference and is possibly extended to practical scenarios (like explainable and safety RL and imitation learning) in the future. One thing I’d like to point out is to enhance the readability by reordering contents and adding some additional explanations to clarify their arguments. There are a few comments and questions I have:
- The exact definition and usage of nominal environments and rewards are still unclear to me. For example in Figure 2 (b), how did you define and get nominal MDP?
- Since Figure 1 is related to the second experiment, I recommend moving it to the experiment section. 
- At 3.2.1., “Because constraints are sets of state-action pairs, imposing a
constraint within an MDP means restricting the set of actions that can be taken from certain states.” -> Need clarification
- At 3.2.1., “For MDPs with deterministic transitions, it is clear that any agent respecting these constraints will not visit an empty state.” -> Why?
- At (8), $\{\}$ to $\emptyset$
- In Figure 3, I was a bit confused about the relationship between the threshold and the false positive rate at first glance. What I understood is that a small threshold leads to lots of iteration for constraint selection, which increases the false positive. I want authors to add some comments on that.
 Reference
[1] Chou, Bereson, Ozay, “Learning Constraints from Demonstrations,” arXiv 2019
[2] Chou, Bereson, Ozay, “Learning Parametric Constraints in High Dimensions from Demonstrations,” CoRL 2019