We propose Support-guided Adversarial Imitation Learning (SAIL), a generic imitation learning framework that unifies support estimation of the expert policy with the family of Adversarial Imitation Learning (AIL) algorithms. SAIL addresses two important challenges of AIL, including the implicit reward bias and potential training instability. We also show that SAIL is at least as efficient as standard AIL.
In an extensive evaluation, we demonstrate that the proposed method mitigates the reward bias and achieves better performance and training stability than other baseline methods on a wide range of benchmark control tasks.
**Summary of the paper: 
The paper proposes an IL method named support-guided adversarial IL (SAIL), which is based on generative adversarial IL (GAIL) (Ho and Ermon, 2016) and random expert distillation (RED) (Wang et al., 2019). The key idea of SAIL is to construct a reward function by multiplying reward functions learned by GAIL and RED. This multiplication yields two benefits; 1) it handles the issue of biased reward in GAIL, since state-action pairs outside the expert’s support are assigned low reward values. 2) SAIL’s reward is more reliable than RED’s reward for state-action pairs inside the expert’s support. The authors show that SAIL is at least as fast as than GAIL in terms of the sample complexity. Experiments on continuous control benchmarks show that SAIL is overall more stable than GAIL. 
**Rating: 
The paper proposes a simple but effective combination of existing methods. The proposed method is well motivated and performs well on benchmarks. Still, the paper has some issues regarding justification, clarity, and evaluation, which should be addressed (see below). I vote for weak acceptance. 
**Major comments/questions: 
- No guarantee of the optimality of the learned policy.
Can it be guaranteed that SAIL learns the expert policy? (assuming the expert policy is realizable). Propositions 1 and 2 show the convergence of the support estimation, but these results are not related to the optimality of a policy learned with the reward function. This is an important point for justifying SAIL, since SAIL does not perform distribution matching to learn the expert policy, and it also does not perform IRL to learn the reward function. Therefore, SAIL lacks the optimality guarantee from both distribution matching and IRL perspectives. Please address and clarify this point. 
- Clarity in the theoretical analysis.
In the theoretical analysis, the paper assumes a rate of GAIL for support estimation. This is quite confusing, since GAIL performs distribution matching and does not estimate the support. Also, given that r_gail = -log D(s,a), the reward’s upper-bound (R_gail) is infinity and the bound in Eq. (9) is not informative. 
- The reward r_red is constant at the optimal.
Eq. (2) and Eq. (3) imply that, for state-action pairs from the expert’s state-action distribution, r_red is constant at the optimal. Specifically, the optimal solution of Eq. (2) is \hat{\theta} = \theta, which yields to a constant value of r_red(s,a) in Eq. (3). In this scenario, SAIL is equivalent to GAIL for the expert state-action distribution. This means that Eq. (2) should not be optimized until optimal, and some early stopping criteria are required. Does this scenario (constant value of r_red) occur in the experiments?
- IRL baseline methods.
The paper should compare SAIL to methods which aim to handle the bias in reward function, e.g., DAC (Kostrikov et al. 2019). While DAC requires the time limit, this time limit is known in the benchmark tasks. Also, IRL methods such as AIRL (Fu et al., 2018) should be compared, since IRL methods are better than GAIL at handling bias in reward function (Kostrikov et al. 2019). 
**Minor comments/questions: 
- Typos: "offline RL algorithms" should be "off-policy RL algorithms". Line 5 of Algorithm 1 should perform gradient ascent instead of gradient descent. An expectation over state-action distribution of expert is missing from Eq. (2). 
- What are the bold numbers in table 1 and 2 indicating? Why does the Hopper task have two bold numbers, but the other tasks have only one?
--After author response--
I have read the author response and other reviews. I thank the authors for including additional experiments. However, the authors' arguments regarding optimality do not fully address my comments (see below). I will keep the vote of weak acceptance.  
The authors argue that "In this asymptotic case, SAIL is equivalent to performing distribution matching via GAIL with the additional *constraint* that candidate distributions need to have the same support of the expert distribution". However, the support of the expert distribution may coincide with the entire state-action space, which makes the additional constraint uninformative in the asymptotic case. Specifically, the expert distribution coincides with the state-action space when the expert policy has an infinite support (e.g., the expert policy is Gaussian). Assuming the asymptotic case, the support estimation in RED will give an indicator function over an entire state-action space, and the support constraint in SAIL is always satisfied. In other words, SAIL is exactly equivalent to GAIL in this case. For these reasons, the authors' arguments regarding optimality do not fully address my comments. I think additional assumptions are required, e.g., the expert policy needs to have a finite support or be deterministic.