State-of-the-art results in imitation learning are currently held by adversarial methods that iteratively estimate the divergence between student and expert policies and then minimize this divergence to bring the imitation policy closer to expert behavior. Analogous techniques for imitation learning from observations alone (without expert action labels), however, have not enjoyed the same ubiquitous successes. Recent work in adversarial methods for generative models has shown that the measure used to judge the discrepancy between real and synthetic samples is an algorithmic design choice, and that different choices can result in significant differences in model performance. Choices including Wasserstein distance and various f -divergences have already been explored in the adversarial networks literature, while more recently the latter class has been investigated for imitation learning . Unfortunately, we find that in practice this existing imitation-learning framework for using f -divergences suffers from numerical instabilities stemming from the combination of function approximation and policygradient reinforcement learning. In this work, we alleviate these challenges and offer a reparameterization of adversarial imitation learning as f -divergence minimization before further extending the framework to handle the problem of imitation from observations only. Empirically, we demonstrate that our design choices for coupling imitation learning and f -divergences are critical to recovering successful imitation policies. Moreover, we find that with the appropriate choice of fdivergence, we can obtain imitation-from-observation algorithms that outperform baseline approaches and more closely match expert performance in continouscontrol tasks with low-dimensional observation spaces. With high-dimensional observations, we still observe a significant gap with and without action labels, offering an interesting avenue for future work.
* Summary:
The paper proposes an IL method based on the f-divergence. Specifically, the paper extends f-VIM (Ke et al., 2019), which uses the f-divergence for IL, by using a sigmoid function for discriminator output’s activation function. This choice of activation function yields an alternative objective function, where the reward function for an RL agent does not directly depend on the convex conjugate of the function f; the paper claims that this independency improves stability of policy learning. This proposed method is named f-VIM-sigmoid. The paper extends f-VIM-sigmoid to the setting of IL with observation and proposes f-VIMO-sigmoid. Experiments on Mujoco locomotion tasks show that f-VIM-sigmoid and f-VIMO-sigmoid perform better than existing methods.
* Rating:
The paper proposes a simple but interesting approach to improve stability of adversarial IL. However, the paper has issues regarding baseline methods, motivation, supports of the claim, and experiments (see below). These issues should be addressed. At the present, I vote for rejection.
* Major comments:
- Discussion and comparing against a simple baseline method based on swapping distributions: 
To make the reward function be independent of the convex conjugate f*, it is possible to simply swapping the distributions P and Q in the definition of the f-divergence. More specifically, instead of minimizing D_f(P||Q), we can minimize D_f(Q||P), where P is a data distribution and Q is a generator. In this case, pi* and pi_theta in Eq. (7) swap, and the RL agent minimizes the cost function g_f(V_w(s,a)). This cost function does not directly depend on f*, similarly to the reward function r(V_w(s,a)) in Eq. (8). This swapping is simpler and more flexible than re-parameterizing, while achieving the same goal as f-VIM-sigmoid. This swapping method should be discussed and compared against the proposed methods.
- Need stronger baseline methods for ILfO: 
The paper should evaluate f-VIMO-sigmoid against stronger baselines, e.g., forward adversarial IL (Sun et al., 2019) which outperforms GAIL-based methods in the ILfO setting. 
[1] Wen Sun, Anirudh Vemula, Byron Boots, and J Andrew Bagnell. Provably efficient imitation learning from observation alone. ICML, 2019.
- Using the f-divergence for ILfO is not well motivated: 
The paper does not provide good motivations for using f-divergence in ILfO. This makes the paper quite difficult to follow, since there is no connection between f-divergence and ILfO.
- The experiments focus on evaluating existing methods rather than the proposed methods: 
Specifically, the proposed methods are evaluated with only one choice of the divergence (TV) in Figure 2. Meanwhile, most of Section 6 and results (Figure 3 and 4, and additional results in the appendix) focus on evaluating the existing methods (f-VIM and f-VIMO) with different choices of divergence. 
- The experiments in Figure 2 do not support the claim regarding stability: 
The paper claims to improve stability of IL by using the proposed re-parameterization. However, the experimental results do not support this claim, and the questions asked in Section 5 are not related to this claimed. Instead, it seems that re-parameterization helps avoiding local optima (possibly due to a biased reward function, see below), while stability is improved by regularizing the discriminator. I could not see how the re-parameterization improves the policy stability as claimed. 
- The experiments in Figure 2 seem unfair, since TV-VIM-sigmoid incorporates priors about survival bonuses: 
Specifically, TV-VIM-sigmoid uses sigmoid which yields strictly positive rewards, while TV-VIM uses tanh which yields positive and negative rewards. As discussed by Kostrikov et al., 2019, using strictly positive rewards incorporate strong priors about the survival bonuses, which exist in the locomotion task used in the experiments. Therefore, TV-VIM-sigmoid uses strong priors while TV-VIM does not. In order to make the comparison fairer, I suggest the authors to evaluate TV-VIM with sigmoid reward output, or include environments that do not have survival bonuses.
[2] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. ICLR, 2019
* Minor comments: 
- The abstract is long and could be shorten. 
- Figures are too small and difficult to see, especially the legends.
- Table 1 should describe the form of f in addition to its conjugate.
- The title of the Algorithm 1 should be f-VIMO-sigmoid instead of f-VIMO. 
** Update after response.
I read the response. I thank the authors for clarifying the claims as well as the new experiments with the swap formulation. However, improving clarity of the claims is considered a major revision. I still keep the vote of rejection. 
Regarding reward bias. As the authors acknowledge, the improvement achieved by using reparameterization+sigmoid can be explained by two equally-plausible reasons: 1) reparameterization+sigmoid improves stability (as claimed) and 2) sigmoid gives biased rewards. The issue here is that we do not know which is the actual reason, given the current experiments in the paper. As I commented, evaluating TV-VIM with sigmoid but without reparameterization will help address this issue.