An important problem that arises in reinforcement learning and Monte Carlo methods is estimating quantities defined by the stationary distribution of a Markov chain. In many real-world applications, access to the underlying transition operator is limited to a fixed set of data that has already been collected, without additional interaction with the environment being available. We show that consistent estimation remains possible in this scenario, and that effective estimation can still be achieved in important applications. Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions, derived from fundamental properties of the stationary distribution, and exploiting constraint reformulations based on variational divergence minimization. The resulting algorithm, GenDICE, is straightforward and effective. We prove the consistency of the method under general conditions, provide a detailed error analysis, and demonstrate strong empirical performance on benchmark tasks, including off-line PageRank and off-policy policy evaluation.
In this paper the authors proposed a framework for off-policy value estimation under the scenario of infinite horizon RL tasks. The new proposed method utilize the variational representation of $f$-divergence, which quantifies the difference between $\mathcal{T}\tau$ and $\tau p$, where $\tau$ is the parametric density ratio between the unknown behavior policy data and the target policy. If only if $\tau$ is the true density ratio, the loss $\mathcal{D}_{f}(\mathcal{T}\tau || \tau p) = 0$. 
Compared with prior work (Nachum et. al 2019), the new proposed framework can generalize the undiscounted case $\gamma = 1$, and the derivation for the new algorithm is quite simple and easy to follow. The experimental results show the advantage of the proposed methods over baseline methods such as model-based, DualDice etc, for both discrete and continuous cases. Moreover, I have two specific questions:
- The choice of $f$-divergence. Although the author mentioned the difficulty of using the dual representation of KL divergence, it would be nice to have an ablation study that shows the effectiveness of various $f$-divergence (Personally I think Jensen-Shannon Divergence may be also a good choice).
-The authors should also have a discussion that similar idea can be generalized to more general  distribution metrics such as Integral Probability Metrics, specifically wasserstein-1 distance (similar to wasserstein-gan) or maximum mean discrepancy (Maybe it is unnecessary to conduct experiments, some discussion should be enough to clarify the relationship. I think there is a concurrent submission using MMD metrics).
 
Overall I think this is a good paper and I recommend for acceptance. 
Reference Papers:
- Nowozin, Sebastian, Botond Cseke, and Ryota Tomioka. "f-gan: Training generative neural samplers using variational divergence minimization." Advances in neural information processing systems. 2016.
- Arjovsky, Martin, Soumith Chintala, and Léon Bottou. "Wasserstein gan." arXiv preprint arXiv:1701.07875 (2017).
- Nachum, Ofir, et al. "DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections." arXiv preprint arXiv:1906.04733 (2019).
- Anonymous, “Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning”, submitted to ICLR 2020.