Infinite horizon off-policy policy evaluation is a highly challenging task due to the excessively large variance of typical importance sampling (IS) estimators. Recently, Liu et al. (2018a) proposed an approach that substantially reduces the variance of infinite-horizon off-policy evaluation by estimating the stationary density ratio, but at the cost of introducing potentially high biases due to the error in density ratio estimation. In this paper, we develop a bias-reduced augmentation of their method, which can take advantage of a learned value function to improve accuracy. Our method is doubly robust in that the bias vanishes when either the density ratio or value function estimation is perfect. In general, when either of them is accurate, the bias can also be reduced. Both theoretical and empirical results show that our method yields significant advantages over previous methods.
Comments : 
This paper provides an approach for reducing bias in long horizon off-policy evaluation (OPE) problems, extending recent work from Liu et al., 2018 that estimates the ratio of the stationary state distributions in off-policy evaluation for reducing variance. The paper provides a doubly robust method for reducing bias, since it requires separately estimating a value function. The key idea of the paper is to provide low variance, low bias OPE since their approach relies on accurately estimating the state distribution ratio and also the estimation of the value function. 
The paper provides a follow up on Liu et al., 2018 and other recent works in off-policy learning that tries to estimate the state distribution ratio directly, but can introduce high inaccuracy if the obtained ratio estimates are inaccurate. In line of that, this approach seems useful as it tries to reduce the error from inaccurate ratio estimates by incorporating prior works with an additional value function estimation. 
Furthermore, the paper introduces a new perspective to doubly robust estimation that tries to reduce bias, instead of variance in previously known OPE literature (Jiang et al., 2016; Thomas et al., 2016). It is interesting to see how doubly robust can be related to primal-dual relations and the connections between these approaches, which is a novel contribution to the best of my knowledge. 
- The fundamental connection comes from observing OPE withdoubly robust estimator that estimates \hat{V} and equation 6 that incorporates the ratio of the state density. This is clearly written in equations 7 and 8, that are two ways of evaluating the value function with off-policy samples known in the literature.
- It uses the property of the Bellman recursive expression for the estimated value function of doubly robust OPE estimator and uses it in the OPE with state density ratio, leading to equation 9, and obtaining the bridge estimator that now relies on both estimates of value function \hat{V} and the state density ratio. Although initially this does not seem useful, but the authors clarify this and how a bias reducting method can be achieved as in equation 11. 
- It is a nice and elegant trick, exploiting the connections between OPE estimators leading to a bias reduction method that seems quite interesting. 
- The most interesting part of the paper comes from section 4 that exploits the connection between doubly robust approaches with Lagrangian duality, and that their approach is equivalent to a primal dual formulation of policy evaluation. This stands out in itself as a novel contribution of the paper. Equations 14 and 15 best writes down the connections, as to how policy evaluation can be formualted as a Lagrangian function for optimization. 
- Although the authors point out how equation 15 is equivalent to equation 11 - this does not seem straight forward at first unless carefully looked at. I would encourate the authors to perhaps add more clarity that exploits this connection, to make the paper more self-contained. 
- In terms of experiments, the paper compares the doubly robust approach with previous works that estimates the density ratio, along with other baseline comparisons such as weighted DR. In both the two evaluated problems, their approach seems useful in terms of obtaining better accuracy (lower MSE). 
- However, I am not sure to what extent this approach can be scaled to more complicated tasks for OPE? Are there any example domains where the proposed method fails, or is difficult to scale up to more complicated tasks? 
- The current set of experiment results seems adequate, given the theoretical contribution and that most OPE papers evaluate on such domains. But overall, it might be useful to analyse the significance of this approach when scaling to more complicated domains. 
Overall, I think the paper has a neat and elegant theoretical contribution, exploiting the connection of OPE with primal-dual frameworks that seems quite novel to me. Experimental set of results are properly presented too showing significance of the approach compared to previously known baselines. I think such papers exploting connections with other literature is useful for the community in general, and this paper has significantly novel theoretical contribution. Hence, I would recommend for acceptance of this paper.