We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies. Recent work has shown the key role played by the state or state-action stationary distribution corrections in the infinite horizon context for off-policy policy evaluation. We propose estimated mixture policy (EMP), a partially policy-agnostic methods to accurately estimate those quantities. With careful analysis, we show that EMP gives rise to estimates with reduced variance for estimating the state stationary distribution correction while it also offers a useful induction bias for estimating the state-action stationary distribution correction. In extensive experiments with both continuous and discrete environments, we demonstrate that our algorithm offers significantly improved accuracy compared to the state-of-the-art methods.
*Synopsis*:
  The main contribution of this paper is the development of estimated mixture policy (EMP), which takes ideas from the new off-policy policy evaluation infinite horizon estimators (i.e. Liu) and from a recent development in more traditional importance weight approaches using regression importance sampling (i.e. Hannah). This new method provides a nice extension of Liu's algorithm to many policies, and to when the policy is unknown. The paper provides some nice analysis inspired by Hannah. Finally, they provide empirical results.
  *Review*:
  While I think the method has potential interest to the community, I found the empirical results lacking (particularly in missing competitors). I also have some concerns about the theory as there seems to be many typos and consistency issues making much of it hard to follow. Overall, I think this paper is not quite up for publication, but if the authors address my consistency/missing proofs issues and provide a comparison to DualDice I will increase my score.
  1. I don't find the reason provided for not including DualDICE compelling and think it is an important competitor here, as there are many similarities between the two methods.
     - It would also be interesting to reproduce the results provided by Liu et al with the model based approach, and the on-policy oracle. I don't think these are as pressing as DualDice but still interesting.
  2. There are many consistency issues with respect to notation, and some odd notation choices as compared to the rest of the literature:
     - What is script \epsilon in the equation in 2.1? It looks like it should be an expectation over d_\pi, but I've never seen this notation before.
     - The indicies of sums and sets often change between one based and zero based indexing. This should be unified (preferably to one based). For example, section 2.1 \pi_j(j=0,1,2,...,m) and m=1 for one policy doesn't work. Either \pi_j(j=0,1,2,...m-1) or m=0. This occurs throughout the proofs in the appendix as well.
     - What is script \Epsilon_\theta? Do you mean script F_\theta? And then what does it mean for \theta_0 \in E_1? There seems to be many definitions missing.
  2.5. There are also some issues with the presentation of the theory over the consistency issues already mentioned.
     - The assumptions and conditions for the theorems presented in the main text should be clearly specified in the theory statement.
     - The proof to theorem 2 (i.e. in the appendix) should be provided.
     - The proof of proposition 4 seems to be missing as well.
  *Questions*
  Q1 What are the properties of SADL? Why was this used instead of DualDICE in the comparison?
  Q2 How would BCH do if we used the mixture policy as the behavior policy in the multiple behavior policy case? How would it compare to your method? This could be an interesting comparison, just to test if the lower MSE argument holds up in the multiple behavior policy case.
  Q3 What is the meaning of partially-policy-agnostic? It is unclear to me how it is different from the policy-agnostic approaches. If all that is different is you are estimating the behavior to use in the usual infinite-horizon approach, should this be in a separate category from policy-agnostic approaches? (I would say probably not, but I think you could make a case for it).
  Q4 "Then, a single (s,a,s') tuple simply follows the marginal distribution...". Is this trivially true?
  *Minor comments not taken into account in the review*
  - section 2.1 "target policy \pi via a pre-collected..." -> remove "a"
  - The layout of the related works section is a bit hard to follow.
  - "Recently, Nachum et. al. (2019) proposes DualDice": proposes->proposed
  - "by their estimated values in two folds": do you mean in two ways? This is unclear.
  - Section 3.1: "notation abusion" -> notation abuse
  - Equation right after equation 1: "d_\pi_0(s)\pi(a|s)" -> "d_\pi_0(s)\pi_0(a|s)"
  - "The derivation of kernel method are put in..." -> "The derivation of the kernel method is put in..."
  - "we need introduce more notation" -> "we need to introduce more notation"
  - Section 4: "detailed description on the data sample": "on"->"of"
  - "In this light by pooling the data together...." These two sentences should be put together.
  - I would like if your theorems were restated in the appendix, for ease of reading.
-----------
Post Rebuttal
I'd like to thank the author for their thorough response! Given the major additions to the paper including empirical comparisons and clarity for the theory I've decided to update my score to reflect my new feelings (i.e. to a 6). I think this paper is well worth accepting in its current form.