Representation Balancing MDPs for Off-Policy Policy Evaluation
We study the problem of off-policy policy evaluation (OPPE) in RL. In contrast to prior work, we consider how to estimate both the individual policy value and average policy value accurately. We draw inspiration from recent work in causal reasoning, and propose a new finite sample generalization error bound for value estimates from MDP models. Using this upper bound as an objective, we develop a learning algorithm of an MDP model with a balanced representation, and show that our approach can yield substantially lower MSE in common synthetic benchmarks and a HIV treatment simulation domain.
The paper studies how to fit a model for the purpose of off-policy policy evaluation (OPPE), that is, estimating the value function of a policy using data from a different policy. Instead of using a simple loss function under the data distribution, the proposed method addresses the distribution shift issue by incorporating data distribution under target policy, approximated by importance sampling. Doing so alone, however, results in data sparsity, so the ultimate objective also includes the error of the behavior policy to stabilize and regularize learning. Finite sample generalization error bounds are proved for the proposed approaches, and empirical evaluation compares the approaches to other OPE methods for value function and expected return estimations.   I like the research question of how to estimate a model for OPE that takes into account the distribution shift issue. The ideas in the paper make a lot of intuitive sense. The paper is well written and pleasant to read, and the experiments show some promising results.  Now let me turn to weaknesses. The first major weakness is that, the theory part does not really support the proposed method well (despite the amount of maths involved). One thing I want to highlight here is that, as far as I understand, the introduction of MSE_mu in the objective is not justified at all by the theorems. Namely, if we get rid of MSE_mu we should be able to obtain *better* upper bounds; see Bullet 4. Neither is it justified empirically.  It should be noted that some of the theoretical caveats are addressed by the positive empirical results. That said, the experiment setup on the Sepsis dataset, where the most promising results are shown, is very questionable; see bullet 6 below. Put it bluntly, I would be more comfortable accepting the paper if this section is simply dropped (unless the methodology can be rigorously justified).   Overall this is a borderline paper to me, and I temporarily lean very slightly towards rejection. That said, I really like the research topic and find the ideas of the paper quite nice, so I am ready to be convinced by the authors and the other reviewers (esp. on the Sepsis data issue) and change my mind.  Detailed comments:  1. Why is E_{s0}[(...)^2] the objective to consider at all? In particular, why do we only consider the initial state distribution of s0, and ignore the later time steps? Details follow:  (a) s0 can be deterministic w.l.o.g (just add a fake initial state that transitions to the true initial distribution), in which case this objective is no different from the average value objective (APV, line 43). So for the objective to make a difference, we are at the mercy of the environment---that its initial state distribution is well spread somehow. When is this the case?  (b) Which objective is right really depends on what you want to do with the estimated value function. In the experiment section, the best use case of the proposed method is still average value estimation. If this is the ultimate use case, *for the purpose of theoretical development*, you should start with the average value objective, and then explain why using the proposed objective as a *surrogate* actually leads to better average value estimations.   (c) Now there is definitely more we can do with a value function other than turning it into a scalar return---for example, policy improvement. But if policy improvement is the goal, we need to consider distributions of states beyond the initial step (see e.g., the CPI paper by Kakade and Langford).   2. Eq.4 is said to be an upper bound on the OPPE error. Isn't OPPE error simply the first term? Calling Eq.4 an upper bound is mathematically correct but perhaps conceptually misleading: for example, we say that hinge-loss is an upper bound on 0-1 loss, but seldom say that the L-2 regularized objective is an upper bound on the empirical risk (note it could upper bound population risk as in SRM and that's legit, but that's not what's going on in this paper). Overall I feel that the introduction of the MSE_mu is more heuristic than it sounds to be.  The looseness of Eq.4 also factors into Thm 2. (The first step in the equation between Line 225 and 226 performs this relaxation.) This brings to my next point...  3. One thing that is missing is the comparison to baseline approaches in theory. What guarantees can we give for simply fitting the model on data distribution? Is the guarantee in the current paper better than that of the baseline approach?   4. As far as I understand, the current analyses do not justify the introduction of MSE_mu, since Thm 2 relaxes MSE_pi to MSE_pi + MSE_mu and bounds them separately. If we just drop MSE_mu, we should get better upper bounds? Please comment on this in the rebuttal.  5. The empirical evaluation is thorough and shows promising results. However, given the caveat in bullet 3, I think it's very important to perform an ablation study and see how the method performs without the MSE_mu part.  6. For the Sepsis data, I am not sure why the artificial split of data is valid. For example, if the actions made during data collection depend on the patient's characteristics, such a split of data could give us two subpopulations that are very different in distribution, breaking the basic assumptions of off-policy evaluation. Unfortunately, I cannot find any description of the behavior policy in the paper.   More generally speaking, what this section does is to "mimic" an A/B test using observational data. There is extensive literature related to this from Stats/Econ (e.g., instrumental variables), and this usually needs to be done with extreme care. While I am not an expert on this, I feel that the paper's treatment here is surprisingly casual. Also note that it wouldn't be much of a problem if someone does this in the algorithm: if an algorithm does something wrong it will perform poorly, and the poor performance will be observed correctly. The problem is more serious here because the methodology is used to form the groundtruth, hence any mistake would affect the validity of the conclusions we draw from the results.    Minor - "Individual" vs "average" policy value: the terminologies here are somewhat unconventional. Perhaps consider "value function" vs "expected return" of a policy.  - References need a very careful pass: [5] and [9] should use conference versions instead of arXiv, author name in [8], [20] has "\phi" in it, etc. - Line 197, space before the comma.  ================================== After rebuttal: Thanks for the rebuttal. I am happy to learn that the authors agree with quite a few of my concerns and have proposed concrete revisions to address them. Despite that the revisions seem quite substantial, I am reasonably confident that the authors can implement them faithfully for camera-ready, so I am raising my score accordingly. Re the initial state issue, the authors' argument is in line with my 1(b) in review, so you might want to consider the alternative presentation I mentioned there.