WOULDA, COULDA, SHOULDA: COUNTERFACTUALLY-GUIDED POLICY SEARCH
Learning policies on data synthesized by models can in principle quench the thirst of reinforcement learning algorithms for large amounts of real experience, which is often costly to acquire. However, simulating plausible experience de novo is a hard problem for many complex environments, often resulting in biases for modelbased policy evaluation and search. Instead of de novo synthesis of data, here we assume logged, real experience and model alternative outcomes of this experience under counterfactual actions, i.e. actions that were not actually taken. Based on this, we propose the Counterfactually-Guided Policy Search (CF-GPS) algorithm for learning policies in POMDPs from off-policy experience. It leverages structural causal models for counterfactual evaluation of arbitrary policies on individual off-policy episodes. CF-GPS can improve on vanilla model-based RL algorithms by making use of available logged data to de-bias model predictions. In contrast to off-policy algorithms based on Importance Sampling which re-weight data, CF-GPS leverages a model to explicitly consider alternative outcomes, allowing the algorithm to make better use of experience data. We find empirically that these advantages translate into improved policy evaluation and search results on a non-trivial grid-world task. Finally, we show that CF-GPS generalizes the previously proposed Guided Policy Search and that reparameterization-based algorithms such Stochastic Value Gradient can be interpreted as counterfactual methods.
Summary: by assuming a correct, strongly factored environment model, improved estimators useful for policy search can be derived by "counterfactual reasoning", where data sampled from experience is used to refine initial conditions in the model; this translates into improved estimators of policy values, which improves policy search.
Major comments:
I enjoyed this paper.  I think that model-based RL deserves more work, and I think that this is a simple, reasonably workable approach with some nice theoretical benefits.  I like the idea of SCMs; I like the idea of counterfactual reasoning; I like the idea of leveraging models in this unique way.
On the negative side, I felt that the paper makes some rather strong assumptions - specifically, that the agent has access to a perfect model with no mismatch, and that the model decomposes neatly into noise variables plus deterministic functions.  Given such a model, one wonders if there are other techniques, say, from classical planning, that could also be used for some sort of policy search.
I have a few questions about approximations.  First, I see that probabilistic inference is a core element of each algorithm (where p(u|h) must be computed).  For large, complex models, I assume this must be approximate inference.  This leads naturally to questions about accuracy (does approximate inference result in biased estimators? [probably yes]), efficacy (do the inaccuracies inherent in approximate inference outweigh the benefits of using p(u|h) vs. p(u)?) and scalability (how large of a model can we reasonably cope with before degradation is unacceptable, or no better than non-CF algorithms?).  As far as I can tell, none of this was addressed in the paper, although I do not expect every paper to answer every question; this is a first step.
I wish the experiments were a little more varied.  The experimental results really only show marginal improvement in one small task.  While I understand that this is not an empirical paper, neither does it fit strongly into the category of "theory paper".  For example, there are no theory results indicating what sort of benefit we might expect from using the methods outlined here, and in the absence of such theory, we might reasonably look to various experiments to demonstrate its effectiveness.
Pros:
+ Integration with SCMs is interesting
+ Counterfactual variants of algorithms are clearly motivated and interesting
+ Paper is generally well-written
Cons:
- Assumption that the agent is given a model with no mismatch is very strong
- Model class (noise variables + deterministic functions) seems potentially restrictive
- Questions about impact of approximate inference
- Experiments could have been more varied