Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search
Learning policies on data synthesized by models can in principle quench the thirst of reinforcement learning algorithms for large amounts of real experience, which is often costly to acquire. However, simulating plausible experience de novo is a hard problem for many complex environments, often resulting in biases for model-based policy evaluation and search. Instead of de novo synthesis of data, here we assume logged, real experience and model alternative outcomes of this experience under counterfactual actions, i.e. actions that were not actually taken. Based on this, we propose the Counterfactually-Guided Policy Search (CF-GPS) algorithm for learning policies in POMDPs from off-policy experience. It leverages structural causal models for counterfactual evaluation of arbitrary policies on individual off-policy episodes. CF-GPS can improve on vanilla model-based RL algorithms by making use of available logged data to de-bias model predictions. In contrast to off-policy algorithms based on Importance Sampling which re-weight data, CF-GPS leverages a model to explicitly consider alternative outcomes, allowing the algorithm to make better use of experience data. We find empirically that these advantages translate into improved policy evaluation and search results on a non-trivial grid-world task. Finally, we show that CF-GPS generalizes the previously proposed Guided Policy Search and that reparameterization-based algorithms such Stochastic Value Gradient can be interpreted as counterfactual methods.
 Summary: This paper proposes a policy evaluation and search method assisted by a counterfactual model, in contrast previous work using vanilla (non-causal) models. With “no model mismatch” assumption the policy evaluation estimator is unbiased. Empirically, the paper compares Guided Policy Search with counterfactual model (CF-GPS) with vanilla GPS, model based RL algorithm and show benefit in terms of (empirical) sample complexity. Main comments: This paper studies several interesting problems: 1) policy learning with off-policy data; 2) model based RL and how to use model to help policy learning. By capturing a nice connection between causal models and MDP/POMDP model with off-policy data, this paper can leverage SCMs to help the model guided policy search in POMDP. The combination of those ideas is novel and enjoyable. On the negative side, I find I met several confused points as a reader with more RL background and less causal inference background. It would be better if the authors could clarify what is the prior distribution P(u) and posterior distribution P(u|h) exactly means in terms of CF-PE algorithm and MB-PE algorithm. I would also appreciate if a more detailed proof of corollary 1 and 2 are included in the appendix, and a higher level intuition/justification about those two results in main body. Maybe I am missing these points due to my limited background in causal inference, but I think those clarification can definitely be helpful for RL audience without that much knowledge in causal inference. The main theoretical result seems to be based on the assumption of no model mismatch, and I guess here how the model is estimated from sample are ignored, unless I missed anything. Thus I assume the main contribution of this paper should be algorithmic and empirical. I expect to see the empirical study in more domains with more informative results about how this CF model get the benefit of sampling from p(u|h) rather than p(u) (as an evidence to support motivation paragraph on page 5).
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search
Learning policies on data synthesized by models can in principle quench the thirst of reinforcement learning algorithms for large amounts of real experience, which is often costly to acquire. However, simulating plausible experience de novo is a hard problem for many complex environments, often resulting in biases for model-based policy evaluation and search. Instead of de novo synthesis of data, here we assume logged, real experience and model alternative outcomes of this experience under counterfactual actions, i.e. actions that were not actually taken. Based on this, we propose the Counterfactually-Guided Policy Search (CF-GPS) algorithm for learning policies in POMDPs from off-policy experience. It leverages structural causal models for counterfactual evaluation of arbitrary policies on individual off-policy episodes. CF-GPS can improve on vanilla model-based RL algorithms by making use of available logged data to de-bias model predictions. In contrast to off-policy algorithms based on Importance Sampling which re-weight data, CF-GPS leverages a model to explicitly consider alternative outcomes, allowing the algorithm to make better use of experience data. We find empirically that these advantages translate into improved policy evaluation and search results on a non-trivial grid-world task. Finally, we show that CF-GPS generalizes the previously proposed Guided Policy Search and that reparameterization-based algorithms such Stochastic Value Gradient can be interpreted as counterfactual methods.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Summary: This paper proposes a policy evaluation and search method assisted by a counterfactual model, in contrast previous work using vanilla (non-causal) models. With “no model mismatch” assumption the policy evaluation estimator is unbiased. Empirically, the paper compares Guided Policy Search with counterfactual model (CF-GPS) with vanilla GPS, model based RL algorithm and show benefit in terms of (empirical) sample complexity. Main comments: This paper studies several interesting problems: 1) policy learning with off-policy data; 2) model based RL and how to use model to help policy learning. By capturing a nice connection between causal models and MDP/POMDP model with off-policy data, this paper can leverage SCMs to help the model guided policy search in POMDP. The combination of those ideas is novel and enjoyable. On the negative side, I find I met several confused points as a reader with more RL background and less causal inference background. It would be better if the authors could clarify what is the prior distribution P(u) and posterior distribution P(u|h) exactly means in terms of CF-PE algorithm and MB-PE algorithm. I would also appreciate if a more detailed proof of corollary 1 and 2 are included in the appendix, and a higher level intuition/justification about those two results in main body. Maybe I am missing these points due to my limited background in causal inference, but I think those clarification can definitely be helpful for RL audience without that much knowledge in causal inference. The main theoretical result seems to be based on the assumption of no model mismatch, and I guess here how the model is estimated from sample are ignored, unless I missed anything. Thus I assume the main contribution of this paper should be algorithmic and empirical. I expect to see the empirical study in more domains with more informative results about how this CF model get the benefit of sampling from p(u|h) rather than p(u) (as an evidence to support motivation paragraph on page 5).
assistant
 Summary: This paper proposes a policy evaluation and search method assisted by a counterfactual model, in contrast previous work using vanilla (non-causal) models. With “no model mismatch” assumption the policy evaluation estimator is unbiased. Empirically, the paper compares Guided Policy Search with counterfactual model (CF-GPS) with vanilla GPS, model based RL algorithm and show benefit in terms of (empirical) sample complexity. Main comments: This paper studies several interesting problems: 1) policy learning with off-policy data; 2) model based RL and how to use model to help policy learning. By capturing a nice connection between causal models and MDP/POMDP model with off-policy data, this paper can leverage SCMs to help the model guided policy search in POMDP. The combination of those ideas is novel and enjoyable. On the negative side, I find I met several confused points as a reader with more RL background and less causal inference background. It would be better if the authors could clarify what is the prior distribution P(u) and posterior distribution P(u|h) exactly means in terms of CF-PE algorithm and MB-PE algorithm. I would also appreciate if a more detailed proof of corollary 1 and 2 are included in the appendix, and a higher level intuition/justification about those two results in main body. Maybe I am missing these points due to my limited background in causal inference, but I think those clarification can definitely be helpful for RL audience without that much knowledge in causal inference. The main theoretical result seems to be based on the assumption of no model mismatch, and I guess here how the model is estimated from sample are ignored, unless I missed anything. Thus I assume the main contribution of this paper should be algorithmic and empirical. I expect to see the empirical study in more domains with more informative results about how this CF model get the benefit of sampling from p(u|h) rather than p(u) (as an evidence to support motivation paragraph on page 5).