Causally Correct Partial Models for Reinforcement Learning
In reinforcement learning, we can learn a model of future observations and rewards, and use it to plan the agent*s next actions. However, jointly modeling future observations can be computationally expensive or even intractable if the observations are high-dimensional (e.g. images). For this reason, previous works have considered partial models, which model only part of the observation. In this paper, we show that partial models can be causally incorrect: they are confounded by the observations they don*t model, and can therefore lead to incorrect planning. To address this, we introduce a general family of partial models that are provably causally correct, but avoid the need to fully model future observations.
*Summary* This paper considers the effect of partial models in RL, authors claim that these models can be causally wrong and hence result in a wrong policy (sub optimal set of actions). Authors demonstrate this issue with a simple MDP model, and emphasize the importance of behavior policy and data generation process. Then authors suggest a simple solution using backdoors (Pear et al) to learn causally correct models.They also conduct experiments to support their claims. *Decision* I vote for rejection of this paper, based on the following argument: To my understanding authors are basically solving the “off-policy policy evaluation” problem, without relating to this literature. For example, the MDP example is just an off-policy policy evaluation problem, and it is very well known that in this case you need to consider the behavior policy, for example with importance sampling. Even authors definition of the problem at the end of the page 4, and beginning of page 5, is the problem of “off-policy policy evaluation” when y_t = r_t Authors have not cited any paper in this literature, and did not situate their work with respect to this literature. To my understanding, the proposed solution is basically importance sampling, that is very well known and studied in the field. Additionally, I suggest that authors be more careful with their citations, for example, authors cited Silver et al 2017 [The Predictron: End-To-End Learning and Planning], as one recent paper using the method; however Silver et al 2017 is in MRP setting (Markov Reward process) where there is no action, so the described problem setting doesn*t apply. Improvement The current manuscript needs a major revision, mainly 1) situate the work with respect to off-policy policy evaluation literature, and then 2) Considering step 1, a clarification for what is the novelty/ contribution of the current paper is needed.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Causally Correct Partial Models for Reinforcement Learning
In reinforcement learning, we can learn a model of future observations and rewards, and use it to plan the agent*s next actions. However, jointly modeling future observations can be computationally expensive or even intractable if the observations are high-dimensional (e.g. images). For this reason, previous works have considered partial models, which model only part of the observation. In this paper, we show that partial models can be causally incorrect: they are confounded by the observations they don*t model, and can therefore lead to incorrect planning. To address this, we introduce a general family of partial models that are provably causally correct, but avoid the need to fully model future observations.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



*Summary* This paper considers the effect of partial models in RL, authors claim that these models can be causally wrong and hence result in a wrong policy (sub optimal set of actions). Authors demonstrate this issue with a simple MDP model, and emphasize the importance of behavior policy and data generation process. Then authors suggest a simple solution using backdoors (Pear et al) to learn causally correct models.They also conduct experiments to support their claims. *Decision* I vote for rejection of this paper, based on the following argument: To my understanding authors are basically solving the “off-policy policy evaluation” problem, without relating to this literature. For example, the MDP example is just an off-policy policy evaluation problem, and it is very well known that in this case you need to consider the behavior policy, for example with importance sampling. Even authors definition of the problem at the end of the page 4, and beginning of page 5, is the problem of “off-policy policy evaluation” when y_t = r_t Authors have not cited any paper in this literature, and did not situate their work with respect to this literature. To my understanding, the proposed solution is basically importance sampling, that is very well known and studied in the field. Additionally, I suggest that authors be more careful with their citations, for example, authors cited Silver et al 2017 [The Predictron: End-To-End Learning and Planning], as one recent paper using the method; however Silver et al 2017 is in MRP setting (Markov Reward process) where there is no action, so the described problem setting doesn*t apply. Improvement The current manuscript needs a major revision, mainly 1) situate the work with respect to off-policy policy evaluation literature, and then 2) Considering step 1, a clarification for what is the novelty/ contribution of the current paper is needed.
assistant
*Summary* This paper considers the effect of partial models in RL, authors claim that these models can be causally wrong and hence result in a wrong policy (sub optimal set of actions). Authors demonstrate this issue with a simple MDP model, and emphasize the importance of behavior policy and data generation process. Then authors suggest a simple solution using backdoors (Pear et al) to learn causally correct models.They also conduct experiments to support their claims. *Decision* I vote for rejection of this paper, based on the following argument: To my understanding authors are basically solving the “off-policy policy evaluation” problem, without relating to this literature. For example, the MDP example is just an off-policy policy evaluation problem, and it is very well known that in this case you need to consider the behavior policy, for example with importance sampling. Even authors definition of the problem at the end of the page 4, and beginning of page 5, is the problem of “off-policy policy evaluation” when y_t = r_t Authors have not cited any paper in this literature, and did not situate their work with respect to this literature. To my understanding, the proposed solution is basically importance sampling, that is very well known and studied in the field. Additionally, I suggest that authors be more careful with their citations, for example, authors cited Silver et al 2017 [The Predictron: End-To-End Learning and Planning], as one recent paper using the method; however Silver et al 2017 is in MRP setting (Markov Reward process) where there is no action, so the described problem setting doesn*t apply. Improvement The current manuscript needs a major revision, mainly 1) situate the work with respect to off-policy policy evaluation literature, and then 2) Considering step 1, a clarification for what is the novelty/ contribution of the current paper is needed.