Learning powerful policies and better dynamics models by encouraging consistency
Model-based reinforcement learning approaches have the promise of being sample efficient. Much of the progress in learning dynamics models in RL has been made by learning models via supervised learning. There is enough evidence that humans build a model of the environment, not only by observing the environment but also by interacting with the environment. Interaction with the environment allows humans to carry out "experiments": taking actions that help uncover true causal relationships which can be used for building better dynamics models. Analogously, we would expect such interaction to be helpful for a learning agent while learning to model the environment dynamics. In this paper, we build upon this intuition, by using an auxiliary cost function to ensure consistency between what the agent observes (by acting in the real world) and what it imagines (by acting in the ``learned** world). Our empirical analysis shows that the proposed approach helps to train powerful policies as well as better dynamics models.
 Summary: This paper presents a simple auxiliary loss term for model-based RL that attempts to enforce consistency between observed experience trajectories and hallucinated rollouts. Simple experiments demonstrate that the constraint slightly improves performance. Quality: While I think the idea of a consistency constraint is probably reasonable, I consider this a poorly executed exploration of the idea. The paper makes no serious effort to compare and contrast this idea with other efforts at model-based RL. The most glaring omission is comparison to very old ideas (such as dyna) and new ideas (such as imagination agents), both of which they cite. Clarity: The paper is reasonably clear, although there are some holes. For example, in the experimental section, it is unclear what model-based RL algorithm is being used, and how it was modified to support the consistency constraint. (I did not read the appendix). Originality: It is not clear how novel the central idea is. Significance: This idea is not significant. Pros: + A simple, straightforward idea + A good topic - progress in model-based RL is always welcome Cons: - Unclear how this is significantly different from other related work (such as imagination agents) - Experimental setup is poorly executed. - Statistical significance of improvements is unclear - No attempt to relate to any other method in the field - No explanation of what algorithms are being used
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning powerful policies and better dynamics models by encouraging consistency
Model-based reinforcement learning approaches have the promise of being sample efficient. Much of the progress in learning dynamics models in RL has been made by learning models via supervised learning. There is enough evidence that humans build a model of the environment, not only by observing the environment but also by interacting with the environment. Interaction with the environment allows humans to carry out "experiments": taking actions that help uncover true causal relationships which can be used for building better dynamics models. Analogously, we would expect such interaction to be helpful for a learning agent while learning to model the environment dynamics. In this paper, we build upon this intuition, by using an auxiliary cost function to ensure consistency between what the agent observes (by acting in the real world) and what it imagines (by acting in the ``learned** world). Our empirical analysis shows that the proposed approach helps to train powerful policies as well as better dynamics models.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Summary: This paper presents a simple auxiliary loss term for model-based RL that attempts to enforce consistency between observed experience trajectories and hallucinated rollouts. Simple experiments demonstrate that the constraint slightly improves performance. Quality: While I think the idea of a consistency constraint is probably reasonable, I consider this a poorly executed exploration of the idea. The paper makes no serious effort to compare and contrast this idea with other efforts at model-based RL. The most glaring omission is comparison to very old ideas (such as dyna) and new ideas (such as imagination agents), both of which they cite. Clarity: The paper is reasonably clear, although there are some holes. For example, in the experimental section, it is unclear what model-based RL algorithm is being used, and how it was modified to support the consistency constraint. (I did not read the appendix). Originality: It is not clear how novel the central idea is. Significance: This idea is not significant. Pros: + A simple, straightforward idea + A good topic - progress in model-based RL is always welcome Cons: - Unclear how this is significantly different from other related work (such as imagination agents) - Experimental setup is poorly executed. - Statistical significance of improvements is unclear - No attempt to relate to any other method in the field - No explanation of what algorithms are being used
assistant
 Summary: This paper presents a simple auxiliary loss term for model-based RL that attempts to enforce consistency between observed experience trajectories and hallucinated rollouts. Simple experiments demonstrate that the constraint slightly improves performance. Quality: While I think the idea of a consistency constraint is probably reasonable, I consider this a poorly executed exploration of the idea. The paper makes no serious effort to compare and contrast this idea with other efforts at model-based RL. The most glaring omission is comparison to very old ideas (such as dyna) and new ideas (such as imagination agents), both of which they cite. Clarity: The paper is reasonably clear, although there are some holes. For example, in the experimental section, it is unclear what model-based RL algorithm is being used, and how it was modified to support the consistency constraint. (I did not read the appendix). Originality: It is not clear how novel the central idea is. Significance: This idea is not significant. Pros: + A simple, straightforward idea + A good topic - progress in model-based RL is always welcome Cons: - Unclear how this is significantly different from other related work (such as imagination agents) - Experimental setup is poorly executed. - Statistical significance of improvements is unclear - No attempt to relate to any other method in the field - No explanation of what algorithms are being used