Understanding the Asymptotic Performance of Model-Based RL Methods
In complex simulated environments, model-based reinforcement learning methods typically lag the asymptotic performance of model-free approaches. This paper uses two MuJoCo environments to understand this gap through a series of ablation experiments designed to separate the contributions of the dynamics model and planner. These reveal the importance of long planning horizons, beyond those typically used. A dynamics model that directly predicts distant states, based on current state and a long sequence of actions, is introduced. This avoids the need for many recursions during long-range planning, and thus is able to yield more accurate state estimates. These accurate predictions allow us to uncover the relationship between model accuracy and performance, and translate to higher task reward that matches or exceeds current state-of-the-art model-free approaches.
 This paper proposes learning a transition model that takes an action sequence as an input (instead of a single action), and performing model-based planning by using the cross-entropy method. One obvious concern with this is that this produces a sequence of open-loop plans, rather than a closed-loop policies, with all the inherent limitations. I could see this working well in practice in problems where anticipating how future decisions will react to state changes is not that important, however the authors should discuss the trade-offs more. A larger concern for me revolves around learning the transition model. Taking the action sequence as an input (which is one of the main novelties in the paper) is likely to require a lot of data, and maybe this is fine on relatively simple Mujoco tasks but I see it as a potential issue when trying to expand this to more realistic problems. Finally, I suggest that the authors change the title to something more descriptive of the paper’s contents, as there is no analysis of asymptotic performance in the paper (as I would have thought from the title). I also recommend that they look to see if there is any model-based work in the semi-MDP literature, which could be relevant here.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Understanding the Asymptotic Performance of Model-Based RL Methods
In complex simulated environments, model-based reinforcement learning methods typically lag the asymptotic performance of model-free approaches. This paper uses two MuJoCo environments to understand this gap through a series of ablation experiments designed to separate the contributions of the dynamics model and planner. These reveal the importance of long planning horizons, beyond those typically used. A dynamics model that directly predicts distant states, based on current state and a long sequence of actions, is introduced. This avoids the need for many recursions during long-range planning, and thus is able to yield more accurate state estimates. These accurate predictions allow us to uncover the relationship between model accuracy and performance, and translate to higher task reward that matches or exceeds current state-of-the-art model-free approaches.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposes learning a transition model that takes an action sequence as an input (instead of a single action), and performing model-based planning by using the cross-entropy method. One obvious concern with this is that this produces a sequence of open-loop plans, rather than a closed-loop policies, with all the inherent limitations. I could see this working well in practice in problems where anticipating how future decisions will react to state changes is not that important, however the authors should discuss the trade-offs more. A larger concern for me revolves around learning the transition model. Taking the action sequence as an input (which is one of the main novelties in the paper) is likely to require a lot of data, and maybe this is fine on relatively simple Mujoco tasks but I see it as a potential issue when trying to expand this to more realistic problems. Finally, I suggest that the authors change the title to something more descriptive of the paper’s contents, as there is no analysis of asymptotic performance in the paper (as I would have thought from the title). I also recommend that they look to see if there is any model-based work in the semi-MDP literature, which could be relevant here.
assistant
 This paper proposes learning a transition model that takes an action sequence as an input (instead of a single action), and performing model-based planning by using the cross-entropy method. One obvious concern with this is that this produces a sequence of open-loop plans, rather than a closed-loop policies, with all the inherent limitations. I could see this working well in practice in problems where anticipating how future decisions will react to state changes is not that important, however the authors should discuss the trade-offs more. A larger concern for me revolves around learning the transition model. Taking the action sequence as an input (which is one of the main novelties in the paper) is likely to require a lot of data, and maybe this is fine on relatively simple Mujoco tasks but I see it as a potential issue when trying to expand this to more realistic problems. Finally, I suggest that the authors change the title to something more descriptive of the paper’s contents, as there is no analysis of asymptotic performance in the paper (as I would have thought from the title). I also recommend that they look to see if there is any model-based work in the semi-MDP literature, which could be relevant here.