In complex simulated environments, model-based reinforcement learning methods typically lag the asymptotic performance of model-free approaches. This paper uses two MuJoCo environments to understand this gap through a series of ablation experiments designed to separate the contributions of the dynamics model and planner. These reveal the importance of long planning horizons, beyond those typically used. A dynamics model that directly predicts distant states, based on current state and a long sequence of actions, is introduced. This avoids the need for many recursions during long-range planning, and thus is able to yield more accurate state estimates. These accurate predictions allow us to uncover the relationship between model accuracy and performance, and translate to higher task reward that matches or exceeds current state-of-the-art model-free approaches.
The paper proposes to use a multi-step prediction model in model-based RL. The proposed model maps from current state and a sequence of actions to the state after taking those actions. The paper demonstrates on 2 tasks that in a model-predictive control loop combined with planning by cross-entropy method, this can yield better asymptotic performance than using single-step models.
The insight of using multi-step prediction models is certainly appealing and makes a lot of sense in deterministic tasks. A systematic empirical comparison of multi-step deep models in RL is of interest, which this paper does provide to some extent.
An obvious limitation of the proposed deterministic multi-step forward model is the restriction to deterministic systems. One would expect that the performance deteriorates quickly as the system becomes more stochastic. An extension to the stochastic case along the lines of Chua et al, 2018 is non-trivial as capturing the stochasticity is typically more challenging in long-term predictions. Yet, the paper makes an additional assumption that is less clearly communicated: To be able to plan with a R-step model, one needs to be able to evaluate or approximate the sum of R rewards just from the first and last state in that R-long sequence. This work uses simply the reward at the end r(s_{t+R}) as a proxy which works well in these MuJoCo tasks but can fail horribly in others. One can imagine that a model not only outputs s_{t+R} but also the sum of R rewards given s_t and a_{t:t+R} which could work in more general settings but this is not explored in this paper. The contribution in this paper limited as the proposed approach as well as the experimental comparison is restricted to a relatively specific class of problems and no attempts to generalize are made.
The experiments nicely compare against using single-step dynamics models and the results show that using the multi-step models for MPC performs better in the two considered tasks. However, as fas as I understand both the ACP and Chua et al baseline using the single-step prediction accuracy to train their models. The paper is missing a comparison to single-step models that are trained using multi-step prediction losses ("backprop through time" as in Learning Nonlinear Dynamic Models by Langford et al 2009). These models should be much more robust to error blow-up for multi-step prediction and do not require the specific reward structure assumed in this paper.
The proposed R-step model-based RL approach could be connected to the use of options (the planner and model operate on R-step options, but the MPC does update the policy after every time step). It would be interesting to discuss this potential connection in the paper. The paper does a good job of discussing existing recent work in the deep RL literature but it would also be good to also discuss earlier work on multi-step prediction (e.g. in time-series modeling).
All in all, I think the paper makes a small contribution demonstrating that multi-step models are useful for model-based RL in specific domains -- which is interesting but certainly not surprising. Unfortunately the paper stops somewhat early by not comparing to relevant baselines (single-step models trained with multi-step losses) and by not considering tasks where the benefit of multi-step planning would be less clear.