Despite its potential to improve sample complexity versus model-free approaches, model-based reinforcement learning can fail catastrophically if the model is inaccurate. An algorithm should ideally be able to trust an imperfect model over a reasonably long planning horizon, and only rely on model-free updates when the model errors get infeasibly large. In this paper, we investigate techniques for choosing the planning horizon on a state-dependent basis, where a state's planning horizon is determined by the maximum cumulative model error around that state. We demonstrate that these state-dependent model errors can be learned with Temporal Difference methods, based on a novel approach of temporally decomposing the cumulative model errors. Experimental results show that the proposed method can successfully adapt the planning horizon to account for state-dependent model accuracy, significantly improving the efficiency of policy learning compared to model-based and model-free baselines.
Learning to Combat Compounding-Error in Model-Based Reinforcement Learning
The authors study the problem of choosing the appropriate planning horizon in model-based reinforcement learning. We want to have a balance between the efficiency of planning into the future and while avoiding the compounding error brought by the learnt model.
And the authors propose to learn an error estimation model that is a function of current state, policy and horizon by estimating the compounding error as a new reward using policy evaluation.
Pros -
I think the novelty in this paper is most appealing for me to vote for a clear acceptance. The growth of model-based reinforcement learning is in a big need of a framework to work with the planning horizon.
The proposed method is a very elegant way of estimating the errors and choose the appropriate horizon.
Still, I think there is a lot of room for improvement, as summarized below.
Cons
- One thing that concerns me is that the maximum horizon H looks very small. A max horizon of 3, 5 or even 10 is still much smaller than the value we usually use in planning based reinforcement learning (MPC in your case). A common choice can be up to 30 or 50 for PETS based algorithms [1].
Does that indicate that the proposed framework is not stable or robust when the horizon is big? Or maybe that scale of horizon is not common in Steve and thus not tried in the experimental section?
- Experiments
For the high dimensional control tasks, only Half-Cheetah and Swimmer was tried. It will generalize the conclusion if the results of other benchmarking environments are tested, including complex tasks such as Humanoid, and easier tasks such as Cart-pole.
- Algorithms
The used baselines in the paper are generally not considered to be state-of-the-art. I guess it is sort-of fair comparison since you add the proposed component on top of the baselines. But extending the current state-of-the-art such as TD3 [2] / PETS
I also suggest adding this benchmarking paper into the related work, which empirically studies the compounding error in different state-of-the-art model-free and model-based reinforcement algorithms.
[1] Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018). Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems (pp. 4754-4765).
[2] Fujimoto, Scott, Herke van Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." arXiv preprint arXiv:1802.09477 (2018).
[3] Wang, Tingwu, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel and Jimmy Ba. “Benchmarking Model-Based Reinforcement Learning.” ArXiv abs/1907.02057 (2019): n. pag.