Model-based reinforcement learning (MBRL) is widely seen as having the potential to be significantly more sample efficient than model-free RL. However, research in model-based RL has not been very standardized. It is fairly common for authors to experiment with self-designed environments, and there are several separate lines of research, which are sometimes closed-sourced or not reproducible. Accordingly, it is an open question how these various existing algorithms perform relative to each other. To facilitate research in MBRL, in this paper we gather a wide collection of MBRL algorithms and propose over 18 benchmarking environments specially designed for MBRL. We benchmark these algorithms with unified problem settings, including noisy environments. Beyond cataloguing performance, we explore and unify the underlying algorithmic differences across MBRL algorithms. We characterize three key research challenges for future MBRL research: the dynamics bottleneck, the planning horizon dilemma, and the early-termination dilemma. Finally, to facilitate future research on MBRL, we open-source our benchmark 1 .
This paper performs an empirical comparison of a number of model-based RL (MBRL) algorithms over 18 benchmarking environments. The authors propose a set of common challenges for MBRL algorithms.
I appreciate the effort put into this evaluation and I do think it helps the community gain a better understanding of these types of algorithms. My main issue with the paper is that I don't find the evaluation thorough enough (for instance, no tabular environments are evaluated) and the writing still needs quite a bit of work. I encourage the authors to continue this line of work and improve on what they have for a future submission!
More detailed comments:
- Intro: "model-free algorithms ... high sample complexity limits largely their application to simulated domains." I'm not sure this is a fair criticism. Model-based are also mostly run on simulations, so sample efficiency is not necessarily the cause model-free are only run on simulations. Further, this statement kind of goes against your paper, since all your evaluations are on simulations!
- Intro: "2) Planning horizon dilemma: ... increasing planning horizon... can result in performance drops due to the curse of dimensionality..." and "similar performance gain[sic] are not yet observed in MBRL algorithms, which limits their effectiveness in complex environments" goes against the point you made earlier on in the intro about sample efficiency.
- Preliminaries: "In stochastic settings, it is common to represent the dynamics with a Gaussian distribution", this is only for continuous states. It would be nice if you could evaluate tabular environments as well.
- Sec 4.1: "we modify the reward funciton so that the gradient... exists..." which environments were modified and how did they have to be modified?
- Sec 4.1: You discuss early termination but have not defined what exactly you mean by it.
- Fig 1: 12 curves is still a lot and really hard to make much sense of. A lot of the colors are very similar.
- Sec 4.1: "it takes an impractically long time to train for 1 million time-steps for some of the MBRL algorithms" why?
- Table 1 is a *lot* of numbers and colors and is really hard to make much sense of. There are also so many acronyms on the LHS it's difficult to keep track.
- Table 2: What about memory usage?
- Sec 4.4: "Due to the limited exploration in baseline, the performance is sometimes increased after adding noise that encourages exploration." Why does this noise not help exploration in the baselines?
- Sec 4.5: "This points out that when learning models more data does not result in better performance." This seems like it's closely correlated with the particular form chosen for the model parameterization more than anything.
- Fig 3: y-axis says "Relative performance", relative to what?
- Sec 4.7: "Early termination...is a standard technique... to prevent the agent from visiting unpromising states or damaging states for real robots." I've never seen this used as a justification for this.
- Sec 4.7: "Table 1, indicates that early termination does in fact decrease the performance for MBRL algorithms", and then you say "We argue that to perform efficient learning in complex environments... early termination is almost necessary." Those two statements contradict each other.
Minor comments to improve writing:
- When using citations as nouns, use \citep so you get "success in areas including robotics (Lillicrap et al., 2015)" as opposed to "success in areas including robotics Lillicrap et al. (2015)" (the latter is what you have all throughout the paper).
- Sec 3.1: s/This class of algorithms learn policies/These class of algorithms learn policies/