Most Deep Reinforcement Learning methods perform local search and therefore are prone to get stuck on non-optimal solutions. Furthermore, in simulation based training, such as domain-randomized simulation training, the availability of a simulation model is not exploited, which potentially decreases efficiency. To overcome issues of local search and exploit access to simulation models, we propose the use of kinodynamic planning methods as part of a model-based reinforcement learning method and to learn in an off-policy fashion from solved planning instances. We show that, even on a simple toy domain, D-RL methods (DDPG, PPO, SAC) are not immune to local optima and require additional exploration mechanisms. We show that our planning method exhibits a better state space coverage, collects data that allows for better policies than D-RL methods without additional exploration mechanisms and that starting from the planner data and performing additional training results in as good as or better policies than vanilla D-RL methods, while also creating data that is more fit for re-use in modified tasks.
The paper aims to improve exploration in DRL through the use of planning. This is claimed to increase state space coverage in exploration and yield better final policies than methods not augmented with planner derived data.
The current landscape of DRL research is very broad, but RRT can only directly be applied in certain continuous domains with continuous action spaces. With learned embedding functions, RRT can be applied more broadly (see "Taking the Scenic Route: Automatic Exploration for Videogames" Zhan 2019). The leap from RRT-like motion planning to the general topic of "planning" for policy search is not well motivated explained with respect to the literature. Uses of Monte Carlo Tree Search (as in AlphaGo) seem obviously related here.
This reviewer moves to reject the paper primarily on the grounds of overinterpreting experimental results from a single, extremely simple example RL task. In a domain so small, we can't tease out the role of exploration, we aren't engaging with the "deep" of DRL, and we are only considering one specific kind of planning. The implicit claims of general improvement to exploration and improved downstream policies are not supported by the experimental results. At the same time, no theoretical argument is attempted that would make up for the very narrow nature of the experiments.
Questions for the authors:
- If HalfCheetah is used to motivate the work, and it is so easily available in the open source offerings from OpenAI, why isn't one (or many more) tasks of *at least* this complexity considered? MountainCar is one of the gym environments with a 2D phasespace compatible with the kinds of plots used in this paper.
- Could the authors taxonomize the landscape of planning and provide a specific argument for focusing on RRT? (RRT is a fun algorithm, but how will you draw the attention of other researchers who are currently focused on Atari games?)