We study learning control in an online lifelong learning scenario, where mistakes can compound catastrophically into the future and the underlying dynamics of the environment may change. Traditional model-free policy learning methods have achieved successes in difficult tasks due to their broad flexibility, and capably condense broad experiences into compact networks, but struggle in this setting, as they can activate failure modes early in their lifetimes which are difficult to recover from and face performance degradation as dynamics change. On the other hand, model-based planning methods learn and adapt quickly, but require prohibitive levels of computational resources. Under constrained computation limits, the agent must allocate its resources wisely, which requires the agent to understand both its own performance and the current state of the environment: knowing that its mastery over control in the current dynamics is poor, the agent should dedicate more time to planning. We present a new algorithm, Adaptive Online Planning (AOP), that achieves strong performance in this setting by combining model-based planning with model-free learning. By measuring the performance of the planner and the uncertainty of the model-free components, AOP is able to call upon more extensive planning only when necessary, leading to reduced computation times. We show that AOP gracefully deals with novel situations, adapting behaviors and policies effectively in the face of unpredictable changes in the world -challenges that a continual learning agent naturally faces over an extended lifetime -even when traditional reinforcement learning methods fail.
The authors study continual, lifelong learning. They suggest a new algorithm, named Adaptive Online Planning (AOP) that combines model-based planning with model-free learning. AOP decides how much additional planning is needed based on the uncertainty of the model-free learner and the performance of the planner. Experiments are carried out on three tasks, i.e. Hopper, Ant and a Maze.
This paper should be rejected. The main reason is that the experiments were only performed for 3 different seeds and are therefore not statistically relevant (see Henderson et al. "Deep reinforcement learning that matters." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.). 
Besides the issue of significance of the results section, there are other concerns. Some of them are:
- Page 2: 'The dynamics model is updated immediately at world changes.' - Is this a reasonable assumption? Where does an accurate model come from? Given a perfect model, it is not surprising that a learner that is combined with such a model achieves a superior performance.
- Although the authors state that the 'ability to perform well in old tasks (backward transfer)' is important, they don't explicitly show their algorithm to achieve this goal. Backwards transfer might be included into the experimental section, but I could not find a statement that addresses this explicitly.
- I would like the authors to crisply define their use of the word 'online learning'. Does online learning simply mean to process each sample as it is available or does the term include real-time?
- How is \sigma_{thres} chosen? What is the influence of this parameter?
- The statement that 'AOP uses only 0 - 25% of the number of timesteps as MPC-8, but achieves generally comparable or stronger performance.' is wrong (see Fig 4, d and e). This statement is especially difficult, as results are only averaged over 3 runs.
There are furthermore a few minor concerns:
- the interval for \gamma should exclude 1 in this setting, as the return would otherwise be unbounded.
- In the background section, the authors confuse the definition of the return with reward.
- the term 'deep exploration' is used but not defined
- There are two figures between the subsection header for 4.4 and the text - this is highly confusing