Model-free reinforcement learning (RL) can be used to learn effective policies for complex tasks, such as Atari games, even from image observations. However, this typically requires very large amounts of interaction -substantially more, in fact, than a human would need to learn the same games. How can people learn so quickly? Part of the answer may be that people can learn how the game works and predict which actions will lead to desirable outcomes. In this paper, we explore how video prediction models can similarly enable agents to solve Atari games with fewer interactions than model-free methods. We describe Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models and present a comparison of several model architectures, including a novel architecture that yields the best results in our setting. Our experiments evaluate SimPLe on a range of Atari games in low data regime of 100k interactions between the agent and the environment, which corresponds to two hours of real-time play. In most games SimPLe outperforms state-of-the-art model-free algorithms, in some games by over an order of magnitude.
Summary
This paper proposes a model-based reinforcement learning algorithm suitable for high-dimensional visual environments like Atari. The algorithmic loop is conceptually simple and comprises 1) collecting real data with the current policy 2) updating an environment model with old and newly acquired data and 3) updating the policy "virtually" inside of the environment model using PPO. The approach is evaluated on 26 games from the Atari benchmark and compared against the model-free baselines Rainbow DQN and PPO. The newly proposed model-based method clearly outperforms both model-free baselines in low training regimes (100,000 steps). Further ablation studies are provided, e.g. similar results are obtained in a more stochastic setting of ALE with sticky actions.
Quality
This paper has a strong applied focus and needs to be judged based on its experiments. The quality of those are high. The method is evaluated on a suite of 26 games, compared to strong model-free baselines and results are averaged over 5 seeds. One concern I have is that the method is only evaluated in low training regimes. While I do understand that increasing the training horizon is computationally demanding, results in the appendix (Figure 11a) indicate that the proposed model-based method has worse asymptotic performance compared to the model-free baselines. After 500,000 training steps the effect of sample efficiency vanishes and the final performance results are far away from the final performance results of the model-free baselines after 50,000,000 training steps. Also, a plot similar to Figure 11a) from the appendix for Rainbow DQN would be good (but I do understand this might be difficult to obtain in the course of the review period should this require more experiments).
Clarity
The paper is clearly written and easy to follow. However, the authors could state in the main paper more clearly that their method excels in low training regimes and that the sample efficiency effect seems to vanish when increasing training iterations from 100,000 to 500,000 steps. In fact, Figure 11a) from the appendix should go into the main paper, and it should be also mentioned that there is a huge discrepancy between the maximum performance achieved by the proposed model-based method and the maximum performance achieved by the model-free baselines when training for 50,000,000 steps. Based on the experiments, it is not clear at all if the new method will eventually catch up with best model-free results from the literature when training time is increased, or stall in low-performance regimes indefinitely.
Originality
The originality of this paper is not very high since the proposed algorithm and its components are not novel (there might be some minor novelty in the environment model architecture). However, this paper should not be judged based on its originality but based on its significance. 
Significance
A working model-based RL algorithm for Atari is clearly a huge gap in the current literature and this paper takes an important step towards this direction. Demonstrating improved sample efficiency compared to strong model-free baselines in low training regimes is a significant result. The significance is however decreased by the fact that the paper does not answer the question how to obtain good asymptotic performance that matches (or comes close to) model-free state-of-the-art results. I therefore vote for weak accept at this stage.
Minor details
On a side note, there are two citations missing related to model-based RL in visual domains:
- S. Alaniz. Deep Reinforcement Learning with Model Learning and Monte Carlo Tree Search in Minecraft. In the 3rd Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM), 2017.
- F. Leibfried and P. Vrancx. Model-Based Regularization for Deep Reinforcement Learning with Transcoder Networks. In NIPS Deep Reinforcement Learning Workshop, 2018.
Update
After the author response, my review remains the same. I think this paper is worthwhile publishing at ICLR.