Reinforcement learning (RL) has seen great advancements in the past few years. Nevertheless, the consensus among the RL community is that currently used model-free methods, despite all their benefits, suffer from extreme data inefficiency. To circumvent this problem, novel model-based approaches were introduced that often claim to be much more efficient than their model-free counterparts. In this paper, however, we demonstrate that the state-of-the-art model-free Rainbow DQN algorithm can be trained using a much smaller number of samples than it is commonly reported. By simply allowing the algorithm to execute network updates more frequently we manage to reach similar or better results than existing model-based techniques, at a fraction of complexity and computational costs. Furthermore, based on the outcomes of the study, we argue that the agent similar to the modified Rainbow DQN that is presented in this paper should be used as a baseline for any future work aimed at improving sample efficiency of deep reinforcement learning.
This paper presents an emprical study of how a properly tuned implementation of a model-free RL method can achieve data-efficiency similar to a state-of-the-art model-based method for the Atari domain. 
The paper defines r as ratio of network updates to environment interactions to describe model-free and model-based methods, and hypothesizes that model-based methods are more data-efficient because of a higher ratio r. To test this hypothesis, the authors take Rainbow DQN (model-free) and modify it to increase its ratio r to be closer to that SiMPLe (model-based). Using the modified verison of Rainbow (OTRainbow), the authors replicate an experimental comparison with SiMPLe (Kaiser et al, 2019), showing that Rainbow DQN can be a harder baseline to beat than previously reported (Figure 1). This paper raises an important point about empirical claims without properly tuned baselines, when comparing model-based to model-free methods, identifying the amount of computation as a hyperparameter to tune for fairer comparisons.
I recommend this paper to be accepted only if the following issues are addressed. The first is the presentation of the empirical results. In Figure 1, OTRainbow is compared against the reported results in (Kaiser et al, 2019), along with other baselines, when limiting the experience to 100k interactions. Then, in Figure 2, human normalized scores are reported for varying amounts of experience for the variants of Rainbow, and compared against SiMPLe with 100k interactions, with the claim that the authors couldn't run the method for longer experiences. Unless a comparison can be made with the same amounts of experience, I don't see how Figure 2 can be interpreted objectively. In any case, the results in Figure 1 and the appendix are useful for showing that the baselines used in prior works were not as strong as they could be.
The second has to do with the interpretation of the results. The paper chooses a single method class of model-based methods to do this comparison, namely dyna-style algorithms that use the model to generate new data. But models can also be used for value function estimation (Model Based Value Expansion) and reducing gradient variance(using pathwise derivatives). The paper is written as if the conclusions could be extended to model-based methods in general. Can we get the same conclusions on a different domain where other model-based methods have been successful; e.g. continuous control tasks? A way to improve the paper would be to make it clear from the beginning that these results are about Dyna-style algorithms in the Atari domain.