We introduce an exploration bonus for deep reinforcement learning methods that is easy to implement and adds minimal overhead to the computation performed. The bonus is the error of a neural network predicting features of the observations given by a fixed randomly initialized neural network. We also introduce a method to flexibly combine intrinsic and extrinsic rewards. We find that the random network distillation (RND) bonus combined with this increased flexibility enables significant progress on several hard exploration Atari games. In particular we establish state of the art performance on Montezuma's Revenge, a game famously difficult for deep reinforcement learning methods. To the best of our knowledge, this is the first method that achieves better than average human performance on this game without using demonstrations or having access to the underlying state of the game, and occasionally completes the first level. This suggests that relatively simple methods that scale well can be sufficient to tackle challenging exploration problems.
The algorithm proposed in this paper consists in driving exploration in RL through an intrinsic reward, computed as the prediction error of a neural network whose target is the output of a randomly initialized network (with the state reached by the agent as input). The intuition is that rarely seen states will have a large prediction error, thus encouraging the agent to visit them (until they have been seen often enough that the error goes down). Among potential benefits of this method, compared to previously proposed intrinsic curiosity techniques for RL, are its simplicity and its robustness to environment stochasticity. Extensive experiments on the Atari game Montezuma’s Revenge investigate several variants of this idea (combined with PPO), with the best results significantly outperforming the current state-of-the-art. Other results on five other hard exploration Atari games show competitive performance as well.
The proposed technique definitely exhibits impressive performance on some tasks, in spite of its simplicity. Despite lacking theoretical grounding, I believe such results should be quite interesting to the RL research & applied community, as a novel and easy way to encourage exploration in sparse rewards tasks. I also really appreciate that the authors have included “negative” results contradicting their expectations, and are sharing their code: this is the kind of openness that in my opinion should be highly encouraged.
The paper is overall well written and easy to follow, except (from my point of view) section 2.2.2, which I found rather confusing and not very convincing. First, eq. 1 is a bit surprising since one expects the posterior to be in the same family of functions, i.e. of the form f_theta rather than f_theta + f_theta*. After a (very superficial) look at Osband et al (2018) I see that this particular lemma holds for linear functions, and the extension to nonlinear function approximation seems to be essentially based on intuition. Then the sentence “the optimization problem (...) is equivalent to distilling a randomly drawn function from the prior” ignores the sign mismatch (we are actually distilling the opposite of f_theta*, though I agree it can still make sense with a symmetric prior around 0, which is not mentioned). Finally, the reasoning to reach the conclusion “the distillation error could be seen as a quantification of uncertainty in predicting the constant zero function” seems somewhat unconvincing to me, considering the significant differences compared to Osband et al (2018), in particular: sharing weights among models in the ensemble, ignoring the specific regularization term R(theta), and not adding noise to the training data. As a result I find this link rather weak and I would appreciate if this section could be improved (at the very least with a better explanation of its limitations)
Among the various findings from experiments, one puzzled me in particular: the striking difference between episodic and non episodic intrinsic rewards in Fig. 3. I think this would have deserved a more thorough empirical investigation than the intuitive explanation from 2.3 (e.g. by checking whether the agent trained with non episodic rewards was indeed taking more risks and thus dying more often). What I find particularly surprising is that the beginning of the game should not yield much intrinsic reward relatively fast, since it should be the part the agent sees most often initially. As a result, I would expect that getting zero reward when dying (episodic rewards) should not be much different from getting future (small and discounted) intrinsic rewards, unless maybe early in training. What am I missing here?
I also have some comments regarding a couple of other findings and associated hypotheses:
- Section 3.3 shows some surprising results when varying discount factors (“This is at odds with the results in Figure 3 where increasing gamma_I did not significantly impact performance”). I wonder however to which extent these may be caused by the difference in the scale of discounted returns: for instance increasing gamma_I from 0.99 to 0.999 will (roughly) multiply V_I by 10, giving it more weight in the sum V = V_E + V_I. A fair comparison would either rescale V_I accordingly , or use a weighted sum and optimize the weights (the hyper-parameters table in the Appendix suggests that weights were actually used, but they are not mentioned in the main text and it is not clear how they were chosen).
- 3.7 shows an interesting behavior (“dancing with skulls”). The authors hypothetize it may be due to the inherent danger of such behavior. But could it be also (and possibly more) related to the fact the skulls are moving? (which leads to many varied different states, that the predictor network will take time to learn perfectly).
Here are a few more questions for the authors regarding specific details:
1. In 3.1, “The best return achieved by 4 out 5 runs of this setting was 6,700.” What does this mean?
2. In 3.5 a downsampling scheme is used to keep the training speed of the predictor network constant when increasing the number of actors. This raises the question of the impact of this training speed on the results, which is not investigated in the current experiments: do hyper-parameters influencing the predictor’s training speed (e.g. downsampling ratio, learning rate) need to be very carefully tuned, or are results robust across a wide range of speeds?
3. In A.5 there is mention of “a CNN policy with access to only the last 16 most recent frames”: does that mean the number of “frames stacked” (Table 2) was increased from 4 to 16? If so, why? (it is not clear to me what we learn compared to Fig. 4)
4. Your technique implicitly relies on the assumption that the predictor network’s weights will never be exactly the same as the target network’s (as otherwise nothing will be novel anymore, regardless of the states being visited). Do you foresee potential issues with this, and if yes do you have any idea to solve them? (a short discussion in the paper on this topic would be good as well)
And finally some suggestions for small improvements:
- Please try to find another name than “target” network since it is already widely used in the deep RL literature for something completely different (suggestions: “random”, “distillation”, “feature”, “reference”)
- In 2.1 (last paragraph) there are various papers cited regarding forward or inverse dynamics, but several of them contain both, while the way they are cited suggests they deal only with one. Just moving “and inverse dynamics” before the full list of citations would fix it.
- In first paragraph of section 3 please mention that the algorithm is based on PPO
- In Fig. 3 the x axis seems to be missing a multiplication by 1K (?)
- At end of 3.2, “having two value heads is necessary for combining reward streams with different characteristics”, please specify what are these characteristics. 
- On p. 7, last paragraph: please (briefly) explain how the “random features” are computed
- The reference Ostrovski et al appears twice
- In Alg. 1, “Update reward normalization parameters using it”: the “s” in parameters can be misleading, suggesting that both mean and standard deviation are used for normalization => explicitly saying “Update running standard deviation” would avoid such confusion (or say it on the “Normalize” step below)
- Alg. 1 is not very clear on how returns and advantages are computed (and the corresponding code is not super easy to read). It also seems to be missing the update of the critic V.
- Alg. 1 mentions “number of optimization steps” while Table 4 says “Number of optimization epochs”: I guess they are the same, so they should probably have the same name
- After reading the paper, I felt like one learning was that CNN models worked better than RNN ones. However Table 5 shows that this can vary between games (ex: RND RNN outperforms RND CNN on Gravitar and Solaris) and/or algorithms (ex: PPO RNN outperforms PPO CNN on 3 games). I think the main text should at least point to this table when mentioning the superiority of the CNN.
- In the “Related work” section there is a very short paragraph about “vectorized value functions”. It seems to be overlooking the whole field of multi-objective reinforcement learning. Maybe you could cite a related survey paper like “A Survey of Multi-Objective Sequential Decision-Making”.
- The paper’s title and the OpenReview submission name should probably match
Update following author and reviewer discussion: I agree with others regarding the weakness of the empirical comparison to pseudo-counts in particular, but still believe that the paper deserves to be accepted due to the fact that (1) some of the results are really good, and (2) this is a simple original idea that has the potential to drive further advances (hopefully addressing the empirical and theoretical limitations of the current work)