We propose a reinforcement learning agent to solve hard exploration games by learning a range of directed exploratory policies. We construct an episodic memorybased intrinsic reward using k-nearest neighbors over the agent's recent experience to train the directed exploratory policies, thereby encouraging the agent to repeatedly revisit all states in its environment. A self-supervised inverse dynamics model is used to train the embeddings of the nearest neighbour lookup, biasing the novelty signal towards what the agent can control. We employ the framework of Universal Value Function Approximators (UVFA) to simultaneously learn many directed exploration policies with the same neural network, with different trade-offs between exploration and exploitation. By using the same neural network for different degrees of exploration/exploitation, transfer is demonstrated from predominantly exploratory policies yielding effective exploitative policies. The proposed method can be incorporated to run with modern distributed RL agents that collect large amounts of experience from many actors running in parallel on separate environment instances. Our method doubles the performance of the base agent in all hard exploration in the Atari-57 suite while maintaining a very high score across the remaining games, obtaining a median human normalised score of 1344.0%. Notably, the proposed method is the first algorithm to achieve non-zero rewards (with a mean score of 8,400) in the game of Pitfall! without using demonstrations or hand-crafted features.
-after rebuttal:
I read the replies from the authors and re-read the modified version of the paper and I believe there has been a noticeable improvement in the presentation. I still think it could be improved more (in terms of wording and better exposition of the results) but due to the in-place improvements I increase my score to weak accept. 
----------------------------------------------------------------------------------------------------------------------------------------------------
In this paper, the authors present a methodology for generating intrinsic rewards for reinforcement learning agents targeting hard exploration environments. The intrinsic reward is generated using an episodic memory module and a lifelong novelty module.  A state representation is learnt in such a way that the novelty signals are biased towards what the agent can control. A single neural network learns the q-values of exploratory policies with different degrees of exploration. Several experiments on a simple domain and on Atari domains are conducted to evaluate and compare the performance of the proposed method against the baselines. 
In my opinion, this is an interesting idea as it present a novel combination of methods (Random Network Distillation and Episodic Memory) that works and can inspire other researchers in the field.  However, I would like to see clearer explanations in the experimental section before acceptance. For this reason I rate this paper as a weak reject but if clarity is improved I will increase my score.  See below for more general comments and detailed explanation about the experimental section.
General comments:
- The authors state that a novel contribution is to disentangle exploration and exploitation. This is not true: see [1] for a recent paper on the topic. I believe the authors should cite this paper.
- In page 3: “To determine the bonus, the current observation is compared to the content of the episodic memory. Larger differences produce larger episodic intrinsic rewards” When computing the intrinsic reward for a new state, it must be compared to the episodic memory, which is composed by recent states (and old). Therefore, in this situation the intrinsic reward is always high? Specially because it is compared with the k-nearest neighbours?  Maybe the authors could elaborate on this?
Experiment Section:
This section should be tidied up in my view.  In general, I feel that there are too many fine-grained results / experiments. I think some aggregation of results would be good for clarity. Without this aggregation there are statements made from the authors that are difficult to believe, e.g. a general result statement is valid for one or two games but not for the rest (but still the statement is formulated in a general way). This can lead to misunderstandings and overstatements.  This together with lack of some experimental details makes the section very dense and difficult to parse. Below I make my points. They are in order of appearance in the main manuscript. I marked with (*) the ones I consider the most important. 
Section 4.1:
- Why the exploration policy in this section is set to have $\beta = 0.3$ ? Is my understanding correct that any value of $\beta$ would produce similar results since this is just a scaling of the intrinsic reward (i.e. it does not balance exploration vs exploitation)?
- The parameter $\beta$ appears in the construction of the reward r = r^e + \beta r^i , but also the output of Algorithm 1 scales the similarity by $\beta$ does that mean that the final contribution of $\beta$ is squared? Is this intended? If so, why?
- At the end of this section: “However, staying still is enough: staying still every state will produce ….” Since the agent can only take 4 action {left, right, up, down}, what do the authors mean by “staying still”, is really the agent doing some sort of cyclic policy e.g. left right left right … ?
Section 4.2:
- In the appendix A it is stated “use last 5 frames of the sampled sequences to train the action prediction network… “ Does this refer to frame-stacking? I assume it is not since at the beginning section 4.2 it is stated that there is no frame-stacking. If it is not frame-stacking, the authors could explain in more detail what do they refer to.
- In the paragraph “Architecture” it is stated that 8 games were selected to choose the hyperparemeters and that the results are in Appendix B. However, appendix B only shows 2 games (Pitfall and Montezuma’s Revenge). Is this a typo?
- (*) In paragraph “NGU Agent”: This is the most dense paragraph and the most difficult to parse. First of all Table 1 shows all the results, but as one can see, the different ablations have very similar performance on most games with only a few exceptions. Note that most of the mean performances with error bars, have actually overlapping error bars for many combinations of games and methods. Therefore, further statements about this table are difficult to believe since they could have been just the result of random seeds. I think it is fine to show these fine-grained results on the appendix, but I would say it would be better to aggregate them in the main paper and show that the statements made by the authors still hold for this aggregation.
Specific comments about NGU agent paragraph:
- “ we observe an improvement from increasing the number of mixtures (with diminishing returns) on hard exploration games.” I would say the authors cannot claim this since the hard exploration games are the ones in table 2, which are different from the ones in table 1 (only Pitfall, MR and Private Eye coincide). Also the statement is only true for 2 the three that are hard exploration games (Pitfall and Private Eye).
- (*)  “for smaller $\beta$ we observe better performance on Pong and Beam Rider, but worse performance on all hard exploration games” This is a strange result in my opinion. If I understood correctly, the base agent has $\beta = 0.3$, which supposedly has been selected to be good on hard-exploration games. However, here only changing $\beta$ slightly, reduces the performance on hard-exploration games (Pitfall and Private Eye) significantly. Is this due to the parameter being very sensitive? If so, I believe the authors should report how sensitive are the results to this parameter, specially, on the full hard-exploration games.
- “superhuman performance on 3 games”: which ones?
Comments on paragraph on “hard exploration games”: 
- “with NGU(N=1)-RND ….” what do the authors mean by this? This seems to be the best setting for Pitfall but it is actually not using mixture of explorations and (I guess) neither RND?
- Table 2 first row “best base” what does this mean?
Minor Comments:
-In Equation 3 the squared distance is normalized by a running average. Why?
- Right after Equation 3: “… episodic reward can be found in Alg. 14” . Probably meant Alg 1.
- Similarly why the errors are normalized by a running average when computing \alpha_t ?
[1] MULEX: Disentangling Exploitation from Exploration in Deep RL ( Lucas Beyer et al )