EPISODIC CURIOSITY THROUGH REACHABILITY
Rewards are sparse in the real world and most of today's reinforcement learning algorithms struggle with such sparsity. One solution to this problem is to allow the agent to create rewards for itself -thus making rewards dense and more suitable for learning. In particular, inspired by curious behaviour in animals, observing something novel could be rewarded with a bonus. Such bonus is summed up with the real task reward -making it possible for RL algorithms to learn from the combined reward. We propose a new curiosity method which uses episodic memory to form the novelty bonus. To determine the bonus, the current observation is compared with the observations in memory. Crucially, the comparison is done based on how many environment steps it takes to reach the current observation from those in memory -which incorporates rich information about environment dynamics. This allows us to overcome the known "couch-potato" issues of prior work -when the agent finds a way to instantly gratify itself by exploiting actions which lead to hardly predictable consequences. We test our approach in visually rich 3D environments in VizDoom, DMLab and MuJoCo. In navigational tasks from VizDoom and DMLab, our agent outperforms the state-of-the-art curiosity method ICM. In MuJoCo, an ant equipped with our curiosity module learns locomotion out of the first-person-view curiosity only. The code is available at https://github.com/google-research/episodic-curiosity.
The main idea of this paper is to propose a heuristic method for exploration in deep reinforcement learning. The work is fairly innovative in its approach, where an episodic memory is used to store agent’s observations while rewarding the agent for reaching novel observations not yet stored in memory. The novelty here is determined by a pre-trained network that computes the within k-step-reachability of current observation to the observations stored in memory. The method is quite simple but promising and can be easily integrated with any RL algorithm.
They test their method on a pair of 3D environments, VizDoom and DMLab. The experiments are well executed and analysed. 
Positives:
-	They do a rigorous analysis of parameters, and explicitly count the pre-training interactions with the environment in their learning curves.
-	This method does not hurt when dense environmental rewards are present.
-	The memory buffer is smaller than the episode length, which avoids trivial solutions.
-	The idea of having a discriminator assess distance between states is interesting.
Questions and critics:
-	The tasks explored in this paper are all navigation based tasks, would this method also apply equally successfully to non-navigation domains such as manipulation? 
-	My main concern is that the pre-training of the embedding and comparator networks directly depends on how good the random exploration policy is that collects the data. In navigation domains it makes sense that the random policy could cover the space fairly well, however, this will not be the case for more complex tasks involving more complex dynamics.
-	It was surprising to me that the choice of k does not seem to be that important. As it implicitly defines what “novelty” means for an environment, I would have expected that its value should be calibrated better. Could that be a function of the navigation tasks considered?
-	The DMLab results are not great or comparable to the state-of-the-art methods, which may hinder interpreting how good the policies really are. This was perhaps a conscious choice given they are only interested in early training results, but that seems like a confound.
-	The architecture does not include an RNN which makes certain things very surprising even though they shouldn't (e.g. firing, or moving around a corner, are specifically surprising for ICM) as they cannot be learnt, but perhaps if they had an RNN in the architecture these would be easy to explain? Would be interesting to see what are the authors thoughts on this (apart from their computational complexity argument they mention)?
-	Having the memory contain only information about the current episode with no information transfer between episodes seems a bit strange to me, I would like to hear the motivation behind this?
-	The fact that the memory is reset between episodes, and that the buffer is small, can mean that effectively the method implements some sort of complex pseudo count over meta-states per episode? 
-	The embedding network is only trained during the pre-training phase and frozen during the RL task. This sounds a bit limiting to me: what if the agent starts exploring part of the space that was not covered during pre-training? Obviously this could lead to collapses when allowing to fine-tune it, but I feel this is rather restrictive. Again, I feel that the choice of navigation tasks did not magnify this problem, which would arise more in harder exploration tasks.
-	I think that alluding that their method is similar to babies’ behaviour in their cradle is stretched at best and not a constructive way to motivate their work…
-	In Figure 6 and 7, all individual curves from each seed run are shown, which is a bit distracting. Perhaps showing the mean and std would be a cleaner and easier-to-interpret way to report these results?
Overall, it is a simple and interesting idea and seems quite easy to implement. However, everything is highly dependent on how varying the environment is, how bad the exploration policy used for pre-training is, how good the embeddings are once frozen, and how k, action repeat and memory buffer size interact. Given that the experiments are all navigation based, it makes it hard for me to assess whether this method can work as well in other domains with harder exploration setups.