CMMI10
Traditional exploration methods in reinforcement learning (RL) require agents to perform random actions to find rewards. But these approaches struggle on sparse-reward domains like Montezuma's Revenge where the probability that any random action sequence leads to reward is extremely low. Recent algorithms have performed well on such tasks by encouraging agents to visit new states or perform new actions in relation to all prior training episodes (which we call acrosstraining novelty). But such algorithms do not consider whether an agent exhibits intra-life novelty: doing something new within the current episode, regardless of whether those behaviors have been performed in previous episodes. We hypothesize that across-training novelty might discourage agents from revisiting initially non-rewarding states that could become important stepping stones later in training-a problem remedied by encouraging intra-life novelty. We introduce Curiosity Search for deep reinforcement learning, or Deep Curiosity Search (DeepCS), which encourages intra-life exploration by rewarding agents for visiting as many different states as possible within each episode, and show that DeepCS matches the performance of current state-of-the-art methods on Montezuma's Revenge. We further show that DeepCS improves exploration on Amidar, Freeway, Gravitar, and Tutankham (many of which are hard exploration games). Surprisingly, DeepCS also doubles A2C performance on Seaquest, a game we would not have expected to benefit from intra-life exploration because the arena is small and already easily navigated by naive exploration techniques. In one run, DeepCS achieves a maximum training score of 80,000 points on Seaquest-higher than any methods other than Ape-X. The strong performance of DeepCS on these sparse-and dense-reward tasks suggests that encouraging intra-life novelty is an interesting, new approach for improving performance in Deep RL and motivates further research into hybridizing across-training and intra-life exploration methods.
This paper proposes use of intra-life coverage (an agent must visit all locations within each episode) for effective exploration in Atari games. This is in contrast of approaches that use inter-life coverage or curiosity metrics to incentivize exploration. The paper shows detailed results and analysis on 2 Atari games: Montezuma’s Revenge and Seaquest, and reports results on other games as well.
Strengths
1. Intuitively, the idea of intra-life curiosity is reasonable. The paper pursues this idea and provides experimental evidence towards it on 2 Atari games. It is able to show compelling improvements on the challenging Montezuma’s Revenge game.
Weaknesses
1. The two primary comparison points are missing:
1a. Comparison to other exploration methods. A number of methods that use state visitation counts (also referred to as diversity, eg. [A,B]), or prediction error (also referred to as curiosity, eg [C]) have been proposed in recent years. It is important to place the contributions in this paper in context of these other works. A number of these references are missing and no experimental comparison to these methods has been made. 
1b. Comparison between inter and intra life curiosity. One of the central motivation is the utility of intra-life curiosity vs inter-life curiosity, yet no comparisons to this effect have been provided.
2. Additionally, the paper employs a custom way of computing coverage (or diversity). It is in terms of location of agent on the screen, as opposed to featurization of the full game screen as used in prior works. It is possible that a large part of the gain comes from the clever design of the space for computing intrinsic exploration reward. The paper tries to control for it, however that description is rather short and vague (not clear how the proposed reward is computed without there being a grid, or how is the grid useful without the intrinsic reward). More details should be provided, and when comparisons to past works or inter-life curiosity are made, this should be controlled for. The two ideas (use of grids, and intra-life curiosity vs inter-life curiosity) should be independently investigated and put in context of past work.
3. I will encourage investigation on a more varied set of tasks. Perhaps, also using some MuJoCo environments, or 3D navigation environments. Table 1 tries to provide some comparisons on Atari, however number of samples is different for different methods making the comparisons invalid. Additionally, all of these are still on Atari.
[A] Diversity is All You Need: Learning Skills without a Reward Function Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine
[B] EX2: Exploration with Exemplar Models for Deep Reinforcement Learning Justin Fu, John D. Co-Reyes, Sergey Levine
[C] Curiosity-driven Exploration by Self-supervised Prediction Deepak Pathak, Pulkit Agrawal, Alexei A. Efros and Trevor Darrell International Conference on Machine Learning (ICML), 2017