Reinforcement learning algorithms rely on carefully engineered rewards from the environment that are extrinsic to the agent. However, annotating each environment with hand-designed, dense rewards is difficult and not scalable, motivating the need for developing reward functions that are intrinsic to the agent. Curiosity is such intrinsic reward function which uses prediction error as a reward signal. In this paper: (a) We perform the first large-scale study of purely curiosity-driven learning, i.e. without any extrinsic rewards, across 54 standard benchmark environments, including the Atari game suite. Our results show surprisingly good performance as well as a high degree of alignment between the intrinsic curiosity objective and the hand-designed extrinsic rewards of many games. (b) We investigate the effect of using different feature spaces for computing prediction error and show that random features are sufficient for many popular RL game benchmarks, but learned features appear to generalize better (e.g. to novel game levels in Super Mario Bros.). (c) We demonstrate limitations of the predictionbased rewards in stochastic setups. Game-play videos and code are at https: //doubleblindsupplementary.github.io/large-curiosity/.
The authors consider the setting of a RL agent that exclusively receives intrinsic reward during training that is intended to model curiosity; technically, ‘curiosity’ is quantified by the ability of the agent to predict its own forward dynamics [Pathak, et al., ICML17]. This study primarily centers around an initially somewhat surprising result that non-trivial policies can be learned for many ’simpler’ video games (e.g., Atari, Super Mario, Pong) using just curiosity as reward. While this is primarily an empirical study, one aspect considered was the observation representation (raw pixels, random features, VAE, and inverse dynamics features [Pathak, et al., ICML17]). In examining reward curves (generally extrinsic during testing), ‘curiosity-based’ reward generally works with the representation effectiveness varying across different testbeds. They also conduct more in-depth experiments on specific testbeds to study the dynamics (e.g., Super Mario, Juggling, Ant Robot, Multi-agent Pong) — perhaps most interestingly showing representation-based transfer of different embeddings across levels in Super Mario. Finally, they consider the Unity maze testbed, combining intrinsic rewards with the end-state goal reward to generate a more dense reward space. 
From a high level perspective, this is an interesting result that ostensibly will lead to a fair amount of discussion within the RL community (and already has based on earlier versions of this work). However, it isn’t entirely clear if the primary contribution is showing that ‘curiosity reward’ is a potentially promising approach or if game environments aren’t particularly good testbeds for practical RL algorithms — given the lack of significant results on more realistic domains, my intuition leans toward the later (the ant robot is interesting, but one can come up with ‘simulator artifact’ based explanations). And honestly, I think the paper reads as if leaning toward the same conclusion. Regardless, given the prevalence of these types of testbed environments, either is a useful discussion to have. Maybe the end result could minimally be a new baseline that can help quantify the ‘difficulty’ of a particular environment. 
From the perspective of a purely technical contribution, there are fewer exciting results. The basic method is taken from [Parthak, et al., ICML17] (modulo some empirical choices such as using PPO). The comparison of different observation representations doesn’t include any analytical component, the empirical component is primarily inconclusive, and the position statements are fairly non-controversial (and not really conclusively supported). The testbeds all existed previously and this is mostly the effort of pulling then together. Even the ‘focused experiments’ can be explained with the intuitive narrative that in the state/action space, there is always more uncertainty the farther one goes from the starting point and this is more of a result of massive computation being applied primarily to problems that are designed to provide some level of novelly (the Roboschool examples are a bit more interesting, but also less conclusive). Finally, Figure 5 is interesting in showing that ‘curiosity + extrinsic’ improves over extrinsic rewards — although this isn’t particularly surprising for maze navigation that has such sparse rewards and can be viewed as something like ‘active exploration’. With respect to this specific setting, the authors may want to consider [Mirowski, et al., Learning to Navigate in Complex Environments, ICLR17] with respect to auxiliary loss + RL extrinsic rewards to improve performance (in this case, also in maze environments).
In just considering the empirical results, they clearly entail a fair amount of effort and just a dump of the code and experiments on the community will likely lead to new findings (even if they are that game simulators are weaker testbeds than previously thought). It is easy to ask for additional experiments (i.e., other mechanisms of uncertainty such as the count-based discussed in related work, other settings in 2.2) — but the quality seems high enough that I basically trust the settings and findings. Beyond the core findings, the other settings are less convincingly supported by seem more like work in progress and this paper is really just a scaling-up of [Pathak, et al., ICML17] without generating any strong results regarding questions around representation, what to do about stochasticity (although the discussion regarding something like ‘curiosity honeypots’ is interesting). Thus, it reads like one interesting finding around curiosity-driven RL working in games plus a bunch of preliminary findings trying to grasp at some explanations and potential future directions.
Evaluating the paper along the requested dimensions:
= Quality: The paper is well-written with a large set of experiments, making the case that exclusively using curiosity-based reward is very promising for the widely-used game RL testbeds. Modulo a few pointers, the work is well-contextualized and makes reasonable assumptions in conducting its experiments. The submitted code and videos result in a high-quality presentation and trustworthiness of the results. (7/10)
= Clarity: The paper is very clearly written. (7/10)
= Originality: The algorithmic approach is a combination of [Parthak, et al., ICML17] and [Schulman, et al. 2017] (with some experiments using [Kingma & Welling, 2013]). All of the testbeds have been used previously. Other than completely relying on curiously-based reward exclusively, there is little here. In considering combining with extrinsic rewards, I would also consider [Mirowski, et al., ICLR17], which is actually more involved in this regard. (4/10)
= Significance: Primarily, this ‘finishes’ [Parthak, et al., ICML17] to its logical conclusion for game-based environments and should spur interesting conversations and further research. In terms of actual technical contributions, I believe much less significant. (5/10)
=== Pros ===
+ demonstrates that curiosity-based reward works in simpler game environments
+ (implicitly) calls into question the value of these testbed environments
+ well written, with a large set of experiments and some interesting observations/discussions
=== Cons ===
- little methodological innovation or analytical explanations
- offers minimal (but some) evidence that curiosity-based reward works in more realistic settings
- doesn’t answer the one question regarding observation representation that it set out to evaluate
- the more interesting problem, RL + auxiliary loss isn’t evaluated in detail
- presumably, the sample complexity is ridiculous
Overall, I am ambivalent. I think that more casual ML/RL researchers will find these results controversial and surprising while more experienced researchers will see curiosity-driven learning to be explainable primarily by the intuition of the “The fact that the curiosity reward is often sufficient” paragraph of page 6, demanding more complex environments before accepting that this form of curiosity is particularly useful. The ostensible goal of learning more about observation representations is mostly preliminary — and this direction holds promise of for a stronger set of findings. Dealing with highly stochastic environments seems a potential fatal flaw of the assumptions of this method. However, as I said previously, this is probably a discussion worth having given the popularity and visibility of game-based testbeds — so, coupled with the overall quality of the paper, I lean toward a weak accept.