Playing hard exploration games by watching YouTube
Deep reinforcement learning methods traditionally struggle with tasks where environment rewards are particularly sparse. One successful method of guiding exploration in these domains is to imitate trajectories provided by a human demonstrator. However, these demonstrations are typically collected under artificial conditions, i.e. with access to the agent's exact environment setup and the demonstrator's action and reward trajectories. Here we propose a two-stage method that overcomes these limitations by relying on noisy, unaligned footage without access to such data. First, we learn to map unaligned videos from multiple sources to a common representation using self-supervised objectives constructed over both time and modality (i.e. vision and sound). Second, we embed a single YouTube video in this representation to construct a reward function that encourages an agent to imitate human gameplay. This method of one-shot imitation allows our agent to convincingly exceed human-level performance on the infamously hard exploration games MONTEZUMA'S REVENGE, PITFALL! and PRIVATE EYE for the first time, even if the agent is not presented with any environment rewards.
Summary: This paper studies learning from demonstration/imitation learning. While usually supervision has to be intentionally provided by a human, the authors instead use YouTube videos as a form of supervision. They first align videos to a shared representation, using two concept that do not require any labeling: i) predicting the temporal distance between frames in the same video, and ii) predicting the temporal distance between a video and audio frame of the same video. Subsequently, they use the embedding on a novel video to generate checkpoints, which serve a intermediate rewards for an RL agent. Experiments show state-of-the-art performance amongst learning from demonstration approaches.   Strength: - The paper is clearly written, with logical steps between the sections, and good motivations. - Related work is well covered. - The results are impressive, and exceed the state-of-the-art on the studied games.   Weakness/Questions: - Introduction: You mention intrinsic motivation, but you miss another branch of exploration work, based on uncertain value functions. See for example [1,2].  -In the beginning of Section 4 I did not understand why you could suddenly sample checkpoints, while in Sec. 3 you only focused on learning a frame-wise embedding. The end of Sec 5. does explain how this is implemented (with a held-out video), which could be mentioned a bit earlier for clarity. - How much did you select the YouTube demonstrations? Were they expert players, who were very good at solving these games? That might explain why you results are better than e.g. DqfD? - Atari games have a fixed viewpoint, and you additionally spatially align the demonstrations if the videos don’t. To what extend is spatial alignment crucial? Would this approach scale to first-person-view games, where alignment is probably much harder?   Conclusion:  I believe this paper studies an interesting problem. There is big potential for the use of (human) supervision to guide learning, but supervision is a time consuming process. This paper studies video data as a potential form of supervision, which would indeed provide access to a huge amount of demonstrations. The paper contributes a novel method for alignment, and a simple but effective way to incorporate this embedding in an RL agent, which shows good results. It would be good to mention how expert the demonstration videos were, and maybe discuss the potential to generalize to non-fixed viewpoints. Besides that, I think this paper makes a clear contribution.   [1] White, D. J. "Uncertain value functions." Management Science 19.1 (1972): 31-41. [2] Osband, Ian, et al. "Deep exploration via randomized value functions." arXiv preprint arXiv:1703.07608 (2017).  After rebuttal: It is important to mention the scores and acquisition procedure of the demonstration videos indeed. They were expert demonstrations, but as you mention yourself, a benefit of your approach is that (through the alignment) you can leverage a series of unlabeled expert demonstrations.