Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning
The combination of modern Reinforcement Learning and Deep Learning approaches holds the promise of making significant progress on challenging applications requiring both rich perception and policy-selection. The Arcade Learning Environment (ALE) provides a set of Atari games that represent a useful benchmark set of such applications. A recent breakthrough in combining model-free reinforcement learning with deep learning, called DQN, achieves the best realtime agents thus far. Planning-based approaches achieve far higher scores than the best model-free approaches, but they exploit information that is not available to human players, and they are orders of magnitude slower than needed for real-time play. Our main goal in this work is to build a better real-time Atari game playing agent than DQN. The central idea is to use the slow planning-based agents to provide training data for a deep-learning architecture capable of real-time play. We proposed new agents based on this idea and show that they outperform DQN.
Summary.This paper studies a number of variations on the topic of training a deep network using data generated by a Monte-Carlo Tree Search (MCTS) agent. The paper focuses on the Atari 2600 platform and is motivated by the observation that, while MCTS performs extremely well on Atari 2600 games, it is also too computationally expensive to be used in a realistic setting. The authors provide empirical results on a number of Atari 2600 games.Overall.To me, the main contribution of this paper is to propose to ``compile'' the UCT value function (or policy, according to the algorithm used) into a deep network. The paper is clear, and the results of good quality, but the work lacks in significance. While there are some nice results -- and performance improvements on all but one game -- I feel the topic is simply not sufficiently explored. There is little insight gained into how these results might carry over to other domains, or significant algorithmic improvements that could result from this work.A few major hurdles:    . The UCC-I algorithm, described here, is very reminiscent of the DAgger algorithm of Ross et al. (2011). It would probably be good to discuss the relationship. Do we expect better sampling guarantees than their approach?    . Are UCR/UCC really valid competitors to previous learning approaches? They still require full access to the simulator.    . The fact that the empirical results rely on game-specific tunings somewhat devalues said results. Are the results so different when a uniform environment parametrization is used?There are a number of research directions which I believe could improve the paper:    . Further studying how well a policy/value function can be summarized into a deep network. Which is easier? Your results hint that it is better to summarize the policy. Does this tell us something about the nature of Atari 2600 environments?    . What about performing regression on the empirical return, rather than the UCT value? Is there a sense in which one is better than the other?    . The UCT agent doesn't use "features", but these might emerge from the "policy compilation". A related idea was studied by Cobo et al. (2013) in learning features from expert data. Can something like this be investigated here as well?     . How is partial observability handled? The optimal stimuli plots hint at some of this, but it still seems an understudied question.Minor points:    . line 120: "two broad classes of approaches": this seems to suggest no other way would be valid. Can this be rephrased?     . line 136-143: it would be nice if the related work section discussed how the non-deep network features differ from the new stuff. For example: is the main distinction that the visual features are learned, or that we are using a deep architecture?    . line 136: I believe cite [4] is incorrect in the bibliography. Other cites also seemed incorrectly numbered.