Deep reinforcement learning (deep RL) research has grown significantly in recent years. A number of software offerings now exist that provide stable, comprehensive implementations for benchmarking. At the same time, recent deep RL research has become more diverse in its goals. In this paper we introduce Dopamine, a new research framework for deep RL that aims to support some of that diversity. Dopamine is open-source, TensorFlow-based, and provides compact yet reliable implementations of some state-of-the-art deep RL agents. We complement this offering with a taxonomy of the different research objectives in deep RL research. While by no means exhaustive, our analysis highlights the heterogeneity of research in the field, and the value of frameworks such as ours.
Summary:
The authors present an open-source framework TensorFlow-based named Dopamine to facilitate the task of researchers in deep reinforcement learning (deep RL). It allows to build deep RL using existing components such as reinforcement learning agents, as well as handling memory, logs and providing checkpoints for them.
Emphasis is given on providing a unified interface to these agents as well as keeping the framework generic and simple (2000 lines of code).
The framework was demonstrated on Atari games notably using Deep Q-network agents (DQN).
The authors provide numerous examples of parameter files that can be used with their framework.
Performance results are reported for some agents (DQN, C51, Rainbow, IQN).
Given the actual trends in deep learning works, unified frameworks such as that proposed is welcome.
The automatization of checkpointing for instance is particularly useful for long running experiments.
Also, trying to reduce the volume of code is beneficial for long-term maintenance and usability.
Major concerns:
* This type of contribution may not match the scope of ICLR.
* In the abstract and a large fraction of the text, the authors claim that their work is a generic reinforcement learning framework. However, the paper shows that the framework is very dependent on agents playing Atari games. Moreover, the word "Atari" comes out of nowhere on pages 2 and 5.
The authors should mention in the beginning (e.g. in the abstract) that they are handling only agents operating on Atari games.
* The positioning of the paper relative to existing approaches is unclear: state of the art is mentioned but neither discussed nor compared to the proposal.
* The format of the paper should be revised:
                - Section 5 (Related Works) should come before presenting the author's work. When reading the preceding sections, we do not know what to expect from the proposed framework.
                - All the code, especially in the appendices, seems not useful in such a paper, but rather to the online documentation of the author's framework.
* What is the motivation of the author's experiments?
                - Reproduce existing results (claimed on page 1)? Then, use the same settings as published works and show that the author's framework reaches the same level of performances.
                - Show new results (such as the effect of stickiness)? Then the authors should explicitly say that one of the contributions of the paper is to show new results.
* The authors say that they want to compare results in Figure 3. They explain why the same scale is not used. To my opinion, the authors should find a way to bring all comparisons to the same scale.
For all these reasons, I think the paper is not ready for publication at ICLR.