SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
We present a modern scalable reinforcement learning agent called SEED (Scalable, Efficient Deep-RL). By effectively utilizing modern accelerators, we show that it is not only possible to train on millions of frames per second but also to lower the cost. of experiments compared to current methods. We achieve this with a simple architecture that features centralized inference and an optimized communication layer. SEED adopts two state-of-the-art distributed algorithms, IMPALA/V-trace (policy gradients) and R2D2 (Q-learning), and is evaluated on Atari-57, DeepMind Lab and Google Research Football. We improve the state of the art on Football and are able to reach state of the art on Atari-57 twice as fast in wall-time. For the scenarios we consider, a 40% to 80% cost reduction for running experiments is achieved. The implementation along with experiments is open-sourced so results can be reproduced and novel ideas tried out.
The paper proposes a new reinforcement learning agent architecture which is significantly faster and way less costly than previously distributed architectures. To this end, the paper proposes a new architecture that utilizes modern accelerators more efficiently. The paper reads very well and the experimental results indeed demonstrate improvement. Nevertheless, even though working in deep learning for years and have also some experience with Reinforcement learning I am not in the position to provide an expert judgment on the novelty of the work. I do not know if ICLR is the right place of the paper (I would probably suggest a system architectures conference for better assessment of the work).
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
We present a modern scalable reinforcement learning agent called SEED (Scalable, Efficient Deep-RL). By effectively utilizing modern accelerators, we show that it is not only possible to train on millions of frames per second but also to lower the cost. of experiments compared to current methods. We achieve this with a simple architecture that features centralized inference and an optimized communication layer. SEED adopts two state-of-the-art distributed algorithms, IMPALA/V-trace (policy gradients) and R2D2 (Q-learning), and is evaluated on Atari-57, DeepMind Lab and Google Research Football. We improve the state of the art on Football and are able to reach state of the art on Atari-57 twice as fast in wall-time. For the scenarios we consider, a 40% to 80% cost reduction for running experiments is achieved. The implementation along with experiments is open-sourced so results can be reproduced and novel ideas tried out.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper proposes a new reinforcement learning agent architecture which is significantly faster and way less costly than previously distributed architectures. To this end, the paper proposes a new architecture that utilizes modern accelerators more efficiently. The paper reads very well and the experimental results indeed demonstrate improvement. Nevertheless, even though working in deep learning for years and have also some experience with Reinforcement learning I am not in the position to provide an expert judgment on the novelty of the work. I do not know if ICLR is the right place of the paper (I would probably suggest a system architectures conference for better assessment of the work).
assistant
The paper proposes a new reinforcement learning agent architecture which is significantly faster and way less costly than previously distributed architectures. To this end, the paper proposes a new architecture that utilizes modern accelerators more efficiently. The paper reads very well and the experimental results indeed demonstrate improvement. Nevertheless, even though working in deep learning for years and have also some experience with Reinforcement learning I am not in the position to provide an expert judgment on the novelty of the work. I do not know if ICLR is the right place of the paper (I would probably suggest a system architectures conference for better assessment of the work).