Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU
We introduce a hybrid CPU/GPU version of the Asynchronous Advantage Actor-Critic (A3C) algorithm, currently the state-of-the-art method in reinforcement learning for various gaming tasks. We analyze its computational traits and concentrate on aspects critical to leveraging the GPU*s computational power. We introduce a system of queues and a dynamic scheduling strategy, potentially helpful for other asynchronous algorithms as well. Our hybrid CPU/GPU version of A3C, based on TensorFlow, achieves a significant speed up compared to a CPU implementation; we make it publicly available to other researchers at https://github.com/NVlabs/GA3C.
This paper introduces a *GPU-friendly* variant of A3C which relaxes some synchronicity constraints in the original A3C algorithm to make it more friendly to a high-throughput GPU device. The analysis of the effects of this added latency is thorough. The systems analysis of the algorithm is extensive. One caveat is that the performance figures in Table 3 are hard to compare since the protocols vary so much. I understand that DeepMind didn*t provide reproducible code for A3C, but I gather from the comment that the authors have re-implemented vanilla A3C as well, in which case it would be good to show what this reimplementation of A3C achieves in the same setting used by DM, and in the setting of the experiment conducted using GA3C (1 day). It would be good to clarify in the text that the experimental protocol differed (top 5 out of 50 vs single run), and clarify why the discrepancy, even if the answer is that the authors didn*t have time / resources to reproduce the same protocol. A bit more care would go a long way to establishing that indeed, there is no price to pay for the approximations that were made. I applaud the authors for open-sourcing the code, especially since there is a relative shortage of properly tested open-source implementations in that general area, and getting these algorithms right is non-trivial. A disclaimer: having never implemented A3C myself, I have a low confidence in my ability to appropriately assess of the algorithmic aspects of the work.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU
We introduce a hybrid CPU/GPU version of the Asynchronous Advantage Actor-Critic (A3C) algorithm, currently the state-of-the-art method in reinforcement learning for various gaming tasks. We analyze its computational traits and concentrate on aspects critical to leveraging the GPU*s computational power. We introduce a system of queues and a dynamic scheduling strategy, potentially helpful for other asynchronous algorithms as well. Our hybrid CPU/GPU version of A3C, based on TensorFlow, achieves a significant speed up compared to a CPU implementation; we make it publicly available to other researchers at https://github.com/NVlabs/GA3C.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper introduces a *GPU-friendly* variant of A3C which relaxes some synchronicity constraints in the original A3C algorithm to make it more friendly to a high-throughput GPU device. The analysis of the effects of this added latency is thorough. The systems analysis of the algorithm is extensive. One caveat is that the performance figures in Table 3 are hard to compare since the protocols vary so much. I understand that DeepMind didn*t provide reproducible code for A3C, but I gather from the comment that the authors have re-implemented vanilla A3C as well, in which case it would be good to show what this reimplementation of A3C achieves in the same setting used by DM, and in the setting of the experiment conducted using GA3C (1 day). It would be good to clarify in the text that the experimental protocol differed (top 5 out of 50 vs single run), and clarify why the discrepancy, even if the answer is that the authors didn*t have time / resources to reproduce the same protocol. A bit more care would go a long way to establishing that indeed, there is no price to pay for the approximations that were made. I applaud the authors for open-sourcing the code, especially since there is a relative shortage of properly tested open-source implementations in that general area, and getting these algorithms right is non-trivial. A disclaimer: having never implemented A3C myself, I have a low confidence in my ability to appropriately assess of the algorithmic aspects of the work.
assistant
This paper introduces a *GPU-friendly* variant of A3C which relaxes some synchronicity constraints in the original A3C algorithm to make it more friendly to a high-throughput GPU device. The analysis of the effects of this added latency is thorough. The systems analysis of the algorithm is extensive. One caveat is that the performance figures in Table 3 are hard to compare since the protocols vary so much. I understand that DeepMind didn*t provide reproducible code for A3C, but I gather from the comment that the authors have re-implemented vanilla A3C as well, in which case it would be good to show what this reimplementation of A3C achieves in the same setting used by DM, and in the setting of the experiment conducted using GA3C (1 day). It would be good to clarify in the text that the experimental protocol differed (top 5 out of 50 vs single run), and clarify why the discrepancy, even if the answer is that the authors didn*t have time / resources to reproduce the same protocol. A bit more care would go a long way to establishing that indeed, there is no price to pay for the approximations that were made. I applaud the authors for open-sourcing the code, especially since there is a relative shortage of properly tested open-source implementations in that general area, and getting these algorithms right is non-trivial. A disclaimer: having never implemented A3C myself, I have a low confidence in my ability to appropriately assess of the algorithmic aspects of the work.