Deep Neuroevolution: Genetic Algorithms are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorithms, namely backpropagation. Evolution strategies (ES) can rival backprop-based algorithms such as Q-learning and policy gradients on challenging deep reinforcement learning (RL) problems. However, ES can be considered a gradient-based algorithm because it performs stochastic gradient descent via an operation similar to a finite-difference approximation of the gradient. That raises the question of whether non-gradient-based evolutionary algorithms can work at DNN scales. Here we demonstrate they can: we evolve the weights of a DNN with a simple, gradient-free, population-based genetic algorithm (GA) and it performs well on hard deep RL problems, including Atari and humanoid locomotion. The Deep GA successfully evolves networks with over four million free parameters, the largest neural networks ever evolved with a traditional evolutionary algorithm. These results (1) expand our sense of the scale at which GAs can operate, (2) suggest intriguingly that in some cases following the gradient is not the best choice for optimizing performance, and (3) make immediately available the multitude of neuroevolution techniques that improve performance. We demonstrate the latter by showing that combining DNNs with novelty search, which encourages exploration on tasks with deceptive or sparse reward functions, can solve a high-dimensional problem on which reward-maximizing algorithms (e.g. DQN, A3C, ES, and the GA) fail. Additionally, the Deep GA is faster than ES, A3C, and DQN (it can train Atari in {
aise.17exhbox{ }}4 hours on one workstation or {
aise.17exhbox{ }}1 hour distributed on 720 cores), and enables a state-of-the-art, up to 10,000-fold compact encoding technique.
 This paper looks at a specific implementation of a "genetic algorithm" (GA) when applied to learning Atari games. Using a black-box "random search" technique, with a few heuristic adaptations, they find performance that is roughly competitive with several other baseline methods for "Deep RL". More generally, the authors suggests that we should revisit "old" algorithms in Machine Learning and that, when we couple them with larger amounts of computation, their performance may be good. There are several things to like about this paper: - The authors place a high importance on implementation details + promise to share code. This seems to be a paper that is heavily grounded in the engineering, and I have high confidence the results can be reproduced. - The algorithm appears broadly competitive on several Atari games (although Table 1 is admittedly hard to parse). - The algorithm is generally simple, and it*s good to raise questions of baseline / what are we really accomplishing. However, there are several places where this paper falls down: - The writing/introduction is extremely loose... terms are used and introduced without proper definition for many pages. + How would be think about the venn diagram of "evolutionary strategies", "genetic algorithms", "deep Q networks", "deep RL algorithms" and "random search"... there is clearly a lot of overlap here. + The proposed deep GA has a "deep Q network" (or is it a policy... the paper does not make this clear), forms a type of "evolutionary strategy" and, at its heart is a type of "random search", but is it not also a "deep RL algorithm"? + It is not until page 3 that we get a proper definition of the algorithm, and it*s hard to keep track of the differences the authors want to highlight compared to the "baselines". + What do we gain from the claim "old algorithms work well"... gradient descent is also an old algorithm, as are seemingly all of the alternatives? Is age in-of-itself an asset? + Statements like "compression rate depends on the number of generations, but in practice is always substantial" are very loose... what does the "practice" refer to here, and why? - There is very little insight/analysis into *how* or *why* this algorithm performs better/worse than the alternatives. I don*t feel I understand if/when I should use this algorithm versus another apart from a wall of seemingly random Atari results. In fact, there is a large literature that explains why GAs give up a lot of efficiency due to their "black box" nature... what do the authors think of this? - This paper seems purely focused on results rather than insight, with many hacks/tweaks to get good results... should we believe that GA+novelty search is a general algorithm for AI, or is it just another tool in the arsenal of a research engineer? + In the end, the algorithm doesn*t actually seem to outperform state-of-the-art on these Atari baselines... so what are we supposed to take away from this. Overall, I don*t think that this paper provides very much in the way of scientific insight. Further, the results are not even superior to existing algorithms with stronger groundings. For me, this leads it to be a clear reject... even though they probably have code that reliably solved Atari games in a few hours.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Deep Neuroevolution: Genetic Algorithms are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorithms, namely backpropagation. Evolution strategies (ES) can rival backprop-based algorithms such as Q-learning and policy gradients on challenging deep reinforcement learning (RL) problems. However, ES can be considered a gradient-based algorithm because it performs stochastic gradient descent via an operation similar to a finite-difference approximation of the gradient. That raises the question of whether non-gradient-based evolutionary algorithms can work at DNN scales. Here we demonstrate they can: we evolve the weights of a DNN with a simple, gradient-free, population-based genetic algorithm (GA) and it performs well on hard deep RL problems, including Atari and humanoid locomotion. The Deep GA successfully evolves networks with over four million free parameters, the largest neural networks ever evolved with a traditional evolutionary algorithm. These results (1) expand our sense of the scale at which GAs can operate, (2) suggest intriguingly that in some cases following the gradient is not the best choice for optimizing performance, and (3) make immediately available the multitude of neuroevolution techniques that improve performance. We demonstrate the latter by showing that combining DNNs with novelty search, which encourages exploration on tasks with deceptive or sparse reward functions, can solve a high-dimensional problem on which reward-maximizing algorithms (e.g. DQN, A3C, ES, and the GA) fail. Additionally, the Deep GA is faster than ES, A3C, and DQN (it can train Atari in {
aise.17exhbox{ }}4 hours on one workstation or {
aise.17exhbox{ }}1 hour distributed on 720 cores), and enables a state-of-the-art, up to 10,000-fold compact encoding technique.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper looks at a specific implementation of a "genetic algorithm" (GA) when applied to learning Atari games. Using a black-box "random search" technique, with a few heuristic adaptations, they find performance that is roughly competitive with several other baseline methods for "Deep RL". More generally, the authors suggests that we should revisit "old" algorithms in Machine Learning and that, when we couple them with larger amounts of computation, their performance may be good. There are several things to like about this paper: - The authors place a high importance on implementation details + promise to share code. This seems to be a paper that is heavily grounded in the engineering, and I have high confidence the results can be reproduced. - The algorithm appears broadly competitive on several Atari games (although Table 1 is admittedly hard to parse). - The algorithm is generally simple, and it*s good to raise questions of baseline / what are we really accomplishing. However, there are several places where this paper falls down: - The writing/introduction is extremely loose... terms are used and introduced without proper definition for many pages. + How would be think about the venn diagram of "evolutionary strategies", "genetic algorithms", "deep Q networks", "deep RL algorithms" and "random search"... there is clearly a lot of overlap here. + The proposed deep GA has a "deep Q network" (or is it a policy... the paper does not make this clear), forms a type of "evolutionary strategy" and, at its heart is a type of "random search", but is it not also a "deep RL algorithm"? + It is not until page 3 that we get a proper definition of the algorithm, and it*s hard to keep track of the differences the authors want to highlight compared to the "baselines". + What do we gain from the claim "old algorithms work well"... gradient descent is also an old algorithm, as are seemingly all of the alternatives? Is age in-of-itself an asset? + Statements like "compression rate depends on the number of generations, but in practice is always substantial" are very loose... what does the "practice" refer to here, and why? - There is very little insight/analysis into *how* or *why* this algorithm performs better/worse than the alternatives. I don*t feel I understand if/when I should use this algorithm versus another apart from a wall of seemingly random Atari results. In fact, there is a large literature that explains why GAs give up a lot of efficiency due to their "black box" nature... what do the authors think of this? - This paper seems purely focused on results rather than insight, with many hacks/tweaks to get good results... should we believe that GA+novelty search is a general algorithm for AI, or is it just another tool in the arsenal of a research engineer? + In the end, the algorithm doesn*t actually seem to outperform state-of-the-art on these Atari baselines... so what are we supposed to take away from this. Overall, I don*t think that this paper provides very much in the way of scientific insight. Further, the results are not even superior to existing algorithms with stronger groundings. For me, this leads it to be a clear reject... even though they probably have code that reliably solved Atari games in a few hours.
assistant
 This paper looks at a specific implementation of a "genetic algorithm" (GA) when applied to learning Atari games. Using a black-box "random search" technique, with a few heuristic adaptations, they find performance that is roughly competitive with several other baseline methods for "Deep RL". More generally, the authors suggests that we should revisit "old" algorithms in Machine Learning and that, when we couple them with larger amounts of computation, their performance may be good. There are several things to like about this paper: - The authors place a high importance on implementation details + promise to share code. This seems to be a paper that is heavily grounded in the engineering, and I have high confidence the results can be reproduced. - The algorithm appears broadly competitive on several Atari games (although Table 1 is admittedly hard to parse). - The algorithm is generally simple, and it*s good to raise questions of baseline / what are we really accomplishing. However, there are several places where this paper falls down: - The writing/introduction is extremely loose... terms are used and introduced without proper definition for many pages. + How would be think about the venn diagram of "evolutionary strategies", "genetic algorithms", "deep Q networks", "deep RL algorithms" and "random search"... there is clearly a lot of overlap here. + The proposed deep GA has a "deep Q network" (or is it a policy... the paper does not make this clear), forms a type of "evolutionary strategy" and, at its heart is a type of "random search", but is it not also a "deep RL algorithm"? + It is not until page 3 that we get a proper definition of the algorithm, and it*s hard to keep track of the differences the authors want to highlight compared to the "baselines". + What do we gain from the claim "old algorithms work well"... gradient descent is also an old algorithm, as are seemingly all of the alternatives? Is age in-of-itself an asset? + Statements like "compression rate depends on the number of generations, but in practice is always substantial" are very loose... what does the "practice" refer to here, and why? - There is very little insight/analysis into *how* or *why* this algorithm performs better/worse than the alternatives. I don*t feel I understand if/when I should use this algorithm versus another apart from a wall of seemingly random Atari results. In fact, there is a large literature that explains why GAs give up a lot of efficiency due to their "black box" nature... what do the authors think of this? - This paper seems purely focused on results rather than insight, with many hacks/tweaks to get good results... should we believe that GA+novelty search is a general algorithm for AI, or is it just another tool in the arsenal of a research engineer? + In the end, the algorithm doesn*t actually seem to outperform state-of-the-art on these Atari baselines... so what are we supposed to take away from this. Overall, I don*t think that this paper provides very much in the way of scientific insight. Further, the results are not even superior to existing algorithms with stronger groundings. For me, this leads it to be a clear reject... even though they probably have code that reliably solved Atari games in a few hours.