Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorithms, namely backpropagation. Evolution strategies (ES) can rival backprop-based algorithms such as Q-learning and policy gradients on challenging deep reinforcement learning (RL) problems. However, ES can be considered a gradient-based algorithm because it performs stochastic gradient descent via an operation similar to a finite-difference approximation of the gradient. That raises the question of whether non-gradient-based evolutionary algorithms can work at DNN scales. Here we demonstrate they can: we evolve the weights of a DNN with a simple, gradient-free, population-based genetic algorithm (GA) and it performs well on hard deep RL problems, including Atari and humanoid locomotion. The Deep GA successfully evolves networks with over four million free parameters, the largest neural networks ever evolved with a traditional evolutionary algorithm. These results (1) expand our sense of the scale at which GAs can operate, (2) suggest intriguingly that in some cases following the gradient is not the best choice for optimizing performance, and (3) make immediately available the multitude of neuroevolution techniques that improve performance. We demonstrate the latter by showing that combining DNNs with novelty search, which encourages exploration on tasks with deceptive or sparse reward functions, can solve a high-dimensional problem on which reward-maximizing algorithms (e.g. DQN, A3C, ES, and the GA) fail. Additionally, the Deep GA is faster than ES, A3C, and DQN (it can train Atari in ∼4 hours on one workstation or ∼1 hour distributed on 720 cores), and enables a state-of-the-art, up to 10,000-fold compact encoding technique.
Post-rebuttal update: The review process has identified several issues such as missing citations and lack of clarity with respect to aims of the paper. Although the authors have failed to update the paper within the rebuttal period, their responses show an understanding of the issues that need to be addressed as well as a broad appreciation of work in EC that would be included in a final version, making it a useful resource for the wider ML community. On top of this they will include an even larger amount of empiricial data from the experiments they have already run, which is a valuable resource considering the amount of compute needed to obtain this data.
---
The current landscape of reinforcement learning - particularly in domains with high-dimensional structured input spaces such as images or text - relies heavily on backpropagation-based reinforcement learning algorithms.  An alternative that has re-emerged is ES, due to its simplicity and scalability. However, ES can also be considered a gradient-based method. In this paper, the authors apply a similar treatment to GAs, another simple method at its most basic. The authors claim to have 3 main contributions: extending the scale to which GAs can operate, suggesting that gradient-based methods may not achieve the best performance, and making available a vast array of techniques available from the EC literature; they demonstrate the latter by utilising novelty search (NS).
In my opinion the authors do indeed have a valuable contribution in a) demonstrating that a simple GA can successfully be applied to larger networks than was previously thought to be possible and b) introducing a novel software implementation that allows GAs to be efficiently scaled/distributed (similar in nature to the work of Salimans et al.). This is by itself valuable, as, along with recent work on ES, it potentially extends the range of problems that are perhaps best tackled using black-box optimisation techniques. Going against the prevailing trends in order to investigate alternative methods is an underappreciated service to the community, and I believe the evaluation of the methods and the choice of comparative methods to be just about satisfactory. As exemplified by NS, there is a wealth of techniques from the EC literature that could be applied to many topical problems, and the authors' main contributions opens up the road for this.
A lot of care has been put into evaluation on Atari games. The details in the main paper and supplementary material, with, e.g., clear definitions of "frames", make me believe that fair comparisons have taken place. All methods in the table, including GAs, perform best at some games (except for RS, which is a necessary baseline for GAs). It would be better to provide more data points that relate to prior works - such as scores at 200M frames to evaluate sample complexity (indeed, the authors note that good solutions can be found by GAs within a few generations, so it would be best to tabulate this) and at ~ 4d to evaluate wall-clock time (is it possible to push performance even further?). Since the GA presented is very rudimentary, I consider the baselines in the main paper to be reasonable, but it would be misleading to not present newer work. The authors do so in the supplementary material, and it is promising to note that the GA still achieves state-of-the-art performance in a few games even when compared to the most sophisticated/distributed state-of-the-art DRL algorithms developed in a concentrated effort over the last few years. Despite having an NS variant, it is a shame that the authors did not show that this could potentially improve performance on Atari, when BCs such as the game RAM or preferably random CNN features are easily available.
The authors also evaluate on a maze that is a staple task in the EC literature to demonstrate the power of NS. While the results are unsurprising, it is a reasonable sanity check. The final evaluation is on a difficult continuous control task, in which GAs solve the task, but have much poorer sample complexity than ES. Given the range of continuous control tasks used to benchmark RL algorithms nowadays, the authors would do well to present results across more of these tasks. Again, NS was not evaluated on this task.
A major weakness of this paper is the presentation. The authors discuss some interesting findings, but would be better served by being more concise and focused. In particular, the emphasis should be more on showcasing quantitative results. Doing so, with more continuous control tasks, would make the claims of this paper more substantiated.