Evolution-Guided Policy Gradient in Reinforcement Learning
Deep Reinforcement Learning (DRL) algorithms have been successfully applied to a range of challenging control tasks. However, these methods typically suffer from three core difficulties: temporal credit assignment with sparse rewards, lack of effective exploration, and brittle convergence properties that are extremely sensitive to hyperparameters. Collectively, these challenges severely limit the applicability of these approaches to real-world problems. Evolutionary Algorithms (EAs), a class of black box optimization techniques inspired by natural evolution, are well suited to address each of these three challenges. However, EAs typically suffer from high sample complexity and struggle to solve problems that require optimization of a large number of parameters. In this paper, we introduce Evolutionary Reinforcement Learning (ERL), a hybrid algorithm that leverages the population of an EA to provide diversified data to train an RL agent, and reinserts the RL agent into the EA population periodically to inject gradient information into the EA. ERL inherits EA's ability of temporal credit assignment with a fitness metric, effective exploration with a diverse set of policies, and stability of a population-based approach and complements it with off-policy DRL's ability to leverage gradients for higher sample efficiency and faster learning. Experiments in a range of challenging continuous control benchmarks demonstrate that ERL significantly outperforms prior DRL and EA methods.
In this paper, the authors combine an evolutionary algorithm with a Deep RL algorithm so that the combination achieves the best of both worlds. It is successfully applied to 6 standard mujoco benchmarks.  The following part of this review has been edited based on the other reviewers points, the authors rebuttal, an attempt to reproduce the results of the authors,  and investigations in their code.  First, investigations of the results and code revealed several points that are poorly documented in the paper. Here is a summary:  * DDPG:     - The hidden layer sizes in both the Actor and the Critic in the implementation are different from those mentioned in the paper. This is a problem since ddpg's performance can vary a lot with the architecture of the nets.     - The Actor uses tanh non-linearities, and the  Critic elu non-linearties. This is not mentioned.     - The norm of the gradient of the Critic is clipped. This is not mentioned.  * Evolutionary Algorithm:     - Tournament selection is performed with tournament size of 3, and is done with replacement.     - Individuals selected during tournament selection produce offspring with the elite through crossover until the population is filled.     - The way mutations are handled is more complex that what is mentioned in the paper and involves many hyper-parameters. For each non elitist individual there is a fixed probability for his genome to mutate. If the genome      mutates, then each weight of the actor can mutate in 3 different ways. "Normal" mutations involve adding a 10% Gaussian noise; "Super" mutations involve adding 100% Gaussian noise, and "Reset" mutations involve resetting the weight using a normalized center Gaussian noise. It seems that on average, around 10% of the weights of the Actor that undergoes a mutation are changed, but some parts of the code are still obscure. The whole mutation process is really messy and deserves more attention since it is supposed to account for half of the success of the method (the other half being the deep RL part).   The authors should definitely put forward all the above facts in their paper as some of them play an important role in the performance of their system.  About performance itself, in agreement with other reviewers, I consider that the authors should not base their paper on state-of-the-art performance claims, but rather on the simplicity and conceptual efficiency of their approach with respect to alternatives. I also agree with other reviewers that the title is too generic and that the approach should be better positionned with respect to the related literature (Baldwin effect etc.).  For the rest, my opinion has not changed much so I keep the rest of the review mostly as is.  The idea is simple, the execution is efficient, and the results are compelling (but see the remarks about reproducibility above). I am in favor of accepting this paper as it provides a useful contribution. Below I insist on weaknesses to help the authors improve their paper.  A first point is a lack of clarity about deceptive reward signals or gradients. The authors state that the hard inverted pendulum is deceptive, but they don't explain what they mean, though this matters a lot for the main message of the paper. Indeed, ERL is supposed to improve over EA because it incorporates gradient-based information that should speed up convergence. But if this gradient-based information is deceptive, there should be no speed up, in contradiction with results of ERL in "deceptive reward" tasks. I would be glad to see a closer examination of ERL's behaviour in the context of a truly deceptive gradient task: does it reduce to the corresponding EA, i.e. does it consistently reject the deep RL policy until a policy that is close enough to the optimum has been found? In that respect, the authors should have a look at the work of Colas et al. at ICML 2018, which is closely related to theirs, and where the effect of deceptive gradients on deep RL is discussed in more details.  Related to the above, details are missing about the tasks. In the standard inverted pendulum, is the action discrete in {-1,0,1} or continous in [-1,1] or something else? What is the immediate reward signal? Does it incorporate something to favor smaller actions? Is there a "success" state that may stop a trial before 1000 steps? All these details may make a difference in the results. The same kind of basic facts should also be given about the other benchmarks, and the corresponding details could be rejected into the appendices.  The methodology for reported metrics has nice features, but the study could be made more rigorous with additional information: how many seeds did you use? How do you report variance? Can you say something about the statistical significance of your results?  The third weakness is in the experimental results when reading Fig. 3. For instance, results of DDPG on half-cheetah seem to be lower than results published in the literature (see e.g. Henderson et al. "Deep RL that matters"). Could the authors investigate why (different implementation? Insufficient hyper-parameter tuning... ?)  The fourth weakness is the related work section. First, the authors should read more about Learning Classifier Systems (LCSs), which indeed combined EAs and RL from the very start (Holland's 1975 work). The authors may read Lanzi, Butz or Wilson's papers about LCSs to take more distance about that. They should also take more distance about using EAs to obtain RL algorithms, which is now an important trend in Meta-RL at the moment. More generally, in a NIPS paper, we expect the "related work" section to provide a good overview of the many facets of the domain, which is not the case here. Finally, the last sentence "These approaches are complementary to the ERL framework and can be readily combined for potential further improved performance." is very vague and weak. What approach do the authors want to combine theirs with and how can it be "readily combined"? The authors must be much more specific here.  In Section 2.2 it should be stated more clearly which EA is used exactly. There are many families, most algorithms have names, from the section we just get that the authors use an EA. Later on, more details about mutation etc. are missing. For instance, how do they initialize the networks? How many weights are perturbed during mutation?  The authors may also explain why they are not using an ES, as used in most neuroevolution papers involved in the competition to deep RL (see numerous "Uber labs deep-neuroevolution papers"), why not NEAT, etc. Actually, it took me a while to realize that an ES cannot be used here, because there is no way to guarantee that the deep RL policy will comply with the probability distribution corresponding to the covariance matrix of the ES current population. Besides, ESs also perform a form of approximate gradient descent, making the advantage of using deep RL in addition less obvious. All this could be discussed.  I would also be glad to see a "curriculum learning" aspect: does ERL start using EAs a lot, and accepts the deep RL policy more and more along time once the gradient is properly set?  Note also that the authors may include policies obtained from imitation within their framework without harm. This would make it even richer.  Finally, the paper would be stronger if performance was compared to recent state-of-the-art algorithms such as SAC (Haarnoja), TD3 (Fujimoto) or D4PG (Horgan), but I do not consider this as mandatory...  More local points.  l. 52: ref [19] has nothing to do with what is said, please pick a better ref.  l. 195-201: this is a mere repetition of lines 76-81. Probably with a better view of related work, you can get a richer intro and avoid this repetition.  p.2: "However, each of these techniques either rely on complex supplementary structures or introduce sensitive parameters that are task-specific": actually, the author's method also introduce supplementary structures (the EA/the deep RL part), and they also have task-specific parameters...  The authors keep a standard replay buffer of 1e^6 samples. But the more actors, the faster the former content is washed out. Did the authors consider increasing the size of the replay buffer with the number of agents? Any thoughts or results about this would be welcome.  l.202 sq. (diverse exploration): the fact that combining parameter noise exploration with action exploration "collectively lead to an effective exploration strategy" is not obvious and needs to be empirically supported, for instance using ablative studies about both forms of exploration.  typos:  I would always write "(whatever)-based" rather than "(whatever) based" (e;g. gradient-based). Google seems to agree with me.  l. 57: suffer with => from l. 108: These network(s) l. 141: gradients that enable(s) l. 202: is used (to) generate l. 217: minimas => minima (latin plural of minimum) l. 258: favorable => favorably? l.266: a strong local minima => minimum (see above ;))  In ref [55] baldwin => Baldwin.