Published as a conference paper at ICLR 2018 POLICY OPTIMIZATION BY GENETIC DISTILLATION
Genetic algorithms have been widely used in many practical optimization problems. Inspired by natural selection, operators, including mutation, crossover and selection, provide effective heuristics for search and black-box optimization. However, they have not been shown useful for deep reinforcement learning, possibly due to the catastrophic consequence of parameter crossovers of neural networks. Here, we present Genetic Policy Optimization (GPO), a new genetic algorithm for sample-efficient deep policy optimization. GPO uses imitation learning for policy crossover in the state space and applies policy gradient methods for mutation. Our experiments on MuJoCo tasks show that GPO as a genetic algorithm is able to provide superior performance over the state-of-the-art policy gradient methods and achieves comparable or higher sample efficiency.
This is a highly interesting paper that proposes a set of methods that combine ideas from imitation learning, evolutionary computation and reinforcement learning in a novel way. It combines the following ingredients:
a) a population-based setup for RL
b) a pair-selection and crossover operator
c) a policy-gradient based “mutation” operator
d) filtering data by high-reward trajectories
e) two-stage policy distillation
In its current shape it has a couple of major flaws (but those can be fixed during the revision/rebuttal period):
(1) Related work. It is presented in a somewhat ahistoric fashion. In fact, ideas for evolutionary methods applied to RL tasks have been widely studied, and there is an entire research field called “neuroevolution” that specifically looks into which mutation and crossover operators work well for neural networks. I’m listing a small selection of relevant papers below, but I’d encourage the authors to read a bit more broadly, and relate their work to the myriad of related older methods. Ideally, a more reasonable form of parameter-crossover (see references) could be compared to -- the naive one is too much of a straw man in my opinion. To clarify: I think the proposed method is genuinely novel, but a bit of context would help the reader understand which aspects are and which aspects aren’t.
(2) Ablations. The proposed method has multiple ingredients, and some of these could be beneficial in isolation: for example a population of size 1 with an interleaved distillation phase where only the high-reward trajectories are preserved could be a good algorithm on its own. Or conversely, GPO without high-reward filtering during crossover. Or a simpler genetic algorithm that just preserves the kills off the worst members of the population, and replaces them by (mutated) clones of better ones, etc. 
(3) Reproducibility. There are a lot of details missing; the setup is quite complex, but only partially described. Examples of missing details are: how are the high-reward trajectories filtered? What is the total computation time of the different variants and baselines? The x-axis on plots, does it include the data required for crossover/Dagger? What are do the shaded regions on plots indicate? The loss on \pi_S should be made explicit. An open-source release would be ideal.
Minor points:
- naively, the selection algorithm might not scale well with the population size (exhaustively comparing all pairs), maybe discuss that?
- the filtering of high-reward trajectories is what estimation of distribution algorithms [2] do as well, and they have a known failure mode of premature convergence because diversity/variance shrinks too fast. Did you investigate this?
- for Figure 2a it would be clearer to normalize such that 1 is the best and 0 is the random policy, instead of 0 being score 0.
- the language at the end of section 3 is very vague and noncommittal -- maybe just state what you did, and separately give future work suggestions?
- there are multiple distinct metrics that could be used on the x-axis of plots, namely: wallclock time, sample complexity, number of updates. I suspect that the results will look different when plotted in different ways, and would enjoy some extra plots in the appendix. For example the ordering in Figure 6 would be inverted if plotting as a function of sample complexity?
- the A2C results are much worse, presumably because batchsizes are different? So I’m not sure how to interpret them: should they have been run for longer? Maybe they could be relegated to the appendix?
References:
[1] Gomez, F. J., & Miikkulainen, R. (1999). Solving non-Markovian control tasks with neuroevolution.
[2] Larranaga, P. (2002). A review on estimation of distribution algorithms.
[3] Stanley, K. O., & Miikkulainen, R. (2002). Evolving neural networks through augmenting topologies. 
[4] Igel, C. (2003). Neuroevolution for reinforcement learning using evolution strategies.
[5] Hausknecht, M., Lehman, J., Miikkulainen, R., & Stone, P. (2014). A neuroevolution approach to general atari game playing.
[6] Gomez, F., Schmidhuber, J., & Miikkulainen, R. (2006). Efficient nonlinear control through neuroevolution.
Pros:
- results
- novelty of idea
- crossover visualization, analysis
- scalability
Cons:
- missing background
- missing ablations
- missing details
[after rebuttal: revised the score from 7 to 8]