Genetic-Gated Networks for Deep Reinforcement Learning
We introduce the Genetic-Gated Networks (G2Ns), simple neural networks that combine a gate vector composed of binary genetic genes in the hidden layer(s) of networks. Our method can take both advantages of gradient-free optimization and gradient-based optimization methods, of which the former is effective for problems with multiple local minima, while the latter can quickly find local minima. In addition, multiple chromosomes can define different models, making it easy to construct multiple models and can be effectively applied to problems that require multiple models. We show that this G2N can be applied to typical reinforcement learning algorithms to achieve a large improvement in sample efficiency and performance.
The authors proposed the Genetic-Gated Networks (G2Ns) which combine conventional gradient based policy optimization and genetic algorithms. The gates are binary masks injected to the conventional policy neural networks, and their values are adapted via genetic algorithms while learning the policies. Specifically, a population of differently configured gene vectors are initially generated, and the elite gene configuration which has the highest fitness score is selected in order to update the conventional neural network parameters via gradient based policy optimization. The gate values are updated via genetic algorithms periodically as well. The G2Ns are applied to the actor policy of Advantage Actor-Critic (G2AC) on Atari 2600 games and Proximal Policy Optimization (G2PPO) on eight MuJoCo environments. The empirical results show that G2AC achieved improved results on 39 out of 50 Atari games and G2PPO achieved improvement on some MuJoCo environments as well.   The G2Ns are general neural network framework which could be applied to either supervised learning or reinforcement learning. The reinforcement learning related aspect of G2N is in the Two-way Synchronous Optimization part. In the second phase of the two-way synchronous optimization, multiple policies with different gene configuration are leveraged to collect training trajectories. The empirical ablation studies on the two-way synchronous optimization in section 4.3 are very helpful in justifying the positive impact of such approach. The multiple policy-based exploration of G2Ns is related to the ideas of exploration via bootstrapped DQN or noisy networks. An in-depth analysis would also be helpful to highlight the uniqueness of G2Ns on exploration in general.   The G2Ns achieve significant improvement on discrete action problems as it is shown that the final performance on 39 out of 50 Atari games are improved. Some sample plots of learning curves on the Atari games would also be helpful in checking that they do not suffer from the poor initial performance issue as reported in the continuous domains. Given the plots in Figure 5 on the four Atari games, it seems that G2AC is outperformed by A2C in the first 10M training, which suggests the poor initial performance issue may be an inherent issue of G2Ns, not specific to the continuous reinforcement learning domains. Any additional material on this matter would be helpful.   The improvement of G2PPO in continuous domains is less significant compared to the G2AC in discrete domains as most of the curves are mixed in Figure 4.   It is curious to know the hyper-parameter sensitivity of G2Ns to the initial activation probability, the crossover probability and the mutation probability.    The paper is not related to multiagent RL at all. The term “multi-agent” is misused.   ----  I have read the reviews and the rebuttal. The rebuttal did not change my assessment of the paper. I would like to keep my score.