Current deep neuroevolution models are usually trained in a large parameter search space for complex learning tasks, e.g. playing video games, which needs billions of samples and thousands of search steps to obtain significant performance. This raises a question of whether we can make use of sequential data generated during evolution, encode input samples, and evolve in low dimensional parameter space with latent state input in a fast and efficient manner. Here we give an affirmative answer: we train a VAE to encode input samples, then an RNN to model environment dynamics and handle temporal information, and last evolve our low dimensional policy network in latent space. We demonstrate that this approach is surprisingly efficient: our experiments on Atari games show that within 10M frames and 30 evolution steps of training, our algorithm could achieve competitive result compared with ES, A3C, and DQN which need billions of frames.
Summary: They propose "Sample Efficient Deep Neuroevolution" (SEDN) model and experiment on Atari games. In this model, they use a Variational Encoder (VAE) to encode state frames into a latent vector, and use an LSTM to encode the current latent vector and action to predict the next latent space. A policy network (trained using CMA-ES) takes the latent space, and hidden state of the RNN as an input, and outputs an action to execute.
The strengths of this paper is that it is clearly written. They even explain details of RL, background of evolution strategies, motivation of using CMA-ES (and also the algorithm itself, which is no small ordeal), so it might a good background review paper of the literature. The experimental setup is relatively easy to understand. I suggest putting Algorithm 3 before Table 1 since presenting the algorithm before results may be more natural.
That being said, there are issues with this work that needs to be addressed before publication. I will list the issues and some suggestions I have, in order to help make this work better, hopefully good enough for acceptance:
1) The authors cited [1] a few times in the paper, but actually their approach of using a VAE to compress frames into a latent, an LSTM to predict the next latent, and a CMA-ES trained network for the policy is precisely what is proposed in [1] (which had experiments that trained on the actual environment, like in this work, and also the generated environment). This paper reads like they have proposed the setup, and lacks clarity as to which parts are their contribution, and which parts are prior work, which I believe is important for a paper submitted to an academic venue. Not to say at all that there's no contribution or originally in this work - there are many, but I feel they should list out which bits are their contribution, and which bits are prior work more clearly. Doing so will make this paper and their contributions stronger.
In my opinion, their contributions are: Expanding on the approach of [1] to study on a larger set of environments (the Atari suite), where they also incorporated an iterative training loop (described Algorithm 3) that was not used in [1]. Also, unlike [1], they used a multi-layer policy network, and also explained and rationalized the intuition behind the choice of CMA-ES. I think by listing out the contributions, and separating them from previous work, the paper will be much stronger.
2) The results are not terribly strong. They achieved good results on 7 games out of 50 using 10M frames. To me, that's actually not a deal breaker, since research is not a SOTA game, but I would like to see a more detailed analysis of why the algorithm works, and when it fails so people know what future work needs to be done to address this. I'm also not convinced that using CMA-ES would have an advantage over A3C (with the same latent / hidden features going into A3C as inputs), so perhaps the author might achieve better results if A3C was used to train the policy network (or not, but would be nice to see this experiment). It would offer more insight if we know what kind of terminal scores can be achieved using this algorithm, if it were allowed to train for 1B frames like the other 2 setup. Finally, if the author was able to show that training inside a generated environment, even for pre-training before going back to the actual environment, helps sample efficiency, that would be a very interesting result to me.
I'm assigning a preliminary score of 5 for this work for now, but if the author address point (1) to my satisfaction I will revise the score to +1 points, and if the author is able to achieve much better results, or address items in point (2) to my satisfaction, I will revise the score by +1 or potentially +2 points, so the final score of this work may lie in the range of 5 -> 8. I feel the author should be able to improve the paper to get a score of 6-7 in the end, at least from me. Good luck!
[1] https://arxiv.org/abs/1803.10122