Recurrent World Models Facilitate Policy Evolution
A generative recurrent neural network is quickly trained in an unsupervised manner to model popular reinforcement learning environments through compressed spatiotemporal representations. The world model's extracted features are fed into compact and simple policies trained by evolution, achieving state of the art results in various environments. We also train our agent entirely inside of an environment generated by its own internal world model, and transfer this policy back into the actual environment. Interactive version of paper: https://worldmodels.github.io
Summary:  This paper proposes a new way to develop a world model for reinforcement learning. The focus is on the encoding of the visual world, coupled with a world model that learns based on the compressed representation. The world model is a recurrent version of Bishop’s (1995, neural networks book, chapter 6) mixture of gaussians network. That network outputs the weights of an MOG (using softmax), the means of the gaussians (linear outputs), and the variance (modeled as e^var, so it is a scale parameter). I had not seen a recurrent version of this network before.  The model is very simple: 0) The world is explored using a random policy with many rollouts. 1) The visual world is encoded using a VAE using the visited images gotten in the rollout, call it V. 2) The world model is trained using a recurrent NN with a five-component MOG to predict the next hidden state of V given the current hidden state. Call it M. 3) A very simple (one or two layer) controller is trained using the current hidden state of V and the current hidden state of M, using an evolution strategie method (the Covariance-Matrix Adaptation Evolution Strategy (CMA-ES), which has been shown to work well in high dimensional spaces.  The network achieves state of the art (“solves”) the OPENAI Gym CarRacingv0 task, the first system to do so.  A slightly modified network also achieves state of the art on the DoomTakeCover-v0 task, by first training V using a random policy, and then training M, but then training the controller on it’s own world model, rather than training directly on the task. I.e., the world model is iterated to produce the next hidden state of V, without ever using V, and the controller learns from that. In order for this to work, the controller also has to return a prediction of the agent’s death, as that is how rewards are calculated. This is very cool - it is like a tennis player who simulates playing in his or her mind, and improves as a result. The other difference from the previous model is that the controller not only gets h from the world model, but the c’s as well (it’s an LSTM network).   Quality:  Strengths  This is a well-written and well-reasoned paper, and the results are clear. The use of a variety of techniques from different authors shows a mastery of the field, and many approaches are discussed. As a result, the references are humongous for a NIPS paper: 106 references! This alone, while a bit over the top, creates a valuable reading list for new PhD students interested in RL.  While many of the details of the networks and training are in the supplementary material (there is also an interactive web site I did not look at), I think this is ok.   Weaknesses  In the references, about 20% of them are to the author’s own work (it’s obvious who’s work this is). Dial back that ego, guy! ;-)  Clarity:  The paper is quite clear.   Originality: The combination of techniques is novel. The work combines techniques from many researchers.   Significance: This is in some ways, a very simple model that solves a difficult task. I plan to teach it in my next iteration of my deep learning class.