Action-Conditional Video Prediction using Deep Networks in Atari Games
Motivated by vision-based reinforcement learning (RL) problems, in particular Atari games from the recent benchmark Aracade Learning Environment (ALE), we consider spatio-temporal prediction problems where future image-frames depend on control variables or actions as well as previous frames. While not composed of natural scenes, frames in Atari games are high-dimensional in size, can involve tens of objects with one or more objects being controlled by the actions directly and many other objects being influenced indirectly, can involve entry and departure of objects, and can involve deep partial observability. We propose and evaluate two deep neural network architectures that consist of encoding, actionconditional transformation, and decoding layers based on convolutional neural networks and recurrent neural networks. Experimental results show that the proposed architectures are able to generate visually-realistic frames that are also useful for control over approximately 100-step action-conditional futures in some games. To the best of our knowledge, this paper is the first to make and evaluate long-term predictions on high-dimensional video conditioned by control inputs.
The authors present a system for video frame prediction in ATARI games taking into account the game actions. The main novelty is a multiplicative transformation layer that selects weights based on a given action vector. To keep the system scalable, a factorization of the weight tensor is used. Encoding and decoding of images is done by a well-established convolutional network architecture.They propose two different architectures: a feed-forward network that gets a fixed number of previous frames as input and a recurrent network that gets only one frame but has an LSTM layer before the transformation layer. (I think at least once in the paper they should say what's an LSTM and what the abbreviation stands for.)The qualitative assessment of the generated frames as shown by the supplementary videos is convincing. A quantitative evaluation of mean squared pixel error is given in a clear way but it is not surprising that their system beats a linear and a nonlinear predictor that don't take into account the game actions.Another experiment in which the generated frames are used instead of the real frames as input for a DQN agent also shows results as expected. The performance is worse than for real input frames, better that random play, and their system beats the no-action predictors.An interesting application of this system is to use it during training of an agent to improve exploration. They show that if a DQN agent takes an action that leads to a predicted frame that has least similarity with previously seen frames (rather than a random action), the final performance is significantly improved on some games. Finally it is shown that the system can be used to automatically analyze some of the game dynamics. Game actions that have similar effects can be identified from similarities in the transformation weight matrix. Furthermore the system estimates which pixels of the image are controlled directly by actions and which pixels are uncontrollable game dynamics.The authors write that "To the best of our knowledge, this paper is the first to make and evaluate long-term predictions on high-dimensional video conditioned by control inputs." This may be true, because to my knowledge previous such systems did not have LSTM. I know, however, that there is an old paper by Schmidhuber and Huber from 1991 (International Journal of Neural Systems) where a similar neural predictor learns to predict the next visual input frame of a fovea, given previous input and action. The predictor is embedded in a reinforcement learning system that learns sequences of saccades that lead to desired visual objects defined in a separate goal input. Schmidhuber also had papers at IJCNN 1990 and NIPS 1991 where both the predictor and the action generator were recurrent (but no LSTM yet). Nevertheless, it would be good if the authors could point out what's different in their system, which for example also uses a different RL method.The results are clear, the paper is well structured and suggested for acceptance after minor revisions as indicated above.