Learning world dynamics has recently been investigated as a way to make reinforcement learning (RL) algorithms to be more sample efficient and interpretable. In this paper, we propose to capture an environment dynamics with a novel forward model that leverages recent works on adversarial learning and visual control. Such a model estimates future observations conditioned on the current ones and other input variables such as actions taken by an RL-agent. We focus on image generation which is a particularly challenging topic but our method can be adapted to other modalities. More precisely, our forward model is trained to produce realistic observations of the future while a discriminator model is trained to distinguish between real images and the model's prediction of the future. This approach overcomes the need to define an explicit loss function for the forward model which is currently used for solving such a class of problem. As a consequence, our learning protocol does not have to rely on an explicit distance such as Euclidean distance which tends to produce unsatisfactory predictions. To illustrate our method, empirical qualitative and quantitative results are presented on a real driving scenario, along with qualitative results on Atari game Frostbite.
This paper describes an approach for training conditional future frame prediction models, where the conditioning is with respect to the current frame and additional inputs - specifically actions performed in a reinforcement learning (RL) setting. 
The authors suggest that one can predict future frames from a vector comprised of an observation encoding and an action. To train the model, they suggest using a linear combination of three different losses: (1) an adversarial loss that encourages the generated sample to look similarly to training data, (2) an InfoGAN-inspired loss that is supposed to maximise mutual information between the conditioning (e.g. action) and the generated sample, and (3) a content loss, taken to be the mean-squared error of the prediction and ground-truth in the VGG feature space.
The major contribution of this work seems to be using these three losses in conjunction, while doing conditional frame prediction at the same time. While interesting, there exist very similar approaches that also use adversarial losses [1] as well as approaches using different means to reach the same goal [2, 3]. None of these are mentioned in the text, nor evaluated against. It is true that [1] is not action-conditional, but adding actions as conditioning could be a simple extension.
Experimental section consists of an ablation study, which evaluates importance of different components of the loss, and a qualitative study of model predictions. With no comparison to state of the art (e.g. [1, 3]), it is hard to gauge how valuable this particular approach is. 
The qualitative evaluation starts with §4.4¶1 “we follow the customary GAN literature to include some qualitative results for illustration”, as if there was no other reason for including samples than to follow the custom. Since the paper is about action-conditional prediction, it would be interesting to see predictions conditioned on the same initial sequence but different actions, which are not present, however. Moreover, this work is developed in the context of RL applications, and since prior art [4] has shown that better predictive models do not necessarily lead to better RL results, it would be interesting to evaluate the proposed approach against baselines in an RL setting.
The paper is clearly written, but some claims in the text are not supported by any citations (e.g. §1¶2 “More recently, several papers have shown that forward modelling…” without a citation).  Some claims are misleading (e.g. §1¶3 says that by using adversarial training we don’t need to use task-specific losses and it does not put constraints on input modality. While true, using MSE loss is equally general). Some other claims are not supported at all or may not be true (e.g. §3.2¶1 “ResNet … aims at compressing the information in the raw observation” - to the best of my knowledge, there is no evidence for this).
To conclude, the suggested approach is not novel, the experimental evaluation is lacking, and the text contains a number of unsupported statements. I recommend to reject this paper.
[1] Lee, A.X., Zhang, R., Ebert, F., Abbeel, P., Finn, C., & Levine, S. (2018). Stochastic Adversarial Video Prediction. CoRR, abs/1804.01523.
[2] Eslami, S.M., Rezende, D.J., Besse, F., Viola, F., Morcos, A.S., Garnelo, M., Ruderman, A., Rusu, A.A., Danihelka, I., Gregor, K., Reichert, D.P., Buesing, L., Weber, T., Vinyals, O., Rosenbaum, D., Rabinowitz, N.C., King, H., Hillier, C., Botvinick, M.M., Wierstra, D., Kavukcuoglu, K., & Hassabis, D. (2018). Neural scene representation and rendering. Science, 360, 1204-1210.
[3] Denton, E.L., & Fergus, R. (2018). Stochastic Video Generation with a Learned Prior. ICML.
[4] Buesing, L., Weber, T., Racanière, S., Eslami, S.M., Rezende, D.J., Reichert, D.P., Viola, F., Besse, F., Gregor, K., Hassabis, D., & Wierstra, D. (2018). Learning and Querying Fast Generative Models for Reinforcement Learning. CoRR, abs/1802.03006.