LEARNING IMPROVED DYNAMICS MODEL IN RE- INFORCEMENT LEARNING BY INCORPORATING THE LONG TERM FUTURE
In model-based reinforcement learning, the agent interleaves between model learning and planning. These two components are inextricably intertwined. If the model is not able to provide sensible long-term prediction, the executed planner would exploit model flaws, which can yield catastrophic failures. This paper focuses on building a model that reasons about the long-term future and demonstrates how to use this for efficient planning and exploration. To this end, we build a latent-variable autoregressive model by leveraging recent ideas in variational inference. We argue that forcing latent variables to carry future information through an auxiliary task substantially improves long-term predictions. Moreover, by planning in the latent space, the planner's solution is ensured to be within regions where the model is valid. An exploration strategy can be devised by searching for unlikely trajectories under the model. Our method achieves higher reward faster compared to baselines on a variety of tasks and environments in both the imitation learning and model-based reinforcement learning settings.
After the rebuttal and the authors providing newer experimental results, I've increased my score. They have addressed both the issue with the phrasing of the auxiliary loss, which I'm very happy they did as well as provided more solid experimental results, which in my opinion make the paper strong enough for publication. 
#####
The paper proposes a variational framework for learning a Model of both the environment and the actor's policy in Reinforcement Learning. Specifically, the model is a deterministic RNN which at every step takes as input also a new stochastic latent variable z_t. Compared to more standard approaches, the prior over z_t is not standard normal but depends on the previously hidden state. The inference model combines information from the forward generative hidden state and a backward RNN that looks only at future observations. Finally, an auxiliary loss is added to the model that tries to predict the future states of the backward RNN using the latent variable z_t.  The idea of the paper is quite well presented and concise. 
The paper tests the proposed framework on several RL benchmarks. Using it for imitation learning outperforms two baseline models: behaviour cloning and behaviour cloning trained with an auxiliary loss of predicting the next observation. Although the results are good, it would have been much better if there was also a comparison against a Generative model (identical to the one proposed) without the auxiliary loss added? The authors claim that the results of the experiment suggest that the auxiliary loss is indeed helping, where I find the evidence unconvincing given that there is no comparison against this obvious baseline. Extra comparison against the method from [1] or GAIL would make the results even stronger, but it is understandable that one can not compare against everything, hence I do not see this as a major issue. 
The authors also compare on long-horizon video prediction. Although their method outperforms the method proposed in Ha & Schmidhuber, this by no means suggests that the method is really that superior. I would argue that in terms of future video prediction that [3] provides significantly better results than the World Models, nevertheless, at least one more baseline would have supported the authors claims much better. 
On the Model-Based planning, the authors outperform SeCTAR model on the BabyAI tasks and the Wheeled locomotion. This result is indeed interesting and shows that the method is viable for planning. However, given that similar result has been shown in [1] regarding the planning framework it is unclear how novel the result is. 
In conclusion, the paper presents a generative model for training a model-based approach with an auxiliary loss. The results look promising, however, stronger baselines and better ablation of how do different components actually contribute would make the paper significantly stronger than it is at the moment. Below are a few further comments on some specific parts of the paper. 
A few comments regarding relevant literature: 
Both in the introduction and during the main text the authors have not cited [1] which I think is a very closely related method. In this work similarly, a generative model of future segments is learned using a variational framework. In addition, the MPC procedure that the authors present in this paper is not novel, but has already been proposed and tried in [1] - optimizing over the latent variables rather than the actions directly, and there have been named Latent Action Priors. 
The data gathering process is also not a new idea and using the error in a dynamics model for exploration is a well-known method, usually referred to as curiosity, for instance see [2] and some of the cited papers as Pathak et. al., Stadie et. al. - these all should be at least cited in section 3.2.2 as well not only in the background section regarding different topics. 
On the auxiliary loss:
The authors claim that they train the auxiliary loss using Variational Inference, yet they drop the KL term, which is "kinda" an important feature of VI. Auxiliary losses are well understood that often help in RL, hence there is no need to over-conceptualize the idea of adding the extra term log p(b|z) as a VI and then doing something else. It would be much more clear and concise just to introduce it as an extra term and motivate it without referring to the VI framework, which the authors do not use for it (they still use it for the main generative model). The only way that this would have been acceptable if the experiment section contained experiments with the full VI objective as equation (6) suggest and without the sharing of the variational priors and posteriors and compared them against what they have done in the current version of the manuscript. 
A minor mistake seems to be that equation (5) and (7) have double counted log p(z_t|h_t-1) since they are written as an explicit term as well as they appear in the KL(q(z_t|..)|p(z_t|h_t-1)). 
[1] Prediction and Control with Temporal Segment Models [Nikhil Mishra, Pieter Abbeel, Igor Mordatch, 2017]
[2] Large-Scale Study of Curiosity-Driven Learning [Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, Alexei A. Efros, 2018]
[3] Action-Conditional Video Prediction using Deep Networks in Atari Games [Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, Satinder Singh, 2015]