To select effective actions in complex environments, intelligent agents need to generalize from past experience. World models can represent knowledge about the environment to facilitate such generalization. While learning world models from high-dimensional sensory inputs is becoming feasible through deep learning, there are many potential ways for deriving behaviors from them. We present Dreamer, a reinforcement learning agent that solves long-horizon tasks purely by latent imagination. We efficiently learn behaviors by backpropagating analytic gradients of learned state values through trajectories imagined in the compact state space of a learned world model. On 20 challenging visual control tasks, Dreamer exceeds existing approaches in data-efficiency, computation time, and final performance.
Paper summary.
The paper proposes Dreamer, a model-based RL method for high-dimensional inputs such as images. The main novelty in Dreamer is to learn a policy function from latent representation-and-transition models in an end-to-end manner. Specifically, Dreamer is an actor-critic method that learns an optimal policy by backpropagating re-parameterized gradients through a value function, a latent transition model, and a latent representation model. This is unlike existing methods which use model-free or planning methods on simulated trajectories to learn the optimal policy. Meanwhile, Dreamer learns the remaining components, namely a value function, a latent transition model, and a latent representation model, based on existing methods (the world models and PlaNet). Experiments on a large set of continuous control tasks show that Dreamer outperforms existing model-based and model-free methods. 
Comments. 
Efficiently learning a policy from visual inputs is an important research direction in RL. This paper takes a step in this direction by improving existing model-based methods (the world models and PlaNet) using the actor-critic approach. I am leaning towards weak accepting the paper. 
I am reluctant to give a higher score due to its incremental contribution. Specifically, the policy update in Dreamer resembles that of SVG (Heess et al., 2015), which also backpropagates re-parameterized gradients through a value function and a transition model. The main difference between Dreamer and SVG is that Dreamer incorporates a latent representation model. From this viewpoint, the actor-critic component in Dreamer is an incremental contribution. Since the latent models are learned based on existing techniques, the paper presents an incremental contribution. 
Besides the above comments, I have these additional comments. 
- Effectiveness on very long horizon trajectories: 
Simulating long-horizon trajectories with a probabilistic model is known to be unsuitable for model-based RL due to accumulated errors. This is an open issue in model-based RL. The paper attempts to solve this issue by backpropagating policy gradients through the transition model, which is known to be more robust against model errors (see e.g., PILCO (Deisenroth et al., 2011)). However, the issue still exists in Dreamer, since there seems to be an upper limit of effective horizon length (perhaps around 40, according to Figure 4). This horizon length is still short compared to the entire horizon length of many MDPs (e.g., 1000). I think this point should be discussed in the paper. That is, the issue still exists, and Dreamer is less effective with very long horizon.
- Inapplicability to discrete controls: 
One restriction of re-parameterized gradients is that the technique is not applicable to discrete random variables. This restriction exists in Dreamer, and the method cannot be applied to discrete control tasks unless approximation techniques such as Gumbel-softmax are used. Still, such approximations would make learning more challenging, especially with long-horizon backpropagation. This restriction should be noted in the paper. 
- There is no mention about variance of policy gradient estimates. Dreamer does not use any variance reduction technique, so the gradient estimates could have very large variance. 
- q_theta was introduced in Eq. (8) before it is defined in Eq. (11). Also, I suggest moving Section 4 to be right after Section 2, since Section 4 presents existing techniques similarly to Section 2, while Section 3 presents the main contribution. 
Update after authors' response.
I read the response. The paper is more clear after authors' clarification. Though, I still think the contribution is incremental, since back-propagating gradients through values and dynamics has been studied in prior works (albeit with less empirical successes compared to Dreamer). Nonetheless, I am keen to acceptance. I would increase the rating from 6 to 7, but I will keep the rating of 6 since the rating of 7 is not possible.