Model-based reinforcement learning (RL) methods can be broadly categorized as global model methods, which depend on learning models that provide sensible predictions in a wide range of states, or local model methods, which iteratively refit simple models that are used for policy improvement. While predicting future states that will result from the current actions is difficult, local model methods only attempt to understand system dynamics in the neighborhood of the current policy, making it possible to produce local improvements without ever learning to predict accurately far into the future. The main idea in this paper is that we can learn representations that make it easy to retrospectively infer simple dynamics given the data from the current policy, thus enabling local models to be used for policy learning in complex systems. We evaluate our approach against other model-based and model-free RL methods on a suite of robotics tasks, including a manipulation task on a real Sawyer robotic arm directly from camera images.
In this work the authors propose an end to end approach for model based reinforcement learning from images, where the main building blocks are locally-linear dynamical systems and variational auto-encoders (VAE). Specifically, it is assumed that the input features (i.e., the images) are generated from a low dimensional latent representation mapped through parametric random functions; the latter are modeled via neural networks. A recognition model based on convolutional neural networks operates on the reverse way and is responsible for projecting the input features to the latent space, in order to proceed with the reinforcement learning task. The variational framework is employed in order to jointly learn the VAE and the linear dynamics on the latent state. As a final step, once the model is fitted a linear quadratic system (LQS) is solved in order to learn the cost function and the optimal policy. 
* The paper is well motivated and tries to solve an interesting problem, that of data-efficient reinforcement learning. The experiments are well picked and demonstrate the advantages of the proposed approach towards solving the task, however, the method is only evaluated on few environments and compared against only a couple of other methods. I would expect a broader evaluation and/or comparison against more methods. Since the model is able to reach TRPO’s performance in much less steps it would be nice to see how it performs against PPO from [Schulman et al. 2017] (at least on the simulated environments). Also, would it make sense to compare against [Levine et al. 2016] that has been evaluated on similar tasks?
[Schulman et al 2017] “Proximal Policy Optimization Algorithms”.
[Levine et al. 2016] “End-to-End Training of Deep Visuomotor Policies”.
* Methodologically, the paper is sound. The model part (as the authors point out) is based on [Johnson et al. 2016] and is well explained. On the other hand, the policy part, and in particular the policy update in Section 4.2 has some issues regarding readability. There is a strong interplay between Section 4.2, Section 2.1 and Appendix D and the authors did not manage to nicely explain what exactly is happening during the update phase. In the beginning the reader has the impression that we are finding the optimal policy via the closed-form LQS. Later on we switch to constrained optimisation for the cost by accounting for the KL divergence between the policy on two episodes. Finally, in the appendix we are back to the original quadratic cost. The authors need to clarify all the above. Also, they need to explicitly mention why they opt for stochastic optimisation (is it because of minibatching?)
* To continue with the policy, in Section 4.2 the authors argue that although the optimal policy can be found in closed form this is not desirable because the policy will overfit the model and will not generalise well in the real environment. I disagree with this statement. If this happens it effectively means that the learned model or the assumption/learning of the linear dynamics is not right. The authors seem to also agree with this since they clearly state in the the experimental section that “... our method does not heavily rely on an accurate model...”. To my understanding, this means that we need to refine the modelling strategy and not learn a sub-optimal policy. I am really interested in the authors opinion on that.
* The above argument is also directly related to the recognition model and learning of the policy in the latent state (I completely agree with that). The recognition network, which in this case is a convolutional neural network, is used as an inference mechanism to project the observations to the latent space. We learn the (variational) parameters of the recognition model by optimising the likelihood’s lower bound. This means that we are “allowed” to overfit the variational parameters as long as the bound gets tighter. This can possibly result in degraded performance during the policy update. Furthermore, the variational distribution of the latent state, i.e., q(z_t | s_t) is assumed to be mean field across time (independent z’s), while clearly this is not the case in the posterior. You somehow mitigate that by augmenting the observed state (feeding consecutive frames to the network), but still this is not ideal. Finally, is there a reason why we only use the mean of the recognition model to fit the cost on the projected latent states? Why are we throwing away the uncertainty? Especially since you do not use an exact solver and follow a stochastic gradient.
* In the end of Section 2.1, the authors argue regarding the fact that the prior work assumes access to a compact low-dimensional representation which does not allow them to perform well on images. Reference is needed.
* In the related work the authors mention modelling bias as a downside of prior work. Can you please elaborate on that? Where does the bias come from and, more importantly, how does your approach overcome this issue?
* In the experiment and specifically in Figure 4 am I right in assuming that the distance to target is measured in actual pixels? Furthermore, why the relevant plot for the reacher task is depicting rewards instead of the distance to target. To me this suggests that the task is not solved. In general what I find very upsetting in the field are plots that only depict accumulated reward for a specific task. There are many situations where the agent learns a weird behaviour that happens to give good rewards (e.g., spinning around the cart-pole), and unfortunately such behaviours are not spotted on the reward plots.
Overall, the paper is nicely presented and definitely an interesting work. However, given the fact that methodologically we have not learned anything new from this paper and in combination with the not satisfying experimental evaluation I warrant for rejection.