In partially observable (PO) environments, deep reinforcement learning (RL) agents often suffer from unsatisfactory performance, since two problems need to be tackled together: how to extract information from the raw observations to solve the task, and how to improve the policy. In this study, we propose an RL algorithm for solving PO tasks. Our method comprises two parts: a variational recurrent model (VRM) for modeling the environment, and an RL controller that has access to both the environment and the VRM. The proposed algorithm was tested in two types of PO robotic control tasks, those in which either coordinates or velocities were not observable and those that require long-term memorization. Our experiments show that the proposed algorithm achieved better data efficiency and/or learned more optimal policy than other alternative approaches in tasks in which unobserved states cannot be inferred from raw observations in a simple manner.
* Summary
The work considers reinforcement learning in partially observed environments, targeting in particular the case when the agent needs to remember some important information over a longer period of time. 
The work gives an interesting connection between variational recurrent neural networks and reinforcement learning. The paper is well written and technically sound. The method achieves improved performance in several (simpler) benchmark problems in comparison to strong baselines.
* Detailed discussion
The work allocates latent variables to describe unobserved part of the state and learns the state transition probability model (the action-dependent generator (10)). Learning these state transitions becomes an unsupervised learning task, for which the lower bound on the likelihood of the observed data is optimized. This learning of the state transition model is actually similar to classical model-based RL. The policy and value functions are then learned on top of this model using SoTA techniques. From this perspective I would say this is a classical approach to partially observed reinforcement learning with advanced models for transition matrix and value and policy functions.
The work combines together variational recurrent neural network (VRNN) and soft actor critic (SAC) models into a rather advanced system for reinforcement learning. These parts are somewhat extended to fit together, but it is not clear whether all components are needed. For example, why z and d need to be distinguished, cannot they be combined into one latent variable “zd” in order to simplify the number of connections? Or I am missing some important property necessary for the inference?
In (7),(8) are the expectations over the same state / actions or different? The notation is not clear about it.
After (2), given x_t and d_{t-1} instead of d_t?
The experiments are conducted in the setting that the observations were all relevant measurements for the agent (positions, velocities). From Fig 7 RoboscoolHooper (close loop) we see that actually all measurements can be well predicted by generative model up to 8 steps ahead (in dim 2 that stays zero the noise is amplified). This is a nice and desirable property and is the consequence of the generative modeling and the marginal likelihood ELBO optimization. However the paper seem to argue (Appendix E) that the algorithm does not require the model to make accurate predictions and relies on the encoding capacity, which I find confusing. Furthermore, it would be interesting to see how this approach should scale to the setting when there are more sensory inputs, some less relevant than the other. In the end, is a good generative model to predict the future observation needed or not needed in this approach?
The exploration seems to be addressed just by randomizing the policy,  are there any better options?