Deep reinforcement learning (RL) algorithms can use high-capacity deep networks to learn directly from image observations. However, these kinds of observation spaces present a number of challenges in practice, since the policy must now solve two problems: a representation learning problem, and a task learning problem. In this paper, we aim to explicitly learn representations that can accelerate reinforcement learning from images. We propose the stochastic latent actor-critic (SLAC) algorithm: a sample-efficient and high-performing RL algorithm for learning policies for complex continuous control tasks directly from high-dimensional image inputs. SLAC learns a compact latent representation space using a stochastic sequential latent variable model, and then learns a critic model within this latent space. By learning a critic within a compact state space, SLAC can learn much more efficiently than standard RL methods. The proposed model improves performance substantially over alternative representations as well, such as variational autoencoders. In fact, our experimental evaluation demonstrates that the sample efficiency of our resulting method is comparable to that of model-based RL methods that directly use a similar type of model for control. Furthermore, our method outperforms both model-free and model-based alternatives in terms of final performance and sample efficiency, on a range of difficult image-based control tasks. Our code and videos of our results are available at our website.
This paper proposes an actor-critic method that tries to aid learning good policies via learning a good representation of the state space (via a latent variable model). In actor-critic methods, the critic is learnt to evaluate policies in the latent space, which further helps with efficient policy optimization. The proposed method is evaluated on image-based control tasks, with baseline evaluations against both model-based and model-free methods in terms of sample efficiency. 
- The key argument is that learning policies in the latent space is more efficient, as it is possible to learn good representations in the latent space. There are quite a few recent works (e.g DeepMDP, Gelada et al., 2019;  Dadashi et al., 2019) that talks about representation learning in RL, and yet the paper makes no relations or references to previous works. I find it surprising that none of the past related works are mentioned in the paper. 
- I find the arguments on solving a POMDP instead of a MDP a bit vague in this context. I understand that the goal is to solve image based control tasks - for which learning good representations via a latent variable model might be useful, but it does not explicitly require references to a POMDP? In most ALE tasks, we have pixel based observations too, which makes the ALE environments a POMDP in some sense, but we use approximations to it to make it equivalent to a MDP with sufficient history. The arguments on POMDP seems rather an additional mention, with no necessary significance to it?
- The paper mentions solving RL in the learned latent space, which is empirically proposed to be a good approach without theoretical justifications. There are several recent works that tries to understand the representation learning in RL problem from a theoretical perspective too - it would be useful to see where this approach stands in light of those theoretical results? Otherwise, the contribution seems rather limited : solving RL in latent space is useful, but there are no justifications to it? Why should this approach even be adapted or what is the significance of it?
- The proposed actor-critic method in the latent space is built on top of Soft Actor-Critic (SAC). I understand this is a design/implementation approach building from previous works - but it would have been useful to add more context as to what it means to learn a critic in the latent space. If the critic evaluates a policy in the latent space - then is this a good policy evaluation for actor-critic itself? Why or why not? I do not understand why the critic evaluation in the latent space is even a good approach? 
- My first impression was that the paper proposes a separate auxilliary objective for learning good representations based on which actor-critic algorithms can be made more efficient. However, this does not seem to be the case directly? Following on previous point - I find the argument of solving a critic in the latent space rather vague. 
- The sequential latent variable model proposed is based on existing literature. This can be any latent variable model (e.g VAEs), but I understand, as mentioned in the paper, the design choice of using sequential models to capture the temporal aspect. 
- The proposed algorithm is in fact a combination of SAC and sequential latent variable models, both of which are well-known in the literature. The SLAC algorithm combines these to solve image-based control tasks. As per equation 10, which is the regular policy optimization objective with max entropy - the only difference is that the critic is evaluated in the latent space. This appears to me as more of an engineeering choice, and experimentally one that perhaps give good results - but the lack of justifications of why equation 10 is even the right objective to solve makes the paper rather less appealing. 
- I think overall the contribution of the paper is rather limited. It is more of an experimental design and engineering approach that combines previous known techniques. The paper mentions learning good representations for RL, without any references or justifications - and it appears that overall there are bold claims made in the paper but it lacks significant scientific contribution. 
- Experimental evaluations are made on image based control tasks. Experimental results are compared to few baselines - but it is not clear whether these are even the right baselines. For example, it would have been good to include analysis of the proposed model with different latent variable models (including VAE) to perhaps justify the choice of the latent variable model. Results in figure 5 appear a bit concerning to me - these are mostly the standard Mujoco tasks from the OpenAI suite. Are these all image based benchmarks too, or the standard baselines? It is not clear from the text. Assuming they are standard baselines, the comparisons made are rather unfair (for example : SAC and MDP performs much better on tasks like HalfCheetah-v2). Why are the baselines performing so poorly in the results?
- Overall, I think the paper needs more work in terms of writing and justifying the choice of the approach. There are significant references missing in the paper. Most importantly, there are quite a few claims made in the paper which are not properly justified, that makes the overall contribution and novelty of the paper rather limited. I would tend for a rejection of this paper, as it requires more work - both in terms of theoretical justifications (including references) and experimetnal ablation studies and more simpler benchmarks explaining the choice of the approach.