Training an agent to solve control tasks directly from high-dimensional images with model-free reinforcement learning (RL) has proven difficult. The agent needs to learn a latent representation together with a control policy to perform the task. Fitting a high-capacity encoder using a scarce reward signal is not only sample inefficient, but also prone to suboptimal convergence. Two ways to improve sample efficiency are to extract relevant features for the task and use off-policy algorithms. We dissect various approaches of learning good latent features, and conclude that the image reconstruction loss is the essential ingredient that enables efficient and stable representation learning in image-based RL. Following these findings, we devise an off-policy actor-critic algorithm with an auxiliary decoder that trains end-to-end and matches state-of-the-art performance across both model-free and model-based algorithms on many challenging control tasks. We release our code to encourage future research on image-based RL 1 .
The paper aims to tackle the problem of improving sample efficiency of model-free, off-policy reinforcement learning in an image-based environment. They do so by taking SAC and adding a deterministic autoencoder, trained end-to-end with the actor and critic, with the actor and critic trained on top of the learned latent space z. They call this SAC-AE. Experiments in the DeepMind control suite demonstrate that the result models train much faster than SAC directly on the pixels, in some cases reaching close to the performance of SAC on raw state. Ablation studies demonstrate their approach is most stable with deterministic autoencoders proposed by (Ghosh et al, 2019), rather than the beta-VAE autoencoder proposed in (Nair et al, 2018), end-to-end learning of the autoencoder gives improved performance, and the encoder transfers to some similar tasks.
I thought the paper was written well, and its experiments were done quite carefully, but it was lacking on the novelty front. At a high level, the paper has many similarities with the UNREAL paper (Jaderberg et al, 2017), which is acknowledged in the related work. This paper says it differs from UNREAL because they use an off-policy algorithm, and that UNREAL's auxiliary tasks are based off real-world inductive priors.
I don't see the off-policy distinction as very relevant, because in the end, both UNREAL and SAC-AE are actor-critic algorithms (using A3C and SAC respectively). The way that SAC is used in the paper always collects data in a near on-policy manner, and UNREAL includes experience replay from a replay buffer, which introduces some off-policy nature to UNREAL as well. Therefore this doesn't feel like a strong argument.
Furthermore, although some of the auxiliary tasks in UNREAL are based off human intuition for what makes sense in those environments, they also include task-agnostic auxiliary tasks: reward prediction and pixel-level control. These do not depend on real-world inductive priors, and are shown to improve performance.
Overall, this doesn't feel like a strong enough contribution for ICLR.
More specific comments:
* Section 6.1 examines the representation power of the encoder by reconstructing proprioceptive state from the encoder. I am not sure the comparison between SAC+AE and SAC is particularly meaningful here. The predictors are learned on top of the encoder output, and in SAC+AE we would expect task information to be encoded in the learned z. But in baseline SAC, there is no reason to expect this to be true - task information is more likely to be distributed across the entire network architecture. The case for SAC+AE seems much stronger from the reward curves, rather than these plots.
* The paper argues that their approach is stable and sample-efficient, but when looking at the reward curves, it looked about as stable as SAC. Figure 3 (where they do not train the VAE end-to-end in the red curve) has a similar story. This makes me believe that any claims of added stability are more thanks to SAC, rather than proposed methods.
Edit: I would like to clarify that the rating system only provides a 3 for Weak Reject and 6 for Weak Accept. On a 1-10 scale I would rate this as a 5, I feel it is closer to Weak Accept than Weak Reject.
Edit 2: I've read the other author's comments. I'm not particularly convinced by the case for novelty, but I didn't realize that UNREAL's replay buffer was only 2k transitions instead of 1 million transitions. On reflection, I believe the main contribution here is showing that deterministic autoencoders are more reliable than stochastic ones for the RL setting, and this isn't the biggest contribution, but it's enough to make me update to weak accept.