Visual Reinforcement Learning with Imagined Goals
For an autonomous agent to fulfill a wide range of user-specified goals at test time, it must be able to learn broadly applicable and general-purpose skill repertoires. Furthermore, to provide the requisite level of generality, these skills must handle raw sensory input such as images. In this paper, we propose an algorithm that acquires such general-purpose skills by combining unsupervised representation learning and reinforcement learning of goal-conditioned policies. Since the particular goals that might be required at test-time are not known in advance, the agent performs a self-supervised "practice" phase where it imagines goals and attempts to achieve them. We learn a visual representation with three distinct purposes: sampling goals for self-supervised practice, providing a structured transformation of raw sensory inputs, and computing a reward signal for goal reaching. We also propose a retroactive goal relabeling scheme to further improve the sample-efficiency of our method. Our off-policy algorithm is efficient enough to learn policies that operate on raw image observations and goals for a real-world robotic system, and substantially outperforms prior techniques.
The paper introduces an approach for learning general purpose skills by combining learning a latent representation of images with goal conditioned policies. A visual representation of raw observations is learnt using a VAE model that can also be used to generate arbitrary goals for a goal conditioned policy. The resulting learnt policy can operate with raw image observations in physical control tasks. The framework can also be interpreated as a generalization of Hindisght Experience Replay for control tasks.   A latent variable model is used to learnt a representation of raw image inputs based on VAE framework. The authors motivate that such a model not only learns a good representation of raw inputs, but the prior on the latent variables can also be used to generate random goals for learning a goal conditioned policy. The proposed latent variable model is used for several objectives, specifically to compute arbitrary reward functions based on distances in the latent space between the current state and randomly generated goal. The overall RL objective is trained alongside with a \beta-VAE objective._x000b__x000b_Even though the paper is trying to motivate an interesting latent variable model that can handle raw images for control tasks, there are several aspects of the paper which make the overall research contribution complex. The paper proposes several objectives, and it is not clear why the proposed approach exactly works better. I also do not think the proposed method of learning a general purpose goal conditioned policy, as motivated in the paper, is indeed what it claims to be.   _x000b_The experimental results demonstrated are not fully satisfying. Even if the authors compare to several baselines including HER, they mention modifications being made to the baseline algorithms. It would be been useful to see how the proposed method works on other tasks, instead of simply on the Reacher and Pusher baselines, along with a comparision with the original baselines (e.g HER uses similar experiment setups with Pusher and Reacher).  The range of experimental results is very limited, and the modifications made to the baselines raises questions about reproducibility of these results. _x000b__x000b_Another confusing aspect is the use of TD3 - why is TD3 used in the first place instead of standard off-policy RL algorithms for control tasks? Was there a specific reason to use TD3?_x000b__x000b_Overall, the paper introduces an interesting idea to use a latent variable model to achieve several objectives, but fails to sufficiently demonstrate the proposed approach experimentally on a range of tasks.  The limited set of experimental results are not sufficiently convincing, and it is not exactly clear how the proposed approach can be implemented or can be made to work in practice. The paper is trying to motivate an interesting idea, but may not be useful for practitioners in the field for how to use the proposed approach. More detailed discussion of the algorithm and implementation details are required. _x000b__x000b_I suggest the authors should make it more clear how the experiments with the proposed method can be implemented and reproduced in practice._x000b_  Quality : Good.  Clarity : Average. The paper is at times difficult to follow as it tries to motivate the proposed approach from several perspectives, but does not provide enough intuitions or experimental demonstration of how they are achieved. _x000b_ Originality : Slightly above average_x000b__x000b_Significance : Average.  Overall Rating : Marginally above acceptance level. I think the paper is introducing an interesting approach for raw image based control tasks, but does not still provide sufficient experimental results to demonstrate it. However, I think the overall research contribution is novel and maybe considered for acceptance.