Scaling end-to-end reinforcement learning to control real robots from vision presents a series of challenges, in particular in terms of sample efficiency. Against end-to-end learning, state representation learning can help learn a compact, efficient and relevant representation of states that speeds up policy learning, reducing the number of samples needed, and that is easier to interpret. We evaluate several state representation learning methods on goal based robotics tasks and propose a new unsupervised model that stacks representations and combines strengths of several of these approaches. This method encodes all the relevant features, performs on par or better than end-to-end learning, and is robust to hyper-parameters change.
This paper aims at comparing end-to-end learning vs separately learning a state representation and subsequently a controller.
While this would be a relevant and important topic, the paper does not currently present consistent evidence to support this hypothesis.
In particular:
- The approach proposed in the approach is not explained in sufficient details. After reading Sec.4 I have only a very vague and high-level idea of how the proposed approach might work. In Figure.2, what is I_t? What is the model that you are training? How are you learning this model? how do you define L_inverse?
- The cited literature about state representation learning is absolutely incomplete. Papers like Lange et al. , Wahlström et al. and Finn et al. and citations herewithin.
- From the experimental results, it is difficult to say anything definitive about the proposed hypothesis. 1) There are multiple end-to-end approaches in the literature, with significant differences in performance. which one are you using? (it seem A2C and PPO, but to which label do they correspond in the tables?) 2) How do you tune the weights of the reward function proposed? This seems an important design choice, but it is not much discussed. 3) In the table reported (e.g., Table 1) it does not seem to me that SRL consistently outperforms other approaches. Even for the arm tasks, Random features seem to outperform the proposed approach (and indeed all the methods except the ground truth). What is going on there?
Overall¸ the paper would benefit from a clearer and more detailed text, and from improved experiments and comparisons.
Minor comments:
- It is unclear to me what "goal-based robotic tasks" means. How do you define a task without a goal?
- An important and missing characteristic of a suitable state representation should be the generalization. In fact, a good representation would ideally allow the agent to generalize to some degree. 
- It seems very odd to me that the "action should be implicitly encoded into the state representation" could you elaborate of the motivation for this and the effects?
References:
- Autonomous reinforcement learning on raw visual input data in a real-world application
S Lange, M Riedmiller, A Voigtlander
Neural Networks (IJCNN), The 2012 International Joint Conference on, 1-8
- From pixels to torques: Policy learning with deep dynamical models
N Wahlström, TB Schön, MP Deisenroth
arXiv preprint arXiv:1502.02251
-  Deep Visual Foresight for Planning Robot Motion
Chelsea Finn, Sergey Levine
International Conference on Robotics and Automation (ICRA), 2017