Exploration in environments with sparse rewards is a key challenge for reinforcement learning. How do we design agents with generic inductive biases so that they can explore in a consistent manner instead of just using local exploration schemes like epsilon-greedy? We propose an unsupervised reinforcement learning agent which learns a discrete pixel grouping model that preserves spatial geometry of the sensors and implicitly of the environment as well. We use this representation to derive geometric intrinsic reward functions, like centroid coordinates and area, and learn policies to control each one of them with off-policy learning. These policies form a basis set of behaviors (options) which allows us explore in a consistent way and use them in a hierarchical reinforcement learning setup to solve for extrinsically defined rewards. We show that our approach can scale to a variety of domains with competitive performance, including navigation in 3D environments and Atari games with sparse rewards.
This paper proposed an algorithm for structured exploration in deep reinforcement learning via learning the visual abstractions from pixels. The proposed method learns discrete visual abstractions and derives intrinsic reward functions from them so as to help the agent to optimize the policy. 
The proposed method is interesting in that learning the visual abstractions together with the policy may assist in computing an optimal policy. The method is learning a meta Q function and (E * M+1) other Q functions. The authors mentioned that their work is most similar to hierarchical-DQN (Kulkarni et al., 2016) but this work required hand-crafted instance segmentation and the agent architecture do not learn about many intrinsic rewards learners. However, I am concerned if the proposed method solved the problem with the need of hand-crafted instance segmentation since, as shown in the Algorithm 1 and the caption of Figure 2, Q_{meta} acts every T steps. I do not understand why the meta Q function is used to propose actions for every fixed number of steps. Besides that, though the proposed method does have many intrinsic reward functions (in fact, there are E * M additional intrinsic reward functions). However, the authors did not show in the experiments if having too many intrinsic reward functions helps a lot. It will be better if the authors can show that, larger values for E or M can make the performances better.
Another concern I have is on some of the experiment results. For the experiment results in Figure 5 and 6, only in the left figures can the results of the proposed methods outperform the baselines. Besides that, the authors may need to describe the baseline methods in the experiments in more details.
Also, it will be better if the authors can improve the paper a little bit with the writing. For example, it will be better if the authors can explain the variables X, Y and the distribution q when mentioning Equation 1 so that it is easier to understand the paper. Also, there are some typos, such as the section reference on line 10 of Algorithm 1, the definition of the g function on the last line of page 3 (I guess the authors want to write "{0...E}" instead of "{0, E}") and the second sentence of the experiment section (at least I did not see the supplementary sections, but the authors mentioned that). It is better if these typos can be fixed. 
 
References:
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, pp. 3675–3683, 2016.