It has been established that diverse behaviors spanning the controllable subspace of a Markov decision process can be trained by rewarding a policy for being distinguishable from other policies (Gregor et al., 2016;Eysenbach et al., 2018;Warde-Farley et al., 2018). However, one limitation of this formulation is the difficulty to generalize beyond the finite set of behaviors being explicitly learned, as may be needed in subsequent tasks. Successor features (Dayan, 1993;Barreto et al., 2017) provide an appealing solution to this generalization problem, but require defining the reward function as linear in some grounded feature space. In this paper, we show that these two techniques can be combined, and that each method solves the other's primary limitation. To do so we introduce Variational Intrinsic Successor FeatuRes (VISR), a novel algorithm which learns controllable features that can be leveraged to provide enhanced generalization and fast task inference through the successor features framework. We empirically validate VISR on the full Atari suite, in a novel setup wherein the rewards are only exposed briefly after a long unsupervised phase. Achieving human-level performance on 12 games and beating all baselines, we believe VISR represents a step towards agents that rapidly learn from limited feedback.
The authors address the problem of finding optimal policies in reinforcement learning problems after an initial unsupervised phase in which the agent can interact with the environment without receiving rewards.  After this initial phase, the agent can again interact with the environment while having access to the reward function. To address this specific setting, the authors propose to use the successor feature representation of policies and combine it with methods that estimate policies in the unsupervised setting (without a reward function) by maximizing the mutual information of a policy-conditioning variable and the agent behaviour. The result is a method called Variational Intrinsic Successor Features (VISF) which obtains significant performance gains on the full Atari task suite in a setting of few-step RL with unsupervised pre-training. The main contribution seems to parameterize the successor features in terms of a variable specifying the policy. This variable will be the same as the linear weights in the linear model for the reward assume by the successor features representation. Finally, a discriminator aims to predict the linear weights of the policy from the observed state-feature representation.
Clarity:
I think this is one of the weaknesses of the paper. The writing and clarity could be significantly improved. How do you go from equation 8 to equation 9? How does q lower boun p? In the paragraph after equation 9 the authors mention the score function estimator. But very few details are given. After reading the paper, my impression is that the reproducibility of the results could be very hard, because of the lack of details of the specific implementation. The authors also do not mention that the code will be publicly available after acceptance.
I feel that, to better understand the method, the authors should include experiments in simple and easy to understand synthetic environments which can be more illustrative than ATARI.
What is the difference between the policy parameters theta and the conditioning variable z?
Novelty:
The proposed approach is novel up to my knowledge. I find the idea of parameterizing the successor features in terms of the policy parameters very innovative. 
Quality:
The proposed approach seems well justified and the experiments performed indicate that the method can be useful in practice. However, I think it would be very useful to have results on other environments besides ATARI. For example,  DIAYN contains experiments on a wide range of tasks, including gridworld style tabular experiments to illustrate what their method does. This work would benefit from similar simpler and easier to understand synthetic environments (unlike ATARI).
Significance:
The proposed contribution seems significant as illustrated by the experimental results and the novel methodological contributions. However, the lack of clarity and the difficulty in the reproduction of the results limit this.
Some minor comments:
I recommend the authors to remove references in the abstract.
Update after the authors' rebuttal:
After looking at the response from the authors, I believe that they have successfully addressed my concerns. 
Therefore, I have decided to update my rating and vote for acceptance. I am looking forward to seeing the python notebook with the implementation of the VISR algorithm.