Hierarchical Reinforcement Learning is a popular method to exploit temporal abstractions in order to tackle the curse of dimensionality. The options framework is one such hierarchical framework that models the notion of skills or options. However, learning a collection of task-agnostic transferable skills is a challenging task. Option discovery typically entails using heuristics, the majority of which revolve around discovering bottleneck states. In this work, we adopt a method complementary to the idea of discovering bottlenecks. Instead, we attempt to discover "landmark" sub-goals which are prototypical states of well connected regions. These sub-goals are points from which a densely connected set of states are easily accessible. We propose a new model called Successor options that leverages Successor representations to achieve the same. We also design a novel pseudo-reward for learning the intra-option policies. Additionally, we describe an Incremental Successor options model that iteratively builds options and explores in environments where exploration through primitive actions is inadequate to form the Successor representations. Finally, we demonstrate the efficacy of our approach on a collection of grid worlds and on complex high dimensional environments like Deepmind-Lab.
Summary: This paper tries to tackle the option discovery problem, by building on recent work on successor representation and eigenoptions. Although this is an extreme important problem, I feel the paper fails to deliver on its promise. The authors propose a way of clustering states via their SR/SF representation and they argue that this would lead to the discovery of subgoals that are fundamentally different from the popular choices in literature, like bottleneck states. They argue that this discovery procedure would lead to states “better for exploration”, “provide greater accessibility to a larger number of states”. Both of which sound promising, but I felt the actual evaluation fails to show or even assess either of these rigorously. Overall, after going through the paper, it is not clear what are the properties of these discovered subgoal states and why they would be better for exploration and/or control.
Clarity: Can be improved significantly! It requires several reads to get some of the important details. See detailed comments.
Originality and Significance: Very limited, at least in this version. The quantitative, and in some cases qualitative, evaluation lacks considerably. The comparison with the, probably most related, method (eigenoption) yield some slight improvement. But unfortunately, I was not conceived that this grain of empirical evidence would transfer to other scenarios. I can’t see why that would that be the case, or in which scenarios this might happen. At least those insights seem to be missing from the discussion of the results. 
Detailed comments and questions:
1) Section 3.1: Latent Learning. There are a couple of design choices here that should have been more well explained or motivated:
i) The SR were built under the uniformly random policy. This is a design choice that might work well for gridworld/navigation type of domains but there are MDP where the evaluation under this particular policy can yield uninformative evaluations. Nevertheless this is an interesting choice that I think deserved more discussion, especially the connection to previous work on proto-value functions and eigenoptions. For instance, if both of these representations -- eigenoptions and the proposed successor option model -- aim to represent the SR under the uniform policy, why does know do (slightly) better than the other? Moreover, how would these compare under a different policy (non-uniform). 
ii) The choice of reward. The notation is a bit confusing here, as it’s somewhat inconsistent with the definitions (2-4). Also, more generally, It is not clear throughout if we are using the discounted or undiscounted version of SR/SFs -- (2-3) introduce the discounted version, (4) seems to be undiscounted. Not clear if (5) refers to the discounted or undiscounted version. Nevertheless, I am guessing this was meant as a shaping reward, thus \gamma=1 for (5). But if that’s the case, according to eq. (2), most of the time I would expect \psi_s(s_{t+1}) and \psi_s(s_{t}) to have the same value. Could you explain why that is not true (at least in your examples)?
iii) Termination set: Q(s,a)<=0. This again seems a bit of an arbitrary choice and it’s not clear which reward this value function takes into account. 
2) Figure 2: The first 2 figures representing the SRs for the two room domain: the values for one of the rooms seems to be zero, although one would expect a smoother transition around the ‘doorway’, otherwise the shaping won’t point in the right direction for progression. Again, this might suggest that more informative(control) policy might give you more signal.  
3) Section 3.2: ‘The policy used for learning the SR is augmented with the previously learnt options‘. Can you be more precise about how this is done? Which options used? How many of them? And in which way are they used? This seems like a very important detail. Also is this augmented policy used only for exploration? 
4) SRmin < \sum_{s’} ψ(s, :) < SRmax. Is this meant to be an expectation over all reachable next states or all states in the environment? How is this determined or translated in a non-tabular setting. Not sure why this is a proxy to how ‘developed’ this learning problem or approximation is. Can you please expand on your intuition here?
5) Section 3.3. The reward definition seems to represent how much the progress between \phi(s_t+1) - \phi(s) aligns with the direction of the goal. This is very reminest of FuN [2] -- probably a connect worth mentioning and exploring.
6) Figure 4: Can you explain what rho is? It seems to be an intermediate representation for shared representation \phi. Where is this used?
7) Experiments:
“a uniformly random policy among the options and actions (typically used in exploration) will result in the agent spending a large fraction of it’s time near these sub-goals”. Surely this is closely linked to the termination condition of the option and the option policy. How is this assessed?
“in order to effectively explore the environment using the exploration policy, it is important to sample actions and options non-uniformly”. It would be good to include such a comparison, or give a reason why this is the case. It’s also not clear how many of the options we are considering in this policy and how extensive their horizons will be. This comes back to the termination condition in Section 3.1 which could use an interpretation. 
“In all our experiments, we fix the ratio of sampling an option to an action as 1:19.” This seems to be somewhat contradictory to the assumption that primitive actions are not enough to explore effectively this environment. 
Figure 8. I think this experiment could use some a lot more details. Also it would be good to guide the reader through the t-SNE plot in Figure 8a. What’s the observed pattern? How does this compare to the eigenoption counterpart.
8) General comment on the experiments: There seems to be several stages in learning, with non-trivial dependencies. I think the exposition would improve a lot if you were more explicit about these: for instance, if the representation continually refined throughout the process; when the new cluster centers are inferred are the option policies learnt from scratch? Or do they build on the previous ones? Does this procedure converge -- aka do the clusters stabilize?
9) Quantitative performance evaluation was done only for the gridworld scenarios and felt somewhat weak. The proposed tasks (navigation to a goal location) is exactly what SFs are trained to approximate. No composition of (sub)tasks, nor tradeoff-s of goals were studied [1,3] -- although they seem natural scenario of option planning and have been studied in previous SFs work. Moreover, if the SFs are built properly, in these gridworlds acting greedily with respect to the SFs (under the uniformly random policy) should be enough to get you to the goal. Also, probably this should be a baseline to begin with.
References:
[1] Andre Barreto, Will Dabney, Remi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and ´ David Silver. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems, pp. 4055–4065, 2017.
[2] Vezhnevets, A.S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D. and Kavukcuoglu, K., 2017, July. FeUdal Networks for Hierarchical Reinforcement Learning. In International Conference on Machine Learning (pp. 3540-3549).
[3] Barreto, A., Borsa, D., Quan, J., Schaul, T., Silver, D., Hessel, M., Mankowitz, D., Zidek, A. and Munos, R., 2018, July. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning (pp. 510-519).