Exploration in sparse reward reinforcement learning remains an open challenge. Many state-of-the-art methods use intrinsic motivation to complement the sparse extrinsic reward signal, giving the agent more opportunities to receive feedback during exploration. Commonly these signals are added as bonus rewards, which results in a mixture policy that neither conducts exploration nor task fulfillment resolutely. In this paper, we instead learn separate intrinsic and extrinsic task policies and schedule between these different drives to accelerate exploration and stabilize learning. Moreover, we introduce a new type of intrinsic reward denoted as successor feature control (SFC), which is general and not task-specific. It takes into account statistics over complete trajectories and thus differs from previous methods that only use local information to evaluate intrinsic motivation. We evaluate our proposed scheduled intrinsic drive (SID) agent using three different environments with pure visual inputs: VizDoom, DeepMind Lab and DeepMind Control Suite. The results show a substantially improved exploration efficiency with SFC and the hierarchical usage of the intrinsic drives. A video of our experimental results can be found at https://gofile.io/?c=HpEwTd.
## Summary
This paper proposes a novel intrinsic reward for exploration called SFC (successor feature control), to deal with sparse-reward and hard-exploration task. The main idea of SFC is to provide an agent with intrinsic reward defined to be the L2 distance between the successor features of two consecutive states (Equation 4). An underlying motivation of SFC exploration is that high SFC would encourage the agent to enter the "bottleneck states" and therefore helps to explore the entire state space. 
Another line of contribution is SID (scheduled intrinsic drive), where a scheduler is used to determine which of the two separate policies (one for extrinsic reward and intrinsic reward) is chosen, with a fixed probability in their implementation, and executed for the next rollout of experience. It has an effect of longer-term exploration and prevents the agent from collapsing to a local-optimum behavior. 
Empirically, the SFC+SID algorithm is evaluated on custom sparse-reward navigation-type environments such as VizDoom and DeepMind Lab (as well as a simple pixel-based continuous control), and outperforms other intrinsically motivated RL algorithms including RND and ICM.
## Overall Assessment
Overall, I like the idea of this paper and finds it very interesting and promising, but feel it would be on the borderline or slightly below the bar.
This paper studies a very interesting and novel approach of leveraging successor features for exploration. Successor features are a promising way of learning dynamics-related, task-agnostic representation for RL, which can provide a temporally extended exploration signal. The resulting method presents an improvement over existing intrinsic-reward exploration algorithms.
However, I think there are some weaknesses of the paper that would put the paper slightly below the acceptance threshold: empirically the environments are not diverse enough, and they have an implicit structure assumed and favorable for the proposed method --- there remains a question whether the method is general and not task-specific. I also think there are some misleading overclaims. Please see the detailed comment below.
## Detailed Comments
**[Problem Motivation and Significance]**
This paper address an important, long-standing problem in RL of efficient exploration under sparse-reward environments.
A minor comment: in the introduction, it is said that "terminal reward RL settings" are considered (the reward is given when the goal is achieved) --- which is an extreme case of sparse-reward RL problems --- but in the experiments non-terminal reward environments are studied, e.g. "AppleDistractions" where each of apples yields +0.05 reward. I think the overall claim could be a bit toned down to, for instance, dealing with sparse-reward environments.
**[Clarity]**
Overall the paper is clearly written and easy-to-follow. Descriptions of implementation details are well provided. However, there are some parts that can be better clarified and improved more. Please see more detailed comments below.
**[Justification of Method (SFC & SID)]** 
The choice of successor feature for driving a novelty-like intrinsic reward signal seems well-motivated. This is because learning of successor features is task-agnostic and only related to multi-step dynamics (though SF is with respect to "a policy"), which gives a good representation that captures topological characteristics of the environment. The way intrinsic reward is derived, the squared distance of SFs of $S_t$ and $S_{t+1}$, basically encourages the agent to visit and go across the "bottleneck" state. 
In the description of methods, it should be clearly noted that what is the underlying policy being learned for deriving successor features (i.e. $\pi$ in Eq.3 and 4) --- for example, is it a behavoral policy (which is a mixture of two policies) induced by scheduled drive? Another comment related to this about "SFC captures statistics over the full distribution of policies that have been followed, ..." (section 2), which sounds a bit overclaiming to me. Please note that a SF is with respect to a specific policy (e.g. behavioral policy), from the expectation in the definition; I think the use of past experience for minimizing the TD error is basically for estimating the expectation term through approximation, so I am not sure that this claim is well-justified.
It is not very clear to me why the SFC reward agrees with bottleneck states. I don't think the explanation given in Section 3.2 is logically enough. Also, isn't it only true under a random exploration policy? How would you defined the "bottleneck states" (e.g. Tomar et al. 2019) -- which can be helpful for making the main idea more understandable? Moreover, there was no enough explanation or reasoning about why SD (successor distance) is roughly the shortest path between the states.
In SID, a policy for extrinsic rewards and another policy for intrinsic one are learned. But in case the extrinsic reward was never received (in case of terminal-reward environments), the former policy would behave no different than random policies. Is this interpretation correct?
Also, given the presented form of SID, it sounds like a bit overclaiming to say it is a hierarchical RL agent, since the scheduler just picks one of the policies with equal probability (rather than being learned) --- especially one policy would become a random exploration policy --- and there is no notion of abstraction or goal/options.
**[Environment choice]**
I feel (1) that the environments being evaluated on are not diverse enough, and (2) that the environment in the experiment seems to exhibit specific properties that are favorable to the algorithm.
(Bottleneck State) One implicit assumption is that the structure of navigation is chosen such that following bottleneck states would lead to an optimal trajectory. I agree that even on the maze like FlytrapEscape the navigation/exploration problem is not easy in the absence of rich reward signals, but this is exactly a sort of environments on which SFC can perform better, especially compared to RND/ICM which are not attracted by bottleneck states (Appendix A). It is good though, and could be beneficial in many cases with the presence of bottleneck states, but seems general applicability is a little bit short (not as much as claimed).
(Distinctive appearance) Another assumption is about a choice of appearance. One important thing to note about SF learning is that a feature for state or transition (cumulant) is kept fixed after random initialization, rather than being learned as in (Machado et al. 2018; Kulkarni et al. 2016; Barreto et al. 2017). This is because this method does not need to do regression of reward function. Then, the state-feature $\phi(s)$ should be discriminative enough so that it can capture some topological and global characteristic of the state space. In general, this is not an easy problem (for first-person view POMDPs), but seems on the environments (FlytrapEscape, AppleDistractions) it was possible because each room/sector has uniquely identifiable wall color and texture. I feel this is somewhat strong assumption made to make SF work. Thus, "We believe this is the first time that SF are shown to behave in a first-person view environment as one would expect from its definition" would sound a bit overclaiming. Would this method work on more general environments that do not have this property --- specifically, what will happen if rooms are not distinguishable from color and texture (and the walls were looking similar)?
Control from pixels (DM Cartpole) is an example of environment that does not have these assumptions, but one downside is that action space was simplified and discretized. Indeed, the improvement shown on Cartpole over ICM/RND is not substantial enough. To demonstrate that SFC+SID is "generally useful" as claimed in the paper, presenting benchmark results on standard discrete-action Atari environments, or more diverse RL environments would have greatly strengthened the paper to be more convincing.
**[Analysis of successor distance]**
Figure 11 (visualization of successor distance) is a great analysis, and I liked it. It clearly shows a smooth topology of the environment thanks to the temporally-extended representation that SF captures. I found that the difference heatmap is a little bit difficult parse. Also, under which policy the SF was computed (I guess this is a behavior policy derived by SID; it should be clearly mentioned somewhere in the paper)? 
**[More minor comments about experiments]**
- Was the same K-step objective (e.g. K=5) used for all of SFC, ICM and RND? If so, what would the result look like when K=1?
- The ablation study (appendix 1) is interesting and very important. The "Ours" algorithm in the main text is actually a combination of SFC and SID, so the comparison shown in this ablation study could be a main result.
## Feedback for Improvement
More related work:
* Learning decomposed value functions for extrinsic and intrinsic rewards have been discussed in (Burda et al, 2018b), though in their work a single policy is being learned.
* [Comparison with Machado et al. 2018] It is discussed that (Machado et al. 2018: count-based exploration with SR) is very similar because of the use of SR/SF. The ways of how to derive intrinsic reward signal are indeed different, but it would be great to have a detailed discussion about how they are different or similar.
Minor comments:
* Citation needed on section 3.1 --- (Kulkarni et al. 2016 or Barreto et al. 2017)
* Please consider putting the environment name in the title of each learning curve.
* Typo: Therfore (right before section 3.2)
* Typo: temporarily -> temporally (introduction bullet point 2)