EX 2 : Exploration with Exemplar Models for Deep Reinforcement Learning
Deep reinforcement learning algorithms have been shown to learn complex tasks using highly general policy classes. However, sparse reward problems remain a significant challenge. Exploration methods based on novelty detection have been particularly successful in such settings but typically require generative or predictive models of the observations, which can be difficult to train when the observations are very high-dimensional and complex, as in the case of raw images. We propose a novelty detection algorithm for exploration that is based entirely on discriminatively trained exemplar models, where classifiers are trained to discriminate each visited state against all others. Intuitively, novel states are easier to distinguish against other states seen during training. We show that this kind of discriminative modeling corresponds to implicit density estimation, and that it can be combined with countbased exploration to produce competitive results on a range of popular benchmark tasks, including state-of-the-art results on challenging egocentric observations in the vizDoom benchmark.
Review of submission 1489:
EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
Summary:
A discriminative novelty detection algorithm is proposed to improve exploration for policy gradient based reinforcement learning algorithms. The implicitly-estimated density by the discriminative novelty detection of a state is then used to produce a reward bonus added to the original reward for down-stream policy optimization algorithms (TRPO). Two techniques are discussed to improve the computation efficiency. 
Comments
- One motivation of the paper is to utilize implicit density estimation to approximate classic count based exploration. The discriminative novelty detection only maintains a density estimation over the states, but not state-action pairs. To the best of my knowledge, most theoretical results on explorations are built on the state-action visit count, not state visit count. Generally, exploration strategies for RL algorithms investigate reducing the RL agent’s uncertainty over the MDP’s state transition and reward functions. The agent’s uncertainty (in terms of confidence intervals or posterior distributions over environment parameters) decreases as the inverse square root of the state-action visit count. 
- It is not clear from the texts how the added reward bonus is sufficient to guarantee the novel state action pairs to be visited for policy gradient based RL methods. Similar reward bonuses have been shown to improve exploration for value-based reinforcement learning methods, such as MBIE-EB (Strehl & Littman, 2009) and BEB (Kolter & Ng, 2009), but not policy gradient based RL methods. Value-based and policy gradient based RL methods differ a lot on the technical details. 
- Even we assume the added reward bonuses could result in novel state action pairs to be visited as desired, it is not clear why policy gradient methods could benefit from such exploration strategy. The paper, State-Dependent Exploration for Policy Gradient Methods (Thomas Ruckstieß, Martin Felder, and Jurgen Schmidhuber, ECML 2008), showed that limiting the exploration within an episode (i.e., returning the same action for any given state within an episode) could be helpful for policy gradient RL methods by reducing the variance in the gradients and improving the credit assignment. The soundness of the proposed techniques would be improved if necessary explanations were provided. 
- There is one closely related recent paper on improving exploration for policy gradient methods, IMPROVING POLICY GRADIENT BY EXPLORING UNDER-APPRECIATED REWARDS (Ofir Nachum et al. ICLR 2017). It would be good for the paper to discuss the relationship and even compare against it empirically.