VIME: Variational Information Maximizing Exploration Rein Houthooft
Scalable and effective exploration remains a key challenge in reinforcement learning (RL). While there are methods with optimality guarantees in the setting of discrete state and action spaces, these methods cannot be applied in high-dimensional deep RL scenarios. As such, most contemporary RL relies on simple heuristics such as -greedy exploration or adding Gaussian noise to the controls. This paper introduces Variational Information Maximizing Exploration (VIME), an exploration strategy based on maximization of information gain about the agent's belief of environment dynamics. We propose a practical implementation, using variational inference in Bayesian neural networks which efficiently handles continuous state and action spaces. VIME modifies the MDP reward function, and can be applied with several different underlying RL algorithms. We demonstrate that VIME achieves significantly better performance compared to heuristic exploration methods across a variety of continuous control tasks and algorithms, including tasks with very sparse rewards.
This paper looks at the notion of curiosity in deep reinforcement learning. It returns to the idea of equating exploratory behavior with maximizing information. Key contributions are the formulation of a variational perspective on information gain, an algorithm based on Blundell et al.'s Bayesian  neural networks and the exposition of the relationship between their approach and Schmidhuber's compression improvement. Results on simple domains are given.  
The paper shows a pleasant breadth of understanding of the literature. It provides a number of insights into curiosity for RL with neural networks. I think it could be improved by focusing on the development of the variational approach and the immediately resulting algorithm. As is, there are a number of asides that detract from the main contribution.  My main concern is that the proposed algorithm seems relatively brittle. In the case of Eqn 17, computing the Hessian might only be a good idea in the diagonal case. Dividing by the median suggests an underlying instability.  Questions: * The median trick bothers me. Suppose the model & KL have converged. Then at best the intrinsic reward is 1 everywhere and this does not change the value function. In the worst case the KL is close to 0 and you end up with a high variance in your intrinsic rewards. Why isn't this an issue here?  * Eqn 13: updating the posterior at every step is different from updating the posterior given all data from the prior. Do you think there are issues with the resulting "posterior chaining"?  * How good are the learned transition models?  * Can you explain line 230: "very low eta values should reduce VIME to traditional Gaussian control noise"?  * Why do you propose two intrinsic rewards on line 5 of algorithm 1? I'd like to see a clear position.  Suggestions: Eqn 2: P(. | s_t, a_t) Line 116: For another connection, you may want to look at Lopes et al. (2012), "Exploration in Model-based Reinforcement Learning by Empirically Estimating Learning Progress" Line 119: You should specify which description length you mean... the statement is possibly imprecise/incorrect as is Line 128: in expectation propagation (which I know from Bishop (2006)) the KLs end up getting reversed, too... is there a relationship? Eqn 13: log p(s_t | theta) should be p(s_t | s_t-1, a_t-1, theta), no?  It would be good to give empirical evidence showing why the median is needed Graphs: unreadable It might be good to cite more than just [20] as a reference on Bayesian networks. Again, Bishop (2006) (Section 5.7) provides a nice list.