We consider the problem of balancing exploration and exploitation in sequential decision making problems. This trade-off naturally lends itself to probabilistic modelling. For a probabilistic approach to be effective, considering uncertainty about all immediate and long-term consequences of agent's actions is vital. An estimate of such uncertainty can be leveraged to guide exploration even in situations where the agent needs to perform a potentially long sequence of actions before reaching an under-explored area of the environment. This observation was made by the authors of the Uncertainty Bellman Equation model (O'Donoghue et al., 2018), which explicitly considers full marginal uncertainty for each decision the agent faces. However, their model still considers a fully factorised posterior over the consequences of each action, meaning that dependencies vital for correlated long-term exploration are ignored. We go a step beyond and develop Successor Uncertainties, a probabilistic model for the state-action value function of a Markov Decision Process with a non-factorised covariance. We demonstrate how this leads to greatly improved performance on classic tabular exploration benchmarks and show strong performance of our method on a subset of Atari baselines. Overall, Successor Uncertainties provides a better probabilistic model for temporal difference learning at a similar computational cost to its predecessors.
This paper tackles the classical exploration / exploitation problem in reinforcement learning.
The paper argues that it is necessary to propagate uncertainty correctly and argue that they can do so using the successor representation to compute the Bayesian posterior over Q-values conditioned on the data already observed.
Novelty:
This work is similar to “The uncertainty bellman equation” (UBE) (O’Donoghue et al. 2018) which adds a head to a regular DQN agent to predict a function u which is an upper bound of the variance of the posterior distribution over Q-values. The difference here is that the successor features are used here to predict Q-values and the function u.
The successor features can be seen as a discounted state occupancy of the current policy and carry information about the future. While the relation with the UBE is highlighted in section 4.6 an empirical evaluation between the two methods would also be needed.
Clarity: The method is detailed comparing to contextual bandit methods, the authors then argue that applying directly these methods to reinforcement learning case does not propagate uncertainty over several timesteps. While this indeed true it is misleading and imply that propagating uncertainty is not considered by current exploration methods in RL.
Soundness:
The method presented here is relatively reductive. The estimated posterior over the value function is correct only if the transition model P is already known, otherwise Equation 4 would also need to incorporate uncertainty over P. Similarly, as the authors point out, this doesn’t include the max operation. At best, we are learning a posterior over the value function for a fixed policy, for when only the reward is unknown.
As a whole, the authors argue that their method allows a better propagation of Q-values uncertainty but provide little theoretical or experimental evidence that would back this claim.
From a deep RL perspective, the features \phi^l only carry local information. The authors argue that this leads to more stable features as these feature do not depend on the current policy. However it also means that in a sparse reward setting the reward observed would be zero most of the time and no useful features would be learned. In practice methods using the successor representation usually share parts of the network with other tasks to improve representation learning (see e.g. Figure 1 of Machado et al., Eigenoption discovery through the deep successor representation, 2017, & also their 2018 paper).
Experiments:
The experiment are disappointing as they are only limited to tabular and deterministic problems. An obvious missing comparison is to UBE, at the minimum; and other “deep” algorithms such as BDQN, Bootstrap DQN, etc. Some of these algorithms have been shown to perform well on the Atari benchmark, and that seems like a reasonable point of comparison also.
The method also relies on knowing the successor features. While they can be learned easily in a tabular, deterministic MDP it is not clear how the posterior would behave in larger and/or non stochastic domains when it takes more time to learn these successor features.
Overall, I am not sure what I learned from reading this paper. While the idea of using the successor representation in exploration is interesting and has been considered recently, the method presented in this paper needs to be better justified and evaluated on more challenging tasks.
Minor comments
I would like to see a proof of Equation 4, which may be simple but is not immediate.
Some papers of relevance here:
An analysis of model-based Interval Estimation for Markov Decision
Processes, Strehl & Littman (2008)
(More) efficient reinforcement learning via posterior sampling. Osband et al (2013)
Count-Based Exploration with the Successor Representation, Machado et al. (2018)