Optimistic initialisation is an effective strategy for efficient exploration in reinforcement learning (RL). In the tabular case, all provably efficient model-free algorithms rely on it. However, model-free deep RL algorithms do not use optimistic initialisation despite taking inspiration from these provably efficient tabular algorithms. In particular, in scenarios with only positive rewards, Q-values are initialised at their lowest possible values due to commonly used network initialisation schemes, a pessimistic initialisation. Merely initialising the network to output optimistic Q-values is not enough, since we cannot ensure that they remain optimistic for novel state-action pairs, which is crucial for exploration. We propose a simple count-based augmentation to pessimistically initialised Q-values that separates the source of optimism from the neural network. We show that this scheme is provably efficient in the tabular setting and extend it to the deep RL setting. Our algorithm, Optimistic Pessimistically Initialised Q-Learning (OPIQ), augments the Q-value estimates of a DQN-based agent with count-derived bonuses to ensure optimism during both action selection and bootstrapping. We show that OPIQ outperforms non-optimistic DQN variants that utilise a pseudocount-based intrinsic motivation in hard exploration tasks, and that it predicts optimistic estimates for novel state-action pairs.
Optimistic Exploration even with a Pessimistic Initialisation
================================================================
This paper presents an exploration algorithm based on "optimism in the face of uncertainty" via count-based bonus.
The authors observe that typical neural net initializations close to zero can be pessimistic, but show that augmenting a count-based bonus for acting and bootstrapping can overcome this.
The authors support their claim with an adaptation of a regret bound for the tabular case, and a series of didactic experiments with neural net models.
There are several things to like about this paper:
- Exploration with generalization in Deep RL is a large outstanding problem, with few effective options and none really commonly used in the field beyond epsilon-greedy or Boltzmann.
- This algorithm is reasonably well thought out, building on an established literature of exploration bonuses, but with a slightly different take on the structure of the bonus.
- The paper is well structured, building from intuition to theory to toy examples in DQN setting.
- Overall the algorithm appears to perform well against a wide variety of related variants (although that presentation is a little confusing / overwhelming).
There are several places the paper might be improved:
- I don't think the authors make a clear enough case for why this method is *better* than the other optimistic bonus approaches listed... Yes there is a regret bound, but this is not as good as some other methods... Yes there are ablations... but they're not really clear about what the mechanism that makes this method better than others!
- Although this algorithm is motivated by applications to *deep* RL, the key choice of the "count" (and thus the method for optimism bonus) is mostly sidestepped. It amounts to an essentially tabular bonus in the space of the hashing function... and it's not clear why this approach should work any better or worse than other similar approaches that the paper complains about. For example, if you used "Randomized Prior Functions" or "Random Network Distillation" with that same hashing functions you would likely end up with similar results?
- The comparison to benchmark algorithms seems quite confusing and I'm not sure if it's really presented well. It might be good to focus on fewer comparisons at a time and push remaining less important ones to the appendix.
- It would be great to get an evaluation of these algorithms on a standardized and open-source benchmark... and I think that bsuite could be a really good candidate for this paper https://github.com/deepmind/bsuite particularly the "deep sea" experiments.
Overall I think this is a reasonable paper, and I expect it to improve during the review process.
At the moment I have to say that I don't think there is a clear enough case for why this method is preferable to other similar approaches, or enough insight into the pros and cons to accept.
============
Updating to "weak accept" as part of rebuttal.