We show how an ensemble of Q * -functions can be leveraged for more effective exploration in deep reinforcement learning. We build on well established algorithms from the bandit setting, and adapt them to the Q-learning setting. We propose an exploration strategy based on upper-confidence bounds (UCB). Our experiments show significant gains on the Atari benchmark.
This paper paper uses an ensemble of networks to represent the uncertainty in deep reinforcement learning.
The algorithm then chooses optimistically over the distribution induced by the ensemble.
This leads to improved learning / exploration, notably better than the similar approach bootstrapped DQN.
There are several things to like about this paper:
- It is a clear paper, with a simple message and experiments that back up the claims.
- The proposed algorithm is simple and could be practical in a lot of settings and even non-DQN variants.
- It is interesting that Bootstrapped DQN gets such poor performance, this suggests that it is very important in the original paper https://arxiv.org/abs/1602.04621 that "ensemble voting" is applied to the test evaluation... (why do you think this is by the way, do you think it has something to do with the data being *more* off-policy / diverse under a TS vs UCB scheme?)
On the other hand:
- The novelty/scope of this work is somewhat limited... this is more likely (valuable) incremental work than a game-changer.
- Something feels wrong/hacky/incomplete about just doing "ensemble" for uncertainty without bootstrapping/randomization... if we had access to more powerful optimization techniques then this certainly wouldn't be sensible - I think that you should mention that you are heavily reliant on "random initialization + SGD/Adam + specific network architecture" to maintain this idea of uncertainty. For example, this wouldn't work for linear value functions!
- I think the original bootstrapped DQN used "ensemble voting" at test time, so maybe you should change the labels or the way this is introduced/discussed. It's definitely very interesting that *essentially* the learning benefit is coming from ensembling (rather than "raw" bootstrapped DQN) and UCB still looks like it does better.
- I'm not convinced that page 4 and the "Bayesian" derivation really add too much value to this paper... alternatively, maybe you could introduce the actual algorithm first (train K models in parallel) and then say "this is similar to particle filter" and add the mathematical derivation after, rather than as if it was some complex formula derived. If you want to reference some justification/theory for ensemble-based uncertainty approximation you might consider https://arxiv.org/pdf/1705.07347.pdf instead.
- I think this paper might miss the point of the "bigger" problem of efficient exploration in RL... or even how to get "deep" exploration with deep RL. Yes this algorithm sees improvements across Atari, but it's not clear why/if this is a step change versus simply increasing the amount of replay or tuning the learning rate.  (Actually I do believe this algorithm can demonstrate deep exploration... but it looks like we're not seeing the big improvements on the "sub-human" games you might hope.)
Overall I do think this is a pretty good short paper/evaluation of UCB-ensembles on Atari.
The scope/insight of the paper isn't groundbreaking, but I think it delivers a clear short message on the Atari benchmark.
Perhaps this will encourage people to dig deeper into some of these issues... I vote accept.