We propose Bayesian Deep Q-Network (BDQN), a practical Thompson sampling based Reinforcement Learning (RL) Algorithm. Thompson sampling allows for targeted exploration in high dimensions through posterior sampling, but is usually computationally expensive. We address this limitation by introducing uncertainty only at the output layer of the network through a Bayesian Linear Regression (BLR) model, which can be trained with fast closed-form updates and its samples can be drawn efficiently through the Gaussian distribution. We apply our method to a wide range of Atari Arcade Learning Environments. Since BDQN carries out more efficient exploration, it is able to reach higher rewards substantially faster than a key baseline, DDQN.
The authors describe how to use Bayesian neural networks with Thompson sampling
for efficient exploration in q-learning. The Bayesian neural networks are only
Bayesian in the last layer. That is, the authors learn all the previous layers
by finding point estimates. The Bayesian learning of the last layer is then
tractable since it consists of a linear Gaussian model. The resulting method is
called BDQL. The experiments performed show that the proposed approach, after
hyper-parameter tuning, significantly outperforms the epsilon-greedy
exploration approaches such as DDQN.
Quality:
I am very concern about the authors stating on page 1 "we sample from the
posterior on the set of Q-functions". I believe this statement is not correct.
The Bayesian posterior distribution is obtained by combining an assumed
generative model for the data, data sampled from that model and some prior
assumptions. In this paper there is no generative model for the data and the
data obtained is not actually sampled from the model. The data are just targets
obtained by the q-learning rule. This means that the authors are adapting
Q-learning methods so that they look Bayesian, but in no way they are dealing
with a principled posterior distribution over Q-functions. At least this is my
opinion, I would like to encourage the authors to be more precise and show in
the paper what is the exact posterior distribution over Q-functions and show
how they approximate that distribution, taking into account that a posterior
distribution is obtained as $p(theta|D) \propto p(D|theta)p(\theta)$. In the
case addressed in the paper, what is the likelihood $p(D|\theta)$ and what are
the modeling assumptions that explain how $D$ is generated by sampling from a
model parameterized by \theta?
I am also concerned about the hyper-parameter tuning for the baselines. In
section 5 (choice of hyper-parameters) the authors describe a quite exhaustive
hyper-parameter tuning procedure for BDQL. However, they do not mention whether
they perform a similar hyper-parameter tuning for DDQN, in particular for the
parameter epsilon which will determine the amount of exploration. This makes me
wonder if the comparison in table 2 is fair. Especially, because the authors
tune the amount of data from the replay-buffer that is used to update their
posterior distribution. This will have the effect of tuning the width of their
posterior approximation which is directly related to the amount of exploration
performed by Thompson sampling. You can, therefore, conclude that the authors are
tuning the amount of exploration that they perform on each specific problem.
Is that also being done for the baseline DDQN, for example, by tuning epsilon in
each problem?
The authors also report in table 2 the scores obtained for DDQN by Osband et
al. 2016. What is the purpose of including two rows in table 2 with the same
method? It feels a bit that the authors want to hide the fact that they only
compare with a singe epsilon-greedy baseline (DDQN). Epsilon-greedy methods
have already been shown to be less efficient than Bayesian methods with
Thompson sampling for exploration in q learning (Lipton et al. 2016).
The authors do not compare with variational approaches to Bayesian learning
(Lipton et al. 2016). They indicate that since Lipton et al. "do not
investigate the Atari games, we are not able to have their method as an
additional baseline". This statement seems completely unjustified. The authors
should clearly include a description of why Lipton's approach cannot be applied
to the Atari games or include it as a baseline. 
The method proposed by the authors is very similar to Lipton's approach. The
only difference is that Lipton et al. use variational inference with a
factorized Gaussian distribution to approximate the posterior on all the
network weights. The authors by contrast, perform exact Bayesian inference, but
only on the last layer of their neural network. It would be very useful to know
whether the exact linear Gaussian model in the last layer proposed by the
authors has advantages with respect to a variational approximation on all the
network weights. If Lipton's method would be expensive to apply to large-scale
settings such as the Atari games, the authors could also compare with that
method in smaller and simpler problems.
The plots in Figure 2 include performance in terms of episodes. However, it
would also be useful to know how much is the extra computational costs of
the proposed method. One could imagine that computing the posterior
approximation in equation 6 has some additional cost. How do BDQN and DDQN
compare when one takes into account running time and not episode count into
account?
Clarity:
The paper is clearly written. However, I found a lack of motivation for the
specific design choices made to obtain equations 9 and 10. What is a_t in
equation 9? The parameters \theta are updated just after equation 10 by
following the gradient of the loss in which the weights of the last layer are
fixed to a posterior sample, instead of the posterior mean. Is this update rule
guaranteed to produce convergence of \theta? I could imagine that at different
times, different posterior samples of the weights will be used to compute the
gradients. Does this create any instability in learning? 
I found the paragraph just above section 5 describing the maze-like
deterministic game confusing and not very useful. The authors should improve
this paragraph.
Originality:
The proposed approach in which the weights in the last layer of the neural
network are the only Bayesian ones is not new. The same method was proposed in
Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., ... &
Adams, R. (2015, June). Scalable Bayesian optimization using deep neural
networks. In International Conference on Machine Learning (pp. 2171-2180).
which the authors fail to cite. The use of Thompson sampling for efficient
exploration in deep Q learning is also not new since it has been proposed by
Lipton et al. 2016. The main contribution of the paper is to combine these two
methods (equations 6-10) and evaluate the results in the large-scale setting of
ATARI games, showing that it works in practice.
Significance:
It is hard to determine how significant the work is since the authors only
compare with a single baseline and leave aside previous work on efficient
exploration with Thompson sampling based on variational approximations.
As far as the method is described, I believe it would be impossible to
reproduce their results because of the complexity of the hyper-parameter tuning
performed by the authors. I would encourage the authors to release code that can
directly generate Figure 2 and table 2.