Shallow Updates for Deep Reinforcement Learning
Deep reinforcement learning (DRL) methods such as the Deep Q-Network (DQN) have achieved state-of-the-art results in a variety of challenging, high-dimensional domains. This success is mainly attributed to the power of deep neural networks to learn rich domain representations for approximating the value function or policy. Batch reinforcement learning methods with linear representations, on the other hand, are more stable and require less hyper parameter tuning. Yet, substantial feature engineering is necessary to achieve good results. In this work we propose a hybrid approach -the Least Squares Deep Q-Network (LS-DQN), which combines rich feature representations learned by a DRL algorithm with the stability of a linear least squares method. We do this by periodically re-training the last hidden layer of a DRL network with a batch least squares update. Key to our approach is a Bayesian regularization term for the least squares update, which prevents over-fitting to the more recent data. We tested LS-DQN on five Atari games and demonstrate significant improvement over vanilla DQN and Double-DQN. We also investigated the reasons for the superior performance of our method. Interestingly, we found that the performance improvement can be attributed to the large batch size used by the LS method when optimizing the last layer.
The authors propose to augment value-based methods for deep reinforcement learning (DRL) with batch methods for linear approximation function (SRL). The idea is motivated by interpreting the output of the second-to-last layer of a neural network as linear features. In order to make this combination work, the authors argue that regularization is needed. Experimental results are provided for 5 Atari games on combinations of DQN/Double DQN and LSTD-Q/FQI.
Strengths:
I find the proposition of combining DRL and SRL with Bayesian regularization original and promising.
The explanation provided for the improved performance seems reasonable, but it could have been better validated (see below).
Weaknesses:
The presentation of Algorithm 1 and in particular line 7, is a bit strange to me, given that in Section 4, the authors mention that generating D with the current weights results in poor performance. Why not present line 7 as using ER only? Besides, would it be interesting to use p% of trajectories from ER and (1-p)% of trajectories generated from the current weights?
The experiments could be more complete. For instance, in Table 1, the results for LS-DDQN_{LSTD-Q} are missing and in Fig.2, the curves for Qbert and Bowling are not provided. 
For a fairer comparison between DQN and LS-DQN, I think the authors should compare Algorithm 1 against DQN given an equal time-budget to check that their algorithm indeed provides at the end a better performance.
Besides, the ablative analysis was just performed on Breakout. Would the same conclusions hold on different games?
Detailed comments:
l.41: the comment on data-efficiency of SRL is only valid in the batch setting, isn’t it? 
l.114, l.235: it’s 
l.148: in the equation above, \mu should be w_k. I find the notation w_k for the last hidden layer to be unsatisfying, as w_k also denotes the weights of the whole network at iteration k
l.205, l.245: a LS -> an LS
l.222: algortihm