Deep reinforcement learning requires a heavy price in terms of sample efficiency and overparameterization in the neural networks used for function approximation. In this work, we employ tensor factorization in order to learn more compact representations for reinforcement learning policies. We show empirically that in the low-data regime, it is possible to learn online policies with 2 to 10 times less total coefficients, with little to no loss of performance. We also leverage progress in second order optimization, and use the theory of wavelet scattering to further reduce the number of learned coefficients, by foregoing learning the topmost convolutional layer filters altogether. We evaluate our results on the Atari suite against recent baseline algorithms that represent the state-of-the-art in data efficiency, and get comparable results with an order of magnitude gain in weight parsimony.
This paper investigates a list of methods to reduce the number of weights for deep RL architecture under the low-data regime. These methods include tensor regression, wavelet scattering, as well as second-order optimization (K-FAC). The experiments on the Atari games shows that by using tensor regression to replace the dense layer of the neural nets and using K-FAC for the optimization, one can reduce around 10 times of parameters without losing too much of performance. 
However, I have some concerns on the novelty of this work and therefore I’m giving this paper a weak reject. Here are the reasons:
To begin with, leveraging tensor structure of the neural nets to reduce number of parameters while maintaining similar level or getting even better results have been done before, for example: Tensorizing Neural Networks (Novikov et al, 2015), Learning compact recurrent neural networks with block-term tensor decomposition (Ye et al., 2018) etc. Although the use of tensor regression might be new, the core idea is still to leverage the low rank property of the tensor and obtain a compression of the weight tensors. Moreover, why use Tucker decomposition specifically for the tensor regression? It has been proposed that using tensor train (TT) decomposition can also get very good results (see Garipov et. al. Ultimate tensorization: compressing convolutional and FC layers alike). Is it possible to investigate the use of TT decomposition for the dense layer of the deep RL architecture? Therefore the novelty for this aspect seems a bit weak for me.
The second method the authors have attempted is to swap the convolution layer of the deep RL architecture with wavelet scattering. For one particular game (demon_attack), this approach seems to outperform every other methods by a large margin. However the experiment shows that for the rest of the Atari games, there is a huge drop (45%) of performance. Therefore the significance of this approach is rather thin for me. Maybe some further investigation of the game demon_attack is needed to understand why using scattering in this game in particular gives such a huge performance boost. 
Thirdly, as an approximation of the second order optimization, K-FAC does not really concern with the main theme of the paper, which is an investigation of potential weights reduction methods. It is great that the authors applied this techniques and seems to have great results. However, as the authors pointed out, K-FAC has been wildly applied in the deep RL literature, and the authors did not propose new extension for the K-FAC method, therefore the contribution of this matter is also quite thin. 
Last but not least, the writing of the paper is a bit clumsy, and I was having a hard time to figure out what exactly is the proposed method. I think this paper might need some rework on the writing to describe the idea of the authors in a more clear way for the publication. Due to these reasons, I’m giving this paper a weak reject. 
Some writing comments and potential writing errors (did not affect the decision):
Page 3, first line of “Tensor regression layer”, the shape of the tensor X seems to be a typo. 
Also here, the definition of <X, Y>_N in the paper is to sum over the dimension of I_1…I_N, then the shape of <X, Y>_N should be K_1*…*K_x*L_1*…*L_y, without the I_N in the middle. 
Also in this section, the authors mentioned Tucker decomposition for the tensor regression. However the phrasing of this sentence needs a bit rework. The usage of “For instance here”, gives the readers a feeling that Tucker is just one possible way of doing this decomposition, but not necessarily the actual decomposition for the reported experiments. 
In 2.3, there is a lack of definition for  \Lambda_1 and \Lambda_2. In addition, it would be better for the general readers to add a few definitions for the terminologies in this section. For example, “circular convolution”, “wavelet filter banks” etc. I guess people with corresponding background will understand it with no problem, however I do find myself a bit lost in this section with these terminologies. 
2.4 line 6, “with A and. B smaller, architecture-dependent matrices”. I think it should be “with A and B being…“
3.1, line 5, “This is all the more pressing that….”, I did not understand this sentence. 
In page 6, line 3, there is a lack of definition for “compression rate”. Is it the compression rate w.r.t only the last dense layer, or w.r.t the whole network?
Figure 4 is lacking y-axis and x-axis labels. 
4.2, last bullet point, “However, one must not forget that the conv layers one learns must be somehow be well adapted…”, I get what you are saying, but the sentence is a bit clumsy. 
Table 1 and 2, the row name “Average” is lacking definition. 
 Overall it is a good attempt to reduce the number of weights in the deep RL architecture, but I do think the novelty of this work is a bit thin and the three contributions were not tied together with the main theme of the paper. Therefore, I’m giving this work a weak reject.