Learning Compressed Transforms with Low Displacement Rank
The low displacement rank (LDR) framework for structured matrices represents a matrix through two displacement operators and a low-rank residual. Existing use of LDR matrices in deep learning has applied fixed displacement operators encoding forms of shift invariance akin to convolutions. We introduce a class of LDR matrices with more general displacement operators, and explicitly learn over both the operators and the low-rank component. This class generalizes several previous constructions while preserving compression and efficient computation. We prove bounds on the VC dimension of multi-layer neural networks with structured weight matrices and show empirically that our compact parameterization can reduce the sample complexity of learning. When replacing weight layers in fully-connected, convolutional, and recurrent neural networks for image classification and language modeling tasks, our new classes exceed the accuracy of existing compression approaches, and on some tasks also outperform general unstructured layers while using more than 20x fewer parameters.
# Post rebuttal  Given new results and comparisons provided by the authors I am increasing initial score of (6) to (7).  # Original review  This paper focuses on an low displacement rank matrices used as parametrisations of neural networks and more precisely on defining a rich class of them, which can be used while preserving reasonable VC bounds.  I find presented results interesting and valuable, however it is unclear how significant for the community they really are. Authors seem to view the approach as a way to decrease overparametrisation of the networks with minimal effect on accuracy. However, there are dozens of techniques which try to address this very issue, which are not really compared against. Instead, authors choose to focus on comparing to other, very similar approaches of reparametrising neural network layers with LDRs.  Pros: - Clear message - Visible extension of previous work/results - Showing both empirically and theoretically how proposed method improves upon baselines - Providing implementation of the method, thus increasing reproducability  Cons: - all experiments are relatively low-scale, and the only benefits over not constraining the structure is obtained for MNIST-noise (90 vs 93.5%) and CIFAR-10 (65 vs 66%) which are not very significant differences at this level of accuracy for these problems. Since this paper is not evaluation-centric, I am still in favour of accepting, but in a marginal sense; if provided with stronger empirical evidence for the method I would be keen to change the score. On the other hand there are many other ways of reducing number of parameters of a neural network, so if the evaluation claim is "minimisation of number of weights given small accuracy drop" than one should compare against exactly these approaches, such as: network prunning, even methods like "optimal brain surgeon" from 90s; micro-architectures which were found explicitely to minimise this constraint (like MobileNets etc.).  - significance of these results are quite limited as the empirical evaluation shows limited benefits (or at least the proper evaluation metric, a number authors want to optimise, is unclear to the reviewer) and the theoretical results, while valuable, don't seem strong enough to consider it as a purely math paper.  Minor remarks: - please fix language errors/typos, such as: Table 2 caption "has perplexity is", using "low rank" and "low-rank" (please pick one),