Tensorizing Neural Networks
Deep neural networks currently demonstrate state-of-the-art performance in several domains. At the same time, models of this class are very demanding in terms of computational resources. In particular, a large amount of memory is required by commonly used fully-connected layers, making it hard to use the models on low-end devices and stopping the further increase of the model size. In this paper we convert the dense weight matrices of the fully-connected layers to the Tensor Train [17] format such that the number of parameters is reduced by a huge factor and at the same time the expressive power of the layer is preserved. In particular, for the Very Deep VGG networks [21] we report the compression factor of the dense weight matrix of a fully-connected layer up to 200000 times leading to the compression factor of the whole network up to 7 times.
Fully connected dense layers (W x + b) can be a nasty issue for neural networks since they require storing and working with MxN parameters, and this is often a bottleneck for producing larger models. However past work has shown that there is redundancy in this representation. This motivates decomposing the matrix W into smaller components, but there are several difficulties with doing this in practice.In this paper, the authors utilize a general tensor decomposition form known as "Tensor Train" (Oseledets, 2011). They then describe and implement a standard dense layer utilizing this decomposition. The decomposition has several nice properties: 1) there are fast algorithms for running standard linear algebraic ops, and these can be used to implement f/b-prop, 2) the method is numerically stable even decomposed into many parts. They then use this method to decompose a dense layer into a much smaller number of parameters.The main result of the work is shows that a dense layer in vgg can be replaced with a decomposed layer with a very large reduction in parameters and only a small reduction in accuracy, with no reduction in speed.This is a very large improvement of a baseline method that just learns a low-rank representation. This seems like a impressive and useful result. The paper itself could be much improved in terms of clarity. Some issues:- Generally the figure and table captions are very hard to read and leave several important- Figure 1 in particular is very hard to read and decipher.- Having everything in the paper start with TT- becomes confusing. Naming could be simplified- There are several typos and grammatical issues, be sure to read over closely.  Questions:- How does Tensor train compare here to other tensor decomposition formats?