VITERBI-BASED PRUNING FOR SPARSE MATRIX WITH FIXED AND HIGH INDEX COMPRESSION RATIO
Weight pruning has proven to be an effective method of reducing the model size and computation cost without sacrificing its model accuracy. Conventional sparse matrix formats, however, involve irregular index structures with large storage requirement and a sequential reconstruction process, resulting in inefficient use of highly parallel computing resources. Hence, pruning is usually restricted to inference with a batch size of one, for which an efficient parallel matrix-vector multiplication method exists. In this paper, a new class of sparse matrix representation is proposed utilizing the Viterbi algorithm that has a high, and more importantly, fixed index compression ratio regardless of the pruning rate. In this approach, numerous sparse matrix candidates are first generated by the Viterbi encoder, and the candidate that aims to minimize the model accuracy degradation is then selected by the Viterbi algorithm. The model pruning process based on the proposed Viterbi encoder and Viterbi algorithm is highly parallelizable, and can be implemented efficiently in hardware to achieve low-energy and a high-performance index decoding process. Compared with the existing magnitude-based pruning methods, the index data storage requirement can be further compressed by 85.2% in MNIST and 83.9% in AlexNet while achieving a similar pruning rate. Even compared with the relative index compression technique, our method can still reduce the index storage requirement by 52.7% in MNIST and 35.5% in AlexNet.
The paper proposes VCM, a novel way to store sparse matrices that is based on the Viterbi Decompressor. Only a subset of sparse matrices can be represented in the VCM format, however, unlike CSR format, it allows for faster parallel decoding and requires much less index space. The authors also propose a novel method of pruning of neural network that constructs an (sub)optimal (w.r.t. a weight magnitude based loss) Viterbi-compressed matrix given the weights of a pretrained DNN.
VCM is an interesting analog to the conventional CSR format that may be more computationally efficient given particular software and/or hardware implementations of the Viterbi Decompressor. However, the empirical study of possible acceleration remains as an open question.
However, I have a major concern regarding the efficiency of the pruning procedure. Authors report practically the same level of sparsity, as the pruning procedure from the Deep Compression paper. Both the proposed Viterbi-based pruning, and Deep Compression pruning belong to the previous era of pruning methods. They separate the pruning procedure and the training procedure, so that the model is not trained end-to-end. However, during the last two years a lot of new adaptive pruning methods have been developed, e.g. Dynamic Network Surgery, Soft Weight Sharing, and Sparse Variational DropOut. All of them in some sense incorporate the pruning procedure into the training procedure and achieve a much higher level of sparsity (e.g. DC achieves ~13x compression of LeNet5, and SVDO achieves ~280x compression of the same network). Therefore the reported 35-50% compression of the index storage is not very significant.
It is not clear whether it is possible to take a very sparse matrix and transform it into the VCM format without a high accuracy degradation. It is also not clear whether the VCM format would be efficient for storage of extremely sparse matrices, as they would likely be more sensitive to the mismatch of the original sparsity mask, and the best possible VCM sparsity mask. Therefore I’m concerned whether it would be possible to achieve a close-to-SotA level of compression using this method, and it is not yet clear whether this method can be used for practical acceleration or not.
The paper presents an interesting idea that potentially has useful applications, however the experiments are not convincing enough.