GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking
Model compression is essential for serving large deep neural nets on devices with limited resources or applications that require real-time responses. As a case study, a neural language model usually consists of one or more recurrent layers sandwiched between an embedding layer used for representing input tokens and a softmax layer for generating output tokens. For problems with a very large vocabulary size, the embedding and the softmax matrices can account for more than half of the model size. For instance, the bigLSTM model achieves great performance on the One-Billion-Word (OBW) dataset with around 800k vocabulary, and its word embedding and softmax matrices use more than 6GBytes space, and are responsible for over 90% of the model parameters. In this paper, we propose GroupReduce, a novel compression method for neural language models, based on vocabulary-partition (block) based low-rank matrix approximation and the inherent frequency distribution of tokens (the power-law distribution of words). The experimental results show our method can significantly outperform traditional compression methods such as low-rank approximation and pruning. On the OBW dataset, our method achieved 6.6 times compression rate for the embedding and softmax matrices, and when combined with quantization, our method can achieve 26 times compression rate, which translates to a factor of 12.8 times compression for the entire model with very little degradation in perplexity.
The sizes of embedding matrices in NLP tasks have long posed difficult computational problems, either from the inefficiency of operating (softmaxing) over them, or often from the sheer difficulty in storing them.  In this paper the authors take on the latter problem, introducing a method of using multiple low-rank approximations to reduce the size of these matrices.  They rely on frequency binning — the same observation underlying the hierarchical softmax solution to the former problem — to group words, prioritizing the most frequent words to receive higher rank approximations.  This itself leads to significant compression rates with little loss in accuracy, and when further combined with quantization, yields large reduction in memory.  Importantly quantization appears to play nicely with their methodology, and the combined seem to provide much smaller models overall, while performing at least as well as naive quantiziation on large data sets.  The method is simple but effective, and the paper is clearly written, the problem of obvious importance, so my main critique comes from _how_ the material is presented.  (1)  I thought some of the comparisons were more arbitrary than I’d like.  The size of the parameter reduction vs. performance is, of course, a spectrum.  So, for instance, to present a 40x reduction on small LSTM, when the perplexity hit was huge, is not so informative IMO.  It’s already a small model — it’s less interesting to see it reduced 40x vs. the larger ones.  I’d be more interested to see how much it can be reduced while retaining approximately full performance level.  This could be said for many of the presented results — they all seem arbitrarily chosen and while some provide clear take-aways, these are very small glimpses into the larger size-accuracy trade-off at hand.  I’m even tempted to say that, despite what appear to be very compelling results in specific cases on an important problem, and a technique with great practical importance, I’d rather see it resubmitted with more attention paid to painting a clear picture of these trade-offs.  (2)  Some things were over-stated.  Biggest offender: "On the OBW dataset, our method achieved 6.6x compression rate for the embedding and softmax matrices, and when combined with quantization, our method can achieve 26x compression rate without losing prediction accuracy.”  In this setting perplexity goes from 31.04 to 32.47.  I don’t want to claim this is going to make any practical difference, but I don’t understand the need to overstate the effectiveness of this approach when it already does so well.  This is easy to fix so please ammend strong but unsupported claims.  Question for the authors:  If perplexity is all one cares about, and each token is considered uniformly important, than such frequency binning here is a well-motivated decision.  Still, I wonder for many domains where many of the most frequent words are not so semantically interesting, if this will have a negative effect on end-task performance.  Perhaps something like speech recognition, or other product models, where the most frequent words are seen often by the core model and the language model provides little useful information then about them, but now provides poorer approximations to rarer words the core model is also likely to misidentify.  Have the authors looked into a more qualitative error analysis of where the reduced models goes wrong?  Or along these lines, is there anything to say about the post-GroupReduce cluster membership?  Are there any conceptual trends in how the goal of reducing reconstruction error adjusted the freq-based initial partitionings?  Can you comment on the computation time of these experiments?  Is there a noticeable performance difference on compressing just the embedding matrix vs. just the output matrix?  I’m curious if the compression “gains” come from both about equally, or are coming primarily from one or the other.   Misc:  "One word’ rank is" --> typo  Table 2 needs a label of its metric within the figure  "The overall algorithm is in Algorithm 1. Figure 2 illustrates our overall algorithm." —> confusing  "due to the lack of publicly released model checkpoint." —> grammar  "2-LAYER LSTM-8192-1024” —> this model is not shown as mentioned  AVp as AVp ≈ Up(V p)T —> not sure if this is a type or odd phrasing, but could ditch the “as AVp"