We propose and evaluate new techniques for compressing and speeding up dense matrix multiplications as found in the fully connected and recurrent layers of neural networks for embedded large vocabulary continuous speech recognition (LVCSR). For compression, we introduce and study a trace norm regularization technique for training low rank factored versions of matrix multiplications. Compared to standard low rank training, we show that our method leads to good accuracy versus number of parameter trade-offs and can be used to speed up training of large models. For speedup, we enable faster inference on ARM processors through new open sourced kernels optimized for small batch sizes, resulting in 3x to 7x speed ups over the widely used gemmlowp library. Beyond LVCSR, we expect our techniques and kernels to be more generally applicable to embedded neural networks with large fully connected or recurrent layers.
The authors propose a strategy for compressing RNN acoustic models in order to deploy them for embedded applications. The technique consists of first training a model by constraining its trace norm, which allows it to be well-approximated by a truncated SVD in a second fine-tuning stage. Overall, I think this is interesting work, but I have a few concerns which I’ve listed below:
1. Section 4, which describes the experiments of compressing server sized acoustic models for embedded recognition seems a bit “disjoint” from the rest of the paper. I had a number of clarification questions spefically on this section:
- Am I correct that the results in this section do not use the trace-norm regularization at all? It would strengthen the paper significantly if the experiments presented on WSJ in the first section were also conducted on the “internal” task with more data.
- How large are the training/test sets used in these experiments (for test sets, number of words, for training sets, amount of data in hours (is this ~10,000hrs), whether any data augmentation such as multi-style training was done, etc.)
- What are the “tier-1” and “tier-2” models in this section? It would also aid readability if the various models were described more clearly in this section, with an emphasis on structure, output targets, what LMs are used, how are the LMs pruned for the embedded-size models, etc. Also, particularly given that the focus is on embedded speech recognition, of which the acoustic model is one part, I would like a few more details on how decoding was done, etc.
- The details in appendix B are interesting, and I think they should really be a part of the main paper. That being said, the results in Section B.5, as the authors mention, are somewhat preliminary, and I think the paper would be much stronger if the authors can re-run these experiments were models are trained to convergence.
- The paper focuses fairly heavily on speech recognition tasks, and I wonder if it would be more suited to a conference on speech recognition. 
2. Could the authors comment on the relative training time of the models with the trace-norm regularizer, L2-regularizer and the unconstrained model in terms of convergence time.
3. Clarification question: For the WSJ experiments was the model decoded without an LM? If no LM was used, then the choice of reporting results in terms of only CER is reasonable, but I think it would be good to also report WERs on the WSJ set in either case.
4. Could the authors indicate the range of values of \lambda_{rec} and \lambda_{nonrec} that were examined in the work? Also, on a related note, in Figure 2, does each point correspond to a specific choice of these regularization parameters?
5. Figure 4: For the models in Figure 4, it would be useful to indicate the starting CER of the stage-1 model before stage-2 training to get a sense of how stage-2 training impacts performance.
6. Although the results on the WSJ set are interesting, I would be curious if the same trends and conclusions can be drawn from a larger dataset -- e.g., the internal dataset that results are reported on later in the paper, or on a set like Switchboard. I think these experiments would strengthen the paper.
7. The experiments in Section 3.2.3 were interesting, since they demonstrate that the model can be warm-started from a model that hasn’t fully converged. Could the authors also indicate the CER of the model used for initialization in addition to the final CER after stage-2 training in Figure 5.
8. In Section 4, the authors mention that quantization could be used to compress models further although this is usually degrades WER by 2--4% relative. I think the authors should consider citing previous works which have examined quantization for embedded speech recognition [1], [2]. In particular, note that [2] describes a technique for training with quantized forward passes which results in models that have smaller performance degradation relative to quantization after training.
References:
[1] Vincent Vanhoucke, Andrew Senior, and Mark Mao, “Improving the speed of neural networks on cpus,” in Deep Learning and Unsupervised Feature Learning Workshop, NIPS, 2011.
[2] Raziel Alvarez, Rohit Prabhavalkar, Anton Bakhtin, “On the efficient representation and execution of deep acoustic models,” Proc. of Interspeech, pp. 2746 -- 2750, 2016.
9. Minor comment: The authors use the term “warmstarting” to refer to the process of training NNs by initializing from a previous model. It would be good to clarify this in the text.